installation tarball can/should be cached to avoid re-downloads

vn971 commented 9 years ago

The current installation script https://nixos.org/nix/install does not cache the downloadable ~70Mb installation tarball. This is bad in all cases where installation fails for some reason. Examples:

a network failure may occur. It may be preferable to resume the download: https://github.com/NixOS/nix/issues/379
sudo command may not be available on the target system, practically failing the installation. This is default for Debian 7-8. https://github.com/NixOS/nix/issues/539
the script may be run as root accidentally. This is not recommended and a user may want to install Nix in userland. He will be forced to re-download the tarball again.
if you accidentally run the script as root first, the /nix directory would be owned by root and installation will fail again (asking you to chown).

The proposed alternative is to cache the tarball. For example, the script could download the tarball first and ask the user whether to delete the tarball on exit. This will save both time and brandwidth for the user.

wmertens commented 9 years ago

Ideally it should also use a checksum if available on the system. Perl is available on most systems and has sha256 in core: perl -e 'use Digest::SHA 'sha256_hex'; print sha256_hex(<tarball.tgz>)'.

vn971 commented 9 years ago

@wmertens You mean to avoid out-of-date tarballs? Basically, tarball caching and verification seem to be different jobs.

wmertens commented 9 years ago

No I mean to verify that the tarball is the one that was expected.

On Tue, May 19, 2015, 1:11 PM Vasya Novikov notifications@github.com wrote:

@wmertens https://github.com/wmertens You mean to avoid out-of-date tarballs? Basically, tarball caching and verification seem to be different jobs.

— Reply to this email directly or view it on GitHub https://github.com/NixOS/nix/issues/540#issuecomment-103443673.

Anton-Latukha commented 7 years ago

I step up with critique of this issue:

As a DevOps I never seen installation oneliner (as https://nixos.org/nix/install) that cache tarballs.

Look at Docker, Salt, Ansible... oneliners. There is no caching. If you find one share with me.

It is not the thing that shell script should do or can do great. There are huge projects that solve caching (Apache Traffic Server, Squid... and so on, AWS/GCE/Azure has caching, CDNs has caching). So it is somewhat not fare task to ask shell script to do self-invented caching.

Oneliner with caching introduces deliberate sideeffect. And it must have less side-effects. I can argue with issue on the contratry: if oneliner failed - it should pull/roll back as nothing happened to the system, and oneliner was not ran. And it is what is does it now. It cleans-up. And with caching - cache must be left.

One of main points and uniqueness of Nix is that it is fully transactional.

Caching introduces security concerns.

If do caching, now Nix oneliner has next concerns:

a. Tarball should be in constant place (or script do recurcive search in /tmp to find it). So script can use it. And then this can be exploited, by replacing tarball with malicious one. Than, script must do hash check. It does compare SHA256. But it does it currently to check does file was downloaded properly.
b. SHA256 is not secure nowadays: publications. SHA256 is sufficient to check if file was downloaded properly, without security requirement. We rely on HTTPS security then. SHA256 can be attacked to do malicious execution through "cached" file. So if do caching - hashing process must be changed to other standard.

Now Nix uses classic mktemp -d -t nix-binary-tarball-unpack.XXXXXXXXXX. If error encountered - script does cleanup and removes this directory. It uses https:// to download tarball. Which is point that secure enough for me. And if I use https oneliner - than I agree that it uses https install.

I see current approach as classical and KISS and efficient. And also one points of Unix philosophy - oneliner should work as user expects, and oneliners work this way.

And I see drawbacks to do caching with shell script. And there is a point why noone does that. If it is at stage that is is concern for you. You at scale that you should use different approach in spawning infrastructure, or use caching solutions.

Also Nix guys have a lot of more important actions to do, and your #539 is much more important. It is a bug, that script does not check requirements before it does actions. And it is issue between oneliner&install scripts.

vn971 commented 7 years ago

// SHA2 has no second pre-image attacks. Even MD5 has no second pre-image attacks. It's different attacks that are possible, but not second pre-image. https://en.wikipedia.org/wiki/Preimage_attack

Anyway, I personally don't use NixOS anymore, and I don't have the energy to really participate in the issue. Feel free to hijack.

wmertens commented 7 years ago

The oneliner downloads a script that then downloads the tarball. It can provide caching.

On Tue, Aug 29, 2017, 3:18 PM Vasya Novikov notifications@github.com wrote:

// SHA2 has no second pre-image attacks. Even MD5 has no second pre-image attacks. It's different attacks that are possible, but not second pre-image. https://en.wikipedia.org/wiki/Preimage_attack

Anyway, I personally don't use NixOS anymore, and I don't have the energy to really participate in the issue. Feel free to hijack.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NixOS/nix/issues/540#issuecomment-325660735, or mute the thread https://github.com/notifications/unsubscribe-auth/AADWlvIzVXER2g0LR83piXCO6lD250QGks5sdA-mgaJpZM4EgJDC .

Anton-Latukha commented 7 years ago

@vn971

Yes, you are right about SHA256. Collision resistance is not the same as preimage attack. As I understand, it can be the first or second one here, as input and function are known. In papers claimed they can go as far as 52 out of 64 rounds for SHA-256 to break it. Especially since quantum computers become our reality, SHA-256 probably go away as recommended by security in nearest future.

Point of SHA256 is supplementary one, all other points are still legitimate and more important by my view.

Anton-Latukha commented 7 years ago

@vn971

I wrote message not because I want to argue. I saw a place to raise does caching needed. While I find when Nix installation is by itself not in perfect state. It assumes noncalsical modes to install (install as normal user) while sudo with root escalation should be present. List of preassumed conditions are not true for many Linux distributions in clean state (Arch, Alpine, and I see should hit many others). Building from source also requires reverse-engineering on this systems.

And that is where I try to contribute today (working on scripts locally today). While I check bugreports, that I can close on pull request.

copumpkin commented 7 years ago

If you're concerned about the security of SHA256, we use it all over the place, so I'd open a separate issue. I personally think it's a tad premature (understatement) to advocate moving away from it given the state of the world and pain involved though.

Anton-Latukha commented 7 years ago

@wmertens

You can check that now oneliner script does downloads whole tarball.

Under 'oneliner' I mean https://nixos.org/nix/install script itself.

Anton-Latukha commented 7 years ago

@copumpkin

As I said, SHA256 is a minor topic at least for me here. But it can be a big questiong on big-scale security requirements when disputing SHA256 starts to happen.

I see many other points are important for now.

I propose to close this issue. I don't see this feature important while setup scripts itself are not ironed-out to trouble-free setup on Linux distributions. It is my view. Downloading 30MB several times during manual install on one system is not issue for me. If Nix install deploys on scale, that it needs to be automatic (than script-level caching on instance is not needed), prebuilt in image/cached/setup must meet requirements of Nix installation.

And I try to contribute some pull requests to close couple of that issues.

wmertens commented 7 years ago

@Anton-Latukha The oneliner has to download the script, but the script could use the cached tarball. Between invocations, $TMPDIR is the same, but it's different per login. So the script could just check if the tarball exists in $TMPDIR/tarball.tgz with the correct size and hash, before attempting download.

I'm having a hard time imagining a doctored tgz file that has the same hash and size as the correct tgz file and also unpacks without problems.

Downloading even 30MB can be a lot. No reason to do it if we can just check for the cached file.

Anton-Latukha commented 7 years ago

@wmertens Current script runs:

tmpDir="$(mktemp -d -t nix-binary-tarball-unpack.XXXXXXXXXX || \
          oops "Can\'t create temporary directory for downloading the Nix binary tarball")"

It is different for every invocation. Maybe you talk about older version.

wmertens commented 7 years ago

My point is that it should be caching, not that it is doing it now…

On Sat, Sep 9, 2017, 8:17 PM Anton Latukha notifications@github.com wrote:

@wmertens https://github.com/wmertens Current oneliner runs:

tmpDir="$(mktemp -d -t nix-binary-tarball-unpack.XXXXXXXXXX || \ oops "Can\'t create temporary directory for downloading the Nix binary tarball")"

It is different for every invocation. Maybe you talk about older version.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NixOS/nix/issues/540#issuecomment-328294771, or mute the thread https://github.com/notifications/unsubscribe-auth/AADWltacgzTT0qQFyl-68rvXzNnKN7Enks5sgtZRgaJpZM4EgJDC .

stale[bot] commented 3 years ago

I marked this as stale due to inactivity. → More info

stale[bot] commented 2 years ago

I closed this issue due to inactivity. → More info

NixOS / nix

installation tarball can/should be cached to avoid re-downloads #540