nix package manager?

[abathur]

Stumbled onto this after finding bashup/events. It's probably out-of-scope given the bash orientation of everything here, but just in case it's helpful:

Have you tried out the nix package manager?

One of its strong points is specifying and managing cross-ecosystem dependencies consistently. While they are usually specified in a .nix file, individual scripts can even specify dependencies via shebang line.

[pjeby]

Heard about it, never got around to trying it; the objective of .devkit and most of the bashup tools is to be "zero install" for machines with a few common tools like bash and git. nix sort of seems like "docker minus the containers", which itself is kind of an interesting concept, but one which I haven't really had time or inclination to explore as yet.

How did you find bashup/events? I'm curious about what other people are using it for.

[abathur]

I suspected it might not quite square, but it might still be worth poking at a little. Docker without containers isn't entirely unfair, but the nix-shell or nix run commands support something pragmatically akin to zero-install (after you install nix, at least...).

For a quick example, i'll use nix-shell to access the yq yaml wrapper for jq. The nixpkgs package definition for yq specifies 3 dependencies: pyyaml xmltodict jq

# Nix already installs jq in my user profile/config (path is a symlink)
$ which jq
/run/current-system/sw/bin/jq

# but I don't have any of the others installed
$ which yq
$ python -c 'import xmltodict, yaml; print(xmltodict); print(yaml)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'xmltodict'

# open a nix shell with the package yq, installing dependencies
$ nix-shell -p yq
these paths will be fetched (0.53 MiB download, 3.57 MiB unpacked):
  /nix/store/4ixxl0jdcq0zz7xbzmrqg2cpg1wpsi63-stdenv-darwin
  /nix/store/8gwc4dwxhf9vxiqfagrpwk4nip0mng09-jq-1.6-dev
  /nix/store/d7pi7sg02ywhwg3cyh75fkz9y4v6lgv3-bash-interactive-4.4-p23-doc
  /nix/store/fs712dq71wz98vv2140wmdmgi1md14gn-python3.7-xmltodict-0.12.0
  /nix/store/k4p65gnbzhy97y62blmn83aycj698hph-python3.7-PyYAML-3.13
  /nix/store/k8hmyxw736p585mkvd28p4wfh2gs3il7-adv_cmds-osx-10.5.8-locale
  /nix/store/kjjsrih4093869f2mpf96msmgic74zip-yq-2.7.2
  /nix/store/vxdyd32056p8jnmvj6hgl87g9crhv576-bash-interactive-4.4-p23-dev
copying path '/nix/store/d7pi7sg02ywhwg3cyh75fkz9y4v6lgv3-bash-interactive-4.4-p23-doc' from 'https://cache.nixos.org'...
copying path '/nix/store/k8hmyxw736p585mkvd28p4wfh2gs3il7-adv_cmds-osx-10.5.8-locale' from 'https://cache.nixos.org'...
copying path '/nix/store/vxdyd32056p8jnmvj6hgl87g9crhv576-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/8gwc4dwxhf9vxiqfagrpwk4nip0mng09-jq-1.6-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/k4p65gnbzhy97y62blmn83aycj698hph-python3.7-PyYAML-3.13' from 'https://cache.nixos.org'...
copying path '/nix/store/fs712dq71wz98vv2140wmdmgi1md14gn-python3.7-xmltodict-0.12.0' from 'https://cache.nixos.org'...
copying path '/nix/store/4ixxl0jdcq0zz7xbzmrqg2cpg1wpsi63-stdenv-darwin' from 'https://cache.nixos.org'...
copying path '/nix/store/kjjsrih4093869f2mpf96msmgic74zip-yq-2.7.2' from 'https://cache.nixos.org'...

# now they're all installed, and we're in the nix-shell (bash) with
# all of them explicitly added to our path
nix-shell$ which yq
/nix/store/kjjsrih4093869f2mpf96msmgic74zip-yq-2.7.2/bin/yq

nix-shell$ python -c 'import xmltodict, yaml; print(xmltodict); print(yaml)'
<module 'xmltodict' from '/nix/store/fs712dq71wz98vv2140wmdmgi1md14gn-python3.7-xmltodict-0.12.0/lib/python3.7/site-packages/xmltodict.py'>
<module 'yaml' from '/nix/store/k4p65gnbzhy97y62blmn83aycj698hph-python3.7-PyYAML-3.13/lib/python3.7/site-packages/yaml/__init__.py'>

nix-shell$ which jq
/nix/store/5wyqzbviby0x89kndhraia40k7rnsabl-jq-1.6-bin/bin/jq
# jq is now explicitly on the path, not coming from my profile
# it's the same version of jq in my case, but it wouldn't be a problem 
# if it was another one! (even if i had installed god-knows-what 
# version of jq via apt or pacman or homebrew or built from source)

# leave the nix shell
nix-shell$ exit

# back to my profile's jq
$ which jq
/run/current-system/sw/bin/jq

# other dependencies no longer on path
$ which yq
$ python -c 'import xmltodict, yaml; print(xmltodict); print(yaml)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'xmltodict'

# but they aren't "removed" ...
$ ls /nix/store/5wyqzbviby0x89kndhraia40k7rnsabl-jq-1.6-bin/bin/jq /nix/store/kjjsrih4093869f2mpf96msmgic74zip-yq-2.7.2/bin/yq /nix/store/fs712dq71wz98vv2140wmdmgi1md14gn-python3.7-xmltodict-0.12.0/lib/python3.7/site-packages/xmltodict.py /nix/store/k4p65gnbzhy97y62blmn83aycj698hph-python3.7-PyYAML-3.13/lib/python3.7/site-packages/yaml/__init__.py
/nix/store/5wyqzbviby0x89kndhraia40k7rnsabl-jq-1.6-bin/bin/jq
/nix/store/fs712dq71wz98vv2140wmdmgi1md14gn-python3.7-xmltodict-0.12.0/lib/python3.7/site-packages/xmltodict.py
/nix/store/k4p65gnbzhy97y62blmn83aycj698hph-python3.7-PyYAML-3.13/lib/python3.7/site-packages/yaml/__init__.py
/nix/store/kjjsrih4093869f2mpf96msmgic74zip-yq-2.7.2/bin/yq

# since they're all still here, we could quickly re-enter the nix-shell without reinstalling them

# but since they're no longer part of the profile, we can also clean them up if we're done
$ nix-collect-garbage
finding garbage collector roots...
removing stale temporary roots file '/nix/var/nix/temproots/26555'
deleting garbage...
deleting '/nix/store/kjjsrih4093869f2mpf96msmgic74zip-yq-2.7.2'
deleting '/nix/store/k4p65gnbzhy97y62blmn83aycj698hph-python3.7-PyYAML-3.13'
deleting '/nix/store/g2jpyfnpjcsnn29xwnqwicxhd12irhzp-shell.drv'
deleting '/nix/store/0pblnwnqhsihhh7rvvp37lwgmz358n3w-yq-2.7.2.drv'
deleting '/nix/store/02940281vja2mamqhnr67p3jrmzhaa3k-python3.7-PyYAML-3.13.drv'
deleting '/nix/store/ha8knhkjxvrcz0vd8bqh6hykg8ifbjy0-PyYAML-3.13.tar.gz.drv'
deleting '/nix/store/vxdyd32056p8jnmvj6hgl87g9crhv576-bash-interactive-4.4-p23-dev'
deleting '/nix/store/jaki986ff0y9b7wlks6irjbzrjvmq2bj-yq-2.7.2.tar.gz.drv'
deleting '/nix/store/4ixxl0jdcq0zz7xbzmrqg2cpg1wpsi63-stdenv-darwin'
deleting '/nix/store/8gwc4dwxhf9vxiqfagrpwk4nip0mng09-jq-1.6-dev'
deleting '/nix/store/w6mns4vjn799r1yfyxzgmdng50590nim-python3.7-xmltodict-0.12.0.drv'
deleting '/nix/store/hyfw94hd4maxix36v6j4v1cs1yb8jkxp-xmltodict-0.12.0.tar.gz.drv'
deleting '/nix/store/d7pi7sg02ywhwg3cyh75fkz9y4v6lgv3-bash-interactive-4.4-p23-doc'
deleting '/nix/store/fs712dq71wz98vv2140wmdmgi1md14gn-python3.7-xmltodict-0.12.0'
deleting '/nix/store/k8hmyxw736p585mkvd28p4wfh2gs3il7-adv_cmds-osx-10.5.8-locale'
deleting '/nix/store/trash'
deleting unused links...
note: currently hard linking saves 0.00 MiB
15 store paths deleted, 5.23 MiB freed

# everything installed by the shell is gone (but jq, from my profile, is still present)
$ ls /nix/store/5wyqzbviby0x89kndhraia40k7rnsabl-jq-1.6-bin/bin/jq /nix/store/kjjsrih4093869f2mpf96msmgic74zip-yq-2.7.2/bin/yq /nix/store/fs712dq71wz98vv2140wmdmgi1md14gn-python3.7-xmltodict-0.12.0/lib/python3.7/site-packages/xmltodict.py /nix/store/k4p65gnbzhy97y62blmn83aycj698hph-python3.7-PyYAML-3.13/lib/python3.7/site-packages/yaml/__init__.py
ls: cannot access '/nix/store/kjjsrih4093869f2mpf96msmgic74zip-yq-2.7.2/bin/yq': No such file or directory
ls: cannot access '/nix/store/fs712dq71wz98vv2140wmdmgi1md14gn-python3.7-xmltodict-0.12.0/lib/python3.7/site-packages/xmltodict.py': No such file or directory
ls: cannot access '/nix/store/k4p65gnbzhy97y62blmn83aycj698hph-python3.7-PyYAML-3.13/lib/python3.7/site-packages/yaml/__init__.py': No such file or directory
/nix/store/5wyqzbviby0x89kndhraia40k7rnsabl-jq-1.6-bin/bin/jq

This makes it pretty simple to try out software or use rarely-touched utilities in a fairly zero-install-ish kind of way. It won't clash with anything else on the system (but won't duplicate dependencies I already have) and I can either explicitly collect garbage immediately, schedule it, or do it whenever the mood strikes me.

---

I'm slowly building out a history-related module for myself and there's some overlap between what I'd like it to track and what my profile already checks as part of building my prompt. In the really short run, I'm just trying abstract the overlap out into something both the prompt and history stuff can share without much entanglement. I wrote a minimal array-driven hook implementation (with a hard-coded array for each hook) that mostly met this need for me, but I've also got an eye on eventually releasing it and am thinking ahead a little about how to structure the API to keep implementation details from leaking and what the extensibility model would look like.

I stumbled on bashup/events deep into a largely fruitless review/search for simple low-overhead bash modularization, plugin, and hook patterns. I wasn't really expecting to find something effectively drop-in like bashup/events at this point. A lot of bash is written too defensively to square with my goals, and I've done enough looking that I'm honestly not quite sure how I hadn't already found it. I was really just trying to see if I could find a good example of someone else handling an arbitrary number of subscribers to arbitrarily-named hooks/events.

FWIW, my opinion so far is that bashup/events is criminally under-appreciated.

[pjeby]

A lot of bash is written so defensively that it's trying to be generic sh code. But that's because few people actually want to really write bash code in the first place, so it's not that surprising.

As far as nix goes, though, given that my current uses for .devkit are either 1) building/testing bashup's own shell tools, and 2) running doco and mantle projects under docker, nix would be an unnecessary complication in either use case. Running web apps under docker isn't just about dependencies, but privilege separation and defense-in-depth (especially for anything PHP-based).

On a broader scope, .devkit is a tool for doing polyglot "scripts to rule them all", such that building in any dependencies other than git+bash+curl (and maybe direnv) for the core functionality isn't really productive. That being said, I don't have any issues with supporting, say, dk use: nix to support detection or local install of nix packages within the .deps environment. Indeed it sounds useful to have something like that, but I lack the bandwidth right now to even investigate whether fully local installs (of nix itself) are possible.

If you want to submit such a module for inclusion, you certainly can, though I'd note that you can also just make a .devkit-modules directory in your own projects' root and place a nix file there to do whatever you need. Likewise, you can edit .envrc to export additional variables the tools would need. .devkit's mission is "check out your project and do stuff with it on a typical nix box", so until/unless nix is part of a "typical nix box", it would have to be locally installable/bootstrappable by a regular user to run essentially sandboxed within the .deps of a checked-out .devkit project.

[pjeby]

Update: I took a quick google at how nix is installed and how people do local installs, etc., and I'm pretty sure at this point that nix is pretty much incompatible with how .devkit wants to work, philosophically.

.devkit wants to be, "you can just delete .deps to start over, or wipe out the whole checkout, with no effect on the larger system". Its raison d'etre is isolation and the use of localized dependency installs like virtualenv, node_modules, composer vendor/, etc. Yes, you might need to globally install some tools for a specific project, but simply building or using the project shouldn't inject things into global state.

Nix, OTOH, has an absolute path system for dependencies... which right away makes it impractical since you can't just move a project's checkout around or add/remove such projects without affecting global system state.

So, while there's still some room for having ways to bootstrap global installs of dependencies (e.g. adding boot event handlers that suggest nix commands to install the dependencies), I don't think it'll ever be blessed as a way of directly managing project dependencies, since e.g. proot isn't portable to say, OS X, meaning you can't even get a local install that way. IOW, nix currently requires either global state mutation or lack of portability, which run counter to .devkit's core aims.