Package management through Nix

[Zh0uEnlai]

Hi all,

I'm hoping to unify our efforts across Windows, MacOS, and Linux by using Nix.

Changes

Minor

• Moved files in ./bot to ./src/bot
  • This mimics the expected python package structure for setuptools
• Added setup.py and setup.cfg
  • These are standard files for packaging python applications through setuptools.

Major

Package management through Nix

In light of multiple people bringing up issues with Windows vs. Linux vs. Mac packaging conflicts, this PR introduces Nix to live alongside Docker. Introducing Nix allows us to use a single package manager for all three platforms, without necessarily relying on containerization (though always keeping it as an option, especially for distribution). This allows for easy integration with non-Dockerized tools (VScode, anything else in the user environment), and should make it easier for us to help each other, regardless of operating system.

I created two videos for Windows users (links in the docs folder)

For Mac and Linux users, installing Nix is as easy as running

sh <(curl -L https://github.com/numtide/nix-unstable-installer/releases/download/nix-2.4pre20210823_af94b54/install) --daemon

Then running:

sudo echo "# Configuration for the Nix package manager.
#
# To read detailed explanations about these options, please see [1].
# [1] - https://nixos.org/manual/nix/unstable/command-ref/conf-file.html
#
# Lets us use the 'nix' command and 'flakes' [2]
# [2] - https://nixos.wiki/wiki/Flakes
#
experimental-features = nix-command flakes
# Good for development
keep-outputs = true
# The default
keep-derivations = true
" >> /etc/nix/nix.conf

At this point, the application can be built with nix build

And run with ./result/bin/abbot --help

Detailed instructions + step-by-step videos for Windows users are included in this PR under docs.

Development

You can create an isolated shell environment for testing by running nix develop If you're using VSCode, this will also modify the local workspace settings to use your updated python version.

FAQ:

Q: Is Nix the same as Docker? A: Nix is usually used in combination with Docker, as they solve different (but related) problems. Docker handles containerization, Nix handles dependency/system management. I often use NixOS:latest as the base image, but this is a matter of taste and how important it is that you have a small, hermetic image.

Q: How does Nix interact with Docker?

A: This person has a great post: Speedy Development environments with Nix and Docker. There's a more detailed tutorial at nix.dev: https://nix.dev/tutorials/building-and-running-docker-images

Q: Why not pip + pipEnv + virtualEnv + homebrew + chocolatey + snap + apt-get + Yum?

A: I think it would be beneficial for Windows, MacOS, and Linux users to all use the same package management solution. This allows us to all help each other debug, regardless of operating system.

Q: How do I add new dependencies?

A: For this example, let's say you want to add dependencies on numpy and seaborn.

• First, let's check if they exist in nixpkgs already (you can also just search the Nixpkgs github)

  nix search nixpkgs numpy

When you see that it exists, you can now add it to the flake:

For python, under flake.nix > add your dependencies to propagatedBuildInputs, e.g.

# Let's say you want to add numpy and seaborn
propagatedBuildInputs = [
                # Web drivers
                pkgs.chromedriver

                # Python dependencies
                pythonPkgs.certifi
                pythonPkgs.dnspython
                pythonPkgs.idna
                pythonPkgs.requests
                pythonPkgs.requests-toolbelt
                pythonPkgs.urllib3

               # Add them here 
               pythonPkgs.numpy
               pythonPkgs.seaborn
            ];

Now you can use them as you'd expect (as if you'd installed them through PIP)

Enter the nix shell

nix develop

python -c "import numpy"  
python -c "import seaborn"  

This command should succeed, demonstrating that you have a python environment that knows about the dependencies you specified.

[andria-dev]

@Zh0uEnlai Can you bring up some of the issues that people have been having? We only ask people to install Python 3 (includes pip3) and to install the pip packages in requirements.txt.

Nix doesn't seem to offer anything easier. It sounds like a bunch more hoops to jump through. Like you have two steps to install Nix on Linux and macOS and then you still haven't even installed Python or any of the pip packages.

If you can offer fewer steps than installing Python 3 and pip3 install -r requirements.txt for setting up, then I'd be for adding this.

[andria-dev]

I'd also be cautious about adding complex configuration scripts for things like Nix, it adds more potential for people to introduce malicious code into this repository and is also another thing to maintain.

[SeanKilleen]

I'm not sure I see the benefit of Nix in this case. It seems folks will either want a full image (e.g. run it with Docker) or develop in their own env with their specific setup.

Can you elaborate more on what you think the gains will be here?

[SeanKilleen]

Also FYI, moving the location of the source files will break the Dockerfile currently. So we'll need to adapt that too.

[LakesideMiners]

Yeah, Pythons default thingy is fine for this, If we are paranoid about stuff not working on odd setups.(most likely existing python stuff)

There is always venv. After cloning, all the user needs to do is

python3 -m venv somenamehere

Then

on Windows somenamehere\Scripts\activate.bat if on Windows

source somenamehere/bin/activate if on Unix/Linux/MacOS

Where somenamehere is whatever they want, just needs to stay consistent.

Then they do the

pip3 install -r requirements.txt It's how I ran it.

[npvq]

@LakesideMiners Yup. I also added a bit of documentation for using venv on macOS (OS X) and Linux. I'm unfamiliar with the documentation for Windows, hopefully someone could add that too.

[LakesideMiners]

closing as this seems to have stalled and we are using venv.