Use pyproject.toml to manage dependencies

[breuleux]

All dependencies are now defined in pyproject.toml and most are installed with poetry install. Conda dependencies are put in custom sections in the file and you have to use scripts/make_deps.py to generate a requirements.conda file. scripts/install-deps-[g/c]pu.sh will do it for you.

This also moves some scripts into scripts/.

Oh, and PyTorch is upgraded to 1.3.0. Unfortunately there are a few errors with 1.4.0 (to investigate).

[abergeron]

I like the consolidation, but there are several disadvantages for me:

• installing the package locally is harder (pip install -e .)
• most dependencies are now managed through pip instead of conda and we need to mix a virtualenv with a conda env to install
• The only way for a user to install everything is to use the ci install scripts and those make several assumptions that do not apply to random user machines
• The resulting install (even if done manually) mixes packages from conda with a virtualenv which is generally a bad idea.

In my mind those disadvantages are a much bigger deal than having the requirements repeated multiple times. So I am against this PR.

[abergeron]

Maybe this can help: https://github.com/eigentechnologies/cpip

[breuleux]

@abergeron

• installing the package locally is harder (pip install -e .)

poetry install performs a local install into the active conda env or virtualenv. It won't install the conda packages, but neither does pip install -e .

• most dependencies are now managed through pip instead of conda and we need to mix a virtualenv with a conda env to install

Several of our dependencies are not available on conda anyway, so I'm not sure what difference it makes. We already had to use pip. Poetry will install into a conda environment if it is activated.

• The only way for a user to install everything is to use the ci install scripts and those make several assumptions that do not apply to random user machines

Which assumptions? Anyway, perhaps it'd be better to commit the requirements.conda files.

• The resulting install (even if done manually) mixes packages from conda with a virtualenv which is generally a bad idea.

This was already the case, no?

Maybe this can help: https://github.com/eigentechnologies/cpip

Hm, maybe.

[abergeron]

• most dependencies are now managed through pip instead of conda and we need to mix a virtualenv with a conda env to install

Several of our dependencies are not available on conda anyway, so I'm not sure what difference it makes. We already had to use pip. Poetry will install into a conda environment if it is activated.

Yes we had to use pip, but now there is (potentially) a virtualenv on top of that. poetry will create a virtualenv if you are in the base environment of conda

• The only way for a user to install everything is to use the ci install scripts and those make several assumptions that do not apply to random user machines

Which assumptions? Anyway, perhaps it'd be better to commit the requirements.conda files.

Like that you want to install the conda packages in the current env (which might be the base one). I do this, but other people hate it.

• The resulting install (even if done manually) mixes packages from conda with a virtualenv which is generally a bad idea.

This was already the case, no?

Not unless you explicitly create a virtualenv.

Maybe this can help: https://github.com/eigentechnologies/cpip

Hm, maybe.

Also I found this too: https://pypi.org/project/poetry2conda/

It goes a long way for me since I could just generate a conda env file from the poetry (minus the conda requirements). With some wrangling it could be a better alternative than cpip since that seems like an unholy mess of bash functions with very little error checking and doesn't work on mac.

[breuleux]

• most dependencies are now managed through pip instead of conda and we need to mix a virtualenv with a conda env to install

Several of our dependencies are not available on conda anyway, so I'm not sure what difference it makes. We already had to use pip. Poetry will install into a conda environment if it is activated.

Yes we had to use pip, but now there is (potentially) a virtualenv on top of that. poetry will create a virtualenv if you are in the base environment of conda

Okay, true. I kind of feel the base environment should never be used, but it's true that we should still support all workflows whenever possible.

• The only way for a user to install everything is to use the ci install scripts and those make several assumptions that do not apply to random user machines

Which assumptions? Anyway, perhaps it'd be better to commit the requirements.conda files.

Like that you want to install the conda packages in the current env (which might be the base one). I do this, but other people hate it.

I see. It's not too hard to activate the desired target env before running the script, but point taken.

Maybe this can help: https://github.com/eigentechnologies/cpip

Hm, maybe.

Also I found this too: https://pypi.org/project/poetry2conda/

It goes a long way for me since I could just generate a conda env file from the poetry (minus the conda requirements). With some wrangling it could be a better alternative than cpip since that seems like an unholy mess of bash functions with very little error checking and doesn't work on mac.

Okay, that sounds good. I can try messing around with it and see if it generates a good env file.

[abergeron]

Something that could help is that pytorch has pypi packages and it might be possible to wrangle some wheels out of tvm. They have the appearance of working but I suspect it's never tested and it doesn't seem to work when I tried.

If I have some time I can try to get it to work and add a section to the packages repo to build some wheels that are not horrible. I don't know where we would publish them though and I don't want to put them on PyPI since I don't want to be the official maintainer of said wheels for the larger public.

[breuleux]

I think we can install wheels from a GitHub URL (or any URL).

[abergeron]

Just to leave a trace but I also found this: https://direnv.net

It could be useful to automatically activate the relevant environment when entering the project dir.

[abergeron]

There should be a new version of poerty2conda out next Monday (June 1st) that has some modifications that should help us solve all of our issues.

[abergeron]

I did a rebase and some work to make the file pyproject2conda compatible.

There are some package specifications that break poetry (because they are not available on pip), so we would need a solution for that. Those are currently commented out.

[abergeron]

I have no idea what jenkins doesn't like and the certificate is expired so I can't check.

[abergeron]

Now that we have access to jenkins, I fixed the last few problems with the CI so this is ready to go in.