Use poetry

[caldwellshane]

Here's a proposed retooling to Poetry.

[caldwellshane]

Is it possible along with poetry.lock to also have it generate & include a fallback requirements.txt? I understand the advantages of poetry in theory but never really used it myself and didn't find it too smooth on first trying to use it. Especially for the currently only 3 dependencies, I feel like pip should also be able to work fine.

Yep, definitely. We can use poetry export to convert the poetry.lock to a requirements.lock. (I'd recommend against requirements.txt which suggests it's a hand-written, pip-resolvable list of primary dependencies. Pip's dependency resolution is a disaster at the moment, and last I checked was not on track to get better.) This would/should be a strict duplicate of the deps in poetry.lock if we want deterministic builds. In this case I guess we'd also want to keep a legacy setup.py around per this, and two sets of instructions.

I'll hold off on this for the moment, since I'm not sure it's the best idea. (setup.py is deprecated in favor of pyproject.toml, and this would duplicate both the requirements and the build workflow.) Asking a Python developer to install Poetry these days is pretty common afaict. I've messed around with making this work using the shim suggested here, and I get a traceback that makes no sense but seems true to form for Pip these days:

(vv) ➜  ivfit git:(use-poetry) ✗ pip install --upgrade pip       
Collecting pip
  Using cached https://files.pythonhosted.org/packages/fe/ef/60d7ba03b5c442309ef42e7d69959f73aacccd0d86008362a681c4698e83/pip-21.0.1-py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-21.0.1
(vv) ➜  ivfit git:(use-poetry) ✗ pip install -e .         
Obtaining file:///Users/shane/repos/ivfit
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: panel<0.12.0,>=0.11.3 in ./vv/lib/python3.7/site-packages (from ivfit==0.1.0) (0.11.3)
...
Requirement already satisfied: certifi>=2017.4.17 in ./vv/lib/python3.7/site-packages (from requests->panel<0.12.0,>=0.11.3->ivfit==0.1.0) (2020.12.5)
Installing collected packages: ivfit
  Running setup.py develop for ivfit
    ERROR: Command errored out with exit status 1:
     command: /Users/shane/repos/ivfit/vv/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/shane/repos/ivfit/setup.py'"'"'; __file__='"'"'/Users/shane/repos/ivfit/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
         cwd: /Users/shane/repos/ivfit/
    Complete output (3 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'setuptools'
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/shane/repos/ivfit/vv/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/shane/repos/ivfit/setup.py'"'"'; __file__='"'"'/Users/shane/repos/ivfit/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.
(vv) ➜  ivfit git:(use-poetry) ✗ python
Python 3.7.7 (default, Nov 27 2020, 19:48:45) 
[Clang 12.0.0 (clang-1200.0.26.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import setuptools
>>> quit()

I'm curious what wasn't smooth? Poetry is renowned as a Python build system for being easy, minimal, and independent of local Python, so it might just be a matter of giving a little Poetry help in the README.

[nitrocalcite]

Hmm, another thing recommended in that thread was to have pip read pyproject.toml with python -m pip install ., which works for me locally on Windows anyway, removing the requirement for the setup.py shim. I'd prefer not to solely depend on poetry for source install with no alternatives for relatively new Python users, as the "typical user" in my head is researchers more than developers. Since either this approach or the requirements.lock approach should work, I'm happy to merge this.

[caldwellshane]

Do you mean typical user, or typical developer? The local build workflow is mainly for a developer who is going to contribute code, and this person should not expect that each project they use contains a tutorial on Pip, Conda, Poetry, homebrew, apt, Windows, etc etc. They want their local build to agree exactly with what runs in CI and gets published or deployed. By contrast, once we publish the package (which Poetry makes more ergonomic) a user will just pip install ivfit into whatever build system they want.

Using pip to resolve the dependencies from pyproject.toml is likely to create a build that is inconsistent with poetry.lock, which is not good. The dependency resolver of Poetry is really good and is not matched by Pip or Conda last I knew.

[nitrocalcite]

Yeah, apologies, I conflated the two processes in my head too much. I understand what you're saying much better now that I went through the poetry install process for this repo - it isn't too much overhead at all. Thanks a lot for bearing with me on this!

[caldwellshane]

No worries. And I can't help but feel this is obligatory: https://xkcd.com/1987/