darvid
commented
3 years ago Thanks for the detailed write-up. I added a environment.yml in #2 with poetry2conda. The workflow will look like:
$ conda env create -f environment.yml
$ conda develop .
$ conda install -c anaconda ipykernel jupyterI'm not quite sure how to add jupyter/ipykernel as persistent dependencies in the environment.yml without including them in the pyproject.toml, which I'm not sure we want to do (as main deps, it might make sense to add them as dev deps). As you mentioned, adding the current project as a development package requires some hacks, I feel like just documenting conda develop . is cleaner, but I don't have strong opinions.
emirkmo
Background:
As this seems to be a mix of data science and software development project, there are some things to think about. Conda as a package and python virtual environment manager is ubiquitous in data science, sometimes to the consternation of Python programmers. Conda will just work with traditional (and admittedly janky) setuptools based package requirements. It can also use an environment.yml file specifically for conda, or even just a simple pip requirements file.
Description of the problem:
I went down a rabbit hole chasing down getting poetry to work using my conda based python installation, and setting up poetry inside of a conda based (python) virtual environment. The official docs of poetry recommend a base installation separate of package managers like conda (https://python-poetry.org/docs/#alternative-installation-methods-not-recommended). Documentation is lacking in how poetry interacts with conda virtual environments (https://github.com/python-poetry/poetry/issues/1724). Poetry devs decided to reject even considering conda compatibility and closed the issue (https://github.com/python-poetry/poetry/issues/105). There are some scripts for taking a poetry based pyproject file and converting it to a conda one (https://github.com/dojeda/poetry2conda), however, they are unmaintained, and require individually reviewing packages, knowing whether their Pypi or conda names are used, and editing the pyproject.toml file to point to the correct package manager, source, names, etc, possibly for many dependencies. Poetry also requires a workaround to install a package in editable mode, which is sometimes used for development work.
As a result of this, I spent more than half an hour trying to simply create a virtual environment that I could work on wraeblast with.
Issue:
Wraeblast is difficult to get running for people using conda, without possibly messing up your development environment (as several documented issues on poetry's github page shows). There are issues with poetry's own dependencies bleeding into the virtual environment or even your root/base python environment. This is caused by a lack of providing "the old way" of packaging python software, as existing workflows were configured to work from this (admittedly somewhat flawed) system.
Suggestion:
Provide and maintain either a traditional setup.py, an alternative requirements.txt, or even better, a conda environment.yml file for easily starting to work on wraeblast. It would have taken me about 20 seconds instead of more than half an hour, and especially as you said on reddit, if you want some data scientist type people taking a look at the package, it would make the process of contributing much more accessible for that community.
Conclusion:
Please add a non-poetry way of dependency tracking that works with existing workflows prevalent in other communities, or at least a guide on how to get this running using poetry and the recent PEP suggested pyproject.toml files.