Consider using poetry

[iron3oxide]

The Makefile/setup.py approach is a bit outdated and having three separate requirements.txt files adds clutter. I would recommend to use poetry instead. It can, amongst other things:

• manage and create virtual environments for your project
• easily add dependencies to all relevant files and automatically resolve dependency conflicts
• mark certain dependencies as test, dev or any group you want

It would probably require some minor changes in the CI setup, but CircleCI seems to support it

[clstaudt]

@iron3oxide

My experience working with poetry, especially on macOS with an ARM processor, has included:

• hard to reproduce installation / dependency resolution failures
• installing packages for the wrong architecture
• cryptic errors
• very long running times for dependency resolution
• missing documentation
• trouble installing packages from git repositories
• incompatibility with conda

I have sunk too much time battling with poetry to make this a priority. If someone wants to take this task up and make it work, they may have more success and I am all for it, but we would require thorough testing before switching to poetry.

[iron3oxide]

I see, my experience has been mostly great so far, but then again I don't have a Mac and I don't do data science. Might take this on sometime.

[clstaudt]

My impression: If you require a package that's not on PyPI, e.g. the correct build for your platform from conda-forge, you're out of luck with poetry. And you can't mix poetry and conda like pip and conda.

[iron3oxide]

I had no trouble using local or repository packages, haven't tried conda though. But conda packages shouldn't be a problem in this project luckily

[clstaudt]

Things have improved, but there was a time when getting correct binaries for numpy or pandas was trouble.

I'd love to add some "AI" functionality at a later stage, e.g. parsing text or documents with pytorch or tensorflow models.