search

MycroftAI / mycroft-core

Mycroft Core, the Mycroft Artificial Intelligence platform.
https://mycroft.ai
Apache License 2.0
6.51k stars 1.27k forks source link

CONSIDERATION: upgrade to pipenv (deprecate requirements.txt) #2210

Closed fermulator closed 12 hours ago

fermulator commented 5 years ago

Be clear about the software, hardware and version you are running

How would the project maintainers feel about upgrading the Python package management to use the new-and-amazing pipenv?

I've done this recently (in a professional capacity) on a different project https://github.com/Geotab/mygeotab-python/pull/222, the actions would be similar here and provide all the pipenv benefits.

Try to provide steps that we can use to replicate the Issue

Considerations (off the top of my head)

  1. dev_setup.sh's install_venv() would be affected (instead of manually created the venv, let pipenv manage it)
  2. requirements.txt would be deprecated and replaced with Pipfile and Pipfile.lock
  3. setup.py would get updated too

Be as specific as possible about the expected condition, and the deviation from expected condition.

The benefits of switching to pipenv are vast. The most important one being that we wouldn't have to manage specific versions of dependency packages anymore. The 1st-degree package requirements can be locked to a specific version (along with 2nd-degree deps), but anytime we want we can bump versions and run the CI/CD, the 2nd-degree versions don't need to be hand picked.

Pipenv also has some security checks too that can be added into CI/CD.

Provide log files or other output to help us see the error

N/A

Upload any files to the Issue that will be useful in helping us to investigate

N/A

fermulator commented 5 years ago

(to be clear: i'm offering to do the work :) - just want to get opinions & buy-in before investing the time)

forslund commented 5 years ago

I've never used it in a project myself but I think it looks quite nice. I've held off on trying it out here since there may be a block due to the packaging mechanism we use to create release packages (dh-virtualenv)

It states that it uses the requirements.txt...But maybe a requirements.txt can be generated from a pipfile/lock-file?

fermulator commented 5 years ago

Hm, indeed I'm unfamiliar with dh-virtualenv. At the onset it does not appear to support Pipfile/pipenv. That said, pipenv supports generating a requirements.txt from a Pipfile.lock at any time, (we could just generate it at build time) so it wouldn't be an insurmountable problem.

EDIT: @forslund , where's the code for dh-virtualenv pkg build process? can't find it in the core

kjcole commented 5 years ago

I thought I'd filed an issue on this long ago, but cannot find it now. Perhaps I suggested it in a chat instead. I have been using pipenv for a couple of years now and it has been rather well-behaved... My understanding is that the Python Software Foundation is backing the idea of making Pipfile (and presumably Pipfile.lock) a standard...

forslund commented 5 years ago

@fermulator the packiging repo is still not open. I need to go through it for possible API keys for the upload service.

If you're interested in the mechanics I have an experimental desktop-package's build script here: https://github.com/forslund/mycroft-desktop-deb It's based on the packaging-script used for the mark-1 so you can see most of the workings of it there.

There's a couple of hacks in there and might need a hand or two to build with other pythons than 3.6...

I would be very interested in switching over to pipenv if you think it's possible to generate the requirements.txt let's go for it...we could also simplify things and maintaining a requirements.txt manually if necessary.

fermulator commented 5 years ago

@forslund , absolutely pipenv can generate a requirements.txt at any time for legacy (backwards compatibility) needs, so that wouldn't be any blocker at all.

Ah, if the build is not open source yet that explains why I could not find it ;o.

In terms of multi-python version support, what versions do mycroft support? (Can't find it in the docs in terms of "system requirements")

Open to collaborate/help sometime for sure. (whenever the project deems it time to get rolling' on this)

forslund commented 5 years ago

We currently support Python 3.4-3.7 but will be dropping the Python 3.4 support In the next big release (19.08) since that's at end of life and pip won't be supporting it any longer. And my intention is to checkout what's needed for 3.8 after the release.

forslund commented 12 hours ago

Closing Issue since we're archiving the repo