Using Debian packages and/or installing from PIP

[JedMeister]

I notice that you are installing quite a few python packages installed via pip. That's not bad per se, but where ever possible it's preferable to use the Debian packages. (You probably already know that).

I also note that it seems that the Debian virtualenv package isn't working properly for you so you're reinstalling with pip.

Perhaps there is something that you are missing re using the Debian package? Or if it is actually broken, then we should probably just install from pip (and remove it, or comment it out in the plan). FYI pip and apt install packages to different places, so AFAIK installing/reinstalling a package with pip that you have already installed from Debian will actually install a 2nd version (not update the Debian version).

FWIW here's a few I spotted:

https://github.com/DocCyblade/tkl-mayan-edms/blob/development/conf.d/main#L40 pip - already installed (python-pip) wheel - https://packages.debian.org/jessie/python-wheel setuptools - https://packages.debian.org/jessie/python-setuptools (I note that this is a pretty old looking version so perhaps there is a good reason not to use?)

https://github.com/DocCyblade/tkl-mayan-edms/blob/development/conf.d/main#L46 psycopg2 - https://packages.debian.org/jessie/python-psycopg2 (I thought we were already installing that in LAPP/Postgres apps, perhaps not?!) redis - https://packages.debian.org/jessie/python-redis uswgi - https://packages.debian.org/jessie/python-uwsgidecorators (? I think this is the same package, but I could be wrong...)

Bottom line is that if you have a legitimate reason to not use the Debian package version, that's fine. But ideally please document that.

Alon really doesn't like stuff commented out (he prefers it removed instead), but personally I'd be inclined to add the Debian packages to the plan but comment out the ones you aren't using and note they are installed from pip in the conf. You can then elaborate a little on why you're using the pip versions in the conf (where you install them). That will be helpful in the future, particularly when we do a major release (i.e. v15.0).

Having said that I think we are aiming to release v15.0 using the new TKLX container model, so may not be quite as relevant then (but still may be useful...)

[DocCyblade]

I am using the developers deployment instructions to the letter. I think that since we are using an virtualenv(again recommended by developer) all the PIP commands are installing them inside that virtualenv. Speaking of virtualenv, it looks like the version in jessie is v1.11.6 (released 3/16/2014), current version on PyPi s v1.15.2, (released in 8/2016) I dug a little deeper into why the Debian package of virtualenv is broken, and found this It seems if I add the package virtualenv (for Python 3.x) it will fix the need to install from PIP but we are using an older version but I don't see this as a problem yet, i'll have to test it.

On a side note, it seems that these other package installers (PIP, Composer, RubyGems, PPM, Maven and others) are around because their respective environments are being updated and progressing so quickly (thanks to continuous integration) It seems that software devs/maintainers are slow to create full packages for Debian. I can see why due to some of them use so many shared libraries and not all libraries are packaged and/or updated, so why use the OS package manager when the script environments user space has it's own package manager. With that being said, it would be a good idea to extend TKLBAM/TKLDev to not only use OS packages but script specific package managers as well. I can submit an issue in the tracker as a feature request, but I assume that it would be a pretty big re-write. Maybe if TKLBAM/TKLDev could be "pluggable" so plug-ins could be written to support additional package managers and the like. Ok aside complete...

https://github.com/DocCyblade/tkl-mayan-edms/blob/development/conf.d/main#L40 pip - already installed (python-pip) wheel - https://packages.debian.org/jessie/python-wheel setuptools - https://packages.debian.org/jessie/python-setuptools (I note that this is a pretty old looking version so perhaps there is a good reason not to use?)

This was removed, setuptools had a bug in it's last release, this was a quick workaround so I can keep building. It has since been patched and I just pushed the changes.

I'll reach out to the developer for Mayan and see what his take is on getting a Debian package created, but I think he will be in the same boat with all the needed dependancies, creating it and keeping it up to date will be a challenge as all the dependancies will needed to be maintained as well

[ghost]

Virtualenv is an isolation environment for Python. It is like Docker but at the language level (https://www.davidfischer.name/2010/04/why-you-should-be-using-pip-and-virtualenv/). Debian includes a system wide Python interpreter and Python packages that can conflict with a project's dependencies, this is why even when Dockerizing Python projects it's good to use virtualenv. Virtualenv also makes it easier to use Python 2 projects on distributions where Python 3 is the default interpreter and viceversa.

The Python library packages included in the distributions not always match the requirements of a project. The project has been coded using versions of libraries that are known to work well, or for which workarounds have been included in the project. This is another reason not to deviate from the versions specified in the requirements file: https://gitlab.com/mayan-edms/mayan-edms/blob/master/requirements/base.txt

The pip version included in distributions gets outdated fast and it is not uncommon to do a:

pip install -U pip

inside a virtualenv to force pip to upgrade itself.

The reasons for not building a Debian package are as you mention. Some of the required libraries have not been packaged for Debian and we would need to build and maintain packages for those too. We already maintain a Docker image, API client, Fabric installer, among other subprojects: http://www.mayan-edms.com/ecosystem/ It would be good to have, but there is not enough manpower to add a Debian package.

Virtualenv make Python projects behave like containers and makes the distrution a bit less relevant when it comes to dependencies.

If you have more questions about packaging Django projects or why Mayan does the things it does, you can watch the video of the talk I gave on the topic. Explains the logic behind the file paths, initialsetup and autoadmin commands among other things. https://www.youtube.com/watch?v=bkeIE90hl3Q

Looking forward to firing up tkl-mayan-edms! :)

[DocCyblade]

@JedMeister - I started a branch to quickly test using Debian's older packages over PIP, and there are a lot of changes that need to be made and I need to veer off of the instructions from the developers site for deployment. It looks like it can be done, but I will need to use system packages when creating the virtualenv, not sure how this will break when upgrading since not everything that mayan requires will be inside of the virtualenv, and it will probably break the upgrade/migration commands that developer gives in the README but I can't be certain it would just work too.

uwsgi configuration needs to be tweaked, as it's installed with it's own startup scripts. I will work on trying to get it working with the packages just to say it can be done if Alon decides that is the way to go.

[DocCyblade]

@rosarior - You must be watching my repo :-) Thanks for posting and confirming my thoughts. Feel free to post, it's really nice when the developer of the app your trying to "TurnKey" is in on the conversation.

[ghost]

@DocCyblade Yes, I do :) It's interesting to see what others are building with Mayan and lend a hand when possible. Thanks!

[JedMeister]

@rosarior - Nice looking piece of software you built there! And awesome that you are so proactive to drop in to projects like this one Ken's working on to help out and give guidance! :+1:

@DocCyblade - Thanks for the extra info. Sorry to divert your progress. TBH I didn't mean to add a ton of work. If upstream gives those explicit instructions for installing to Debian, then don't feel the need to persevere with this. As you suggest, even if you can get it working, it may well break down the track.

WRT old versions, that's sort of how Debian works. It keeps the versions static within a release. At it's repos provide a ton of different software, the idea is that none of that will break because of udated versions. TBH I'm not sure why the packages are that old though. Jessie went into version freeze ~2 years ago, so it's something of a surprise that Virtualenv is 4 years old. Someone must have dropped the ball. Or perhaps there was some rational? FWIW Stretch (the next Debian release) will go into freeze probably next month and it currently has Virtualenv 15.0.3.

The main advantage to using Debian packages is auto security updates. Using packages from pip is quite acceptable. Not ideal, but acceptable. :smile:

Having said that, IMO pip is pretty broken security wise. I suspect that it won't be long until someone leverages it for malicious ends. IMO it should be providing signed packages, it doesn't even provide SHA sums and/or https so you can't even 100% be sure that what you downloaded is what the site says it was...!.

[DocCyblade]

@JedMeister - Your post got me googling and I found this issue on PIPs github tracker regarding this very issue security. So I can see why your shy away from PIP. On a interesting note, from what I have read it seems that once we enter into the virtualenv, code that is run there is sandboxed (probably poor wording) so when I enter into the virtualenv and then upgrade PIP or install packages they are running within that python environment.

If your good at using PIP where it makes sense just close this issue, or if there is something you want to see happen specific, close and open a new issue with a specifc issue regarding PIP v Deb pkg like I did in #49

[JedMeister]

If your good at using PIP where it makes sense just close this issue

Closing :smile:

[ghost]

@JedMeister Thanks! :) I'll do what I can to help.