Improve vendoring of dependencies for CKAN apps

User Story

In order to [goal], [stakeholder] wants [change].

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

[ ] GIVEN [a contextual precondition] \ [AND optionally another precondition] \ WHEN [a triggering event] happens \ THEN [a verifiable outcome] \ [AND optionally another verifiable outcome]

Background

https://github.com/GSA/datagov-deploy/issues/2603

Cloud Foundry buildpaks will try to install dependencies from requirements.txt. However, the Cloud Foundry staging environment has a minimal development setup and might not be able to build all dependencies.

In the past, this affected packages like M2Crypto (required swig). This will likely be an issue for cryptography which is now requiring Rust as a build dependency.

This already has some play into https://github.com/GSA/datagov-deploy/issues/2785. If we were vendoring correctly, the plugin should be installed properly.

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch

Remove -e from requirements.txt, consider using poetry to manage development dependencies (-e) separate from production dependencies (no -e) while being able to resolve dependency conflicts better than multiple requirements-*.txt files.

Most of our vendoring issues come from the fact we're using the -e editable flag in our requirements.txt:

   2021-02-11T15:28:33.28-0800 [STG/0] OUT          Running setup.py develop for ckanext-datajson
   2021-02-11T15:28:33.43-0800 [STG/0] ERR            ERROR: Command errored out with exit status 1:
   2021-02-11T15:28:33.43-0800 [STG/0] ERR             command: /tmp/contents847760532/deps/0/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/contents847760532/deps/0/src/ckanext-datajson/setup.py'"'"'; __file__='"'"'/tmp/contents847760532/deps/0/src/ckanext-datajson/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
   2021-02-11T15:28:33.43-0800 [STG/0] ERR                 cwd: /tmp/contents847760532/deps/0/src/ckanext-datajson/
   2021-02-11T15:28:33.43-0800 [STG/0] ERR            Complete output (2 lines):
   2021-02-11T15:28:33.43-0800 [STG/0] ERR            running develop
   2021-02-11T15:28:33.43-0800 [STG/0] ERR            error: error in /home/vcap/.pydistutils.cfg: command 'develop' has no such option 'allows_hosts'
   2021-02-11T15:28:33.43-0800 [STG/0] ERR            ----------------------------------------
   2021-02-11T15:28:33.88-0800 [STG/0] ERR        ERROR: Command errored out with exit status 1: /tmp/contents847760532/deps/0/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/contents847760532/deps/0/src/ckanext-datajson/setup.py'"'"'; __file__='"'"'/tmp/contents847760532/deps/0/src/ckanext-datajson/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.
   2021-02-11T15:28:33.90-0800 [STG/0] OUT        Running pip install without indexes failed. Not all dependencies were vendored. Trying again with indexes.

GSA / data.gov