Add compile-requirements command

[cole]

Relates to: https://github.com/archivematica/Issues/issues/765

We've been having some consistency issues running pip-compile; this change hopefully addresses them via docker standardization:

• avoids platform specific additions/removals (seen with the appnope package)
• runs a consistent version of pip-compile (3.7.0, tagged in dev.in files)
• uses a consistent version of python

To test, checkout this branch, make a change to a .in file, and run make compile-requirements

[scollazo]

We don't use docker in production deployments, is there any way to do the same on an ansible deployed environment?

[cole]

@scollazo we shouldn't need to run any of these commands as part of deploying / in production — the output of pip-compile is checked in to source control, in the same .txt files that have always been there, so we can just reference that.

[ross-spencer]

Ah, @cole @scollazo that makes sense now. We should get this into our process once we've stabilized our approach. @scollazo would you be concerned for folks who might not work with the docker-compose development environment? That seems like a potential still... but I guess we can document the requirements to update the requirements?? (🤯🙂)

[ross-spencer]

Oh, I'll give this branch a whirl too under the Python 3 work.

[cole]

@sevein has pointed out this causes come permissions issues on linux (the requirements files are owned by root). I'm not sure how best to solve this yet.

[sevein]

@sevein has pointed out this causes come permissions issues on linux (the requirements files are owned by root). I'm not sure how best to solve this yet.

Perhaps a note in the README is enough. I can easily revert the permission changes so it's not a big deal, e.g.:

find ../src -user root -path "*/requirements/*" -exec sudo chmod 644 '{}' +

[ross-spencer]

I'm not even getting that far... (just logging it here while I investigate it) (all branches are default qa branches)

compose$ sudo make compile-requirements
docker-compose run --workdir /src/MCPServer/requirements --volume "tmpfs:/pip-cache:rw" -e XDG_CACHE_HOME=/pip-cache --rm  --no-deps --user=root --entrypoint make archivematica-mcp-server all
pip-compile -v  --output-file base.txt base.in
Traceback (most recent call last):
  File "/usr/local/bin/pip-compile", line 10, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/piptools/scripts/compile.py", line 261, in cli
    repository = PyPIRepository(pip_options, session, build_isolation)
  File "/usr/local/lib/python2.7/site-packages/piptools/repositories/pypi.py", line 93, in __init__
    self.finder = PackageFinder(**finder_kwargs)
TypeError: __init__() got an unexpected keyword argument 'index_urls'
Makefile:11: recipe for target 'base.txt' failed
make: *** [base.txt] Error 1
Makefile:115: recipe for target 'compile-requirements' failed
make: *** [compile-requirements] Error 2

Seems to be related to: https://github.com/jazzband/pip-tools/issues/856

~/git/artefactual-labs/am/compose$ sudo docker-compose exec --user=root archivematica-mcp-server bash
root@d53d86849b61:/# pip --version
pip 19.2.1 from /usr/local/lib/python2.7/site-packages/pip (python 2.7)

[cole]

I think upgrading to pip-tools 4.0.0 should fix that then.