Dependency management for apps

[nikochiko]

Right now,

1. Dependencies are not taken from the app's requirements.txt automatically
2. If versions are not correctly specified in the requirements.txt file, apps will crash (they would crash even if it wasn't on hamr, but this happened before and we should take this case into accout while building for logs or writing documentation).
3. There is no way to solve for version conflicts between different apps

Proposed solutions:

Separate site-packages/ for each app

We could install the dependencies for each app into a site-packages folder, and insert it into sys.path before getting the app.

Pros:

• Simple to implement
• Not much engineering required, only a simple pip install --target <dir>

Cons:

• There would be too much duplication of dependencies involved between apps. Even separate deployments of the same app would have copies of dependencies.

Single warehouse for dependencies, with symlinks to each app's site packages

This solves for the storage space problem in the eariler solution by keeping all python dependencies and their versions in a single location, and then symlinking specific versions into each app's site-packages directory.

Pros:

• Saves space

Cons:

• Might take some engineering effort to 1) arrange package directories into name/versions and 2) reading from requirements.txt and getting a suitable version (upgrading packages sometimes)

Notes

I think we could go with the first one for now, and then later when we start running out of storage, we can optimize.

This change would also affect the deployment flow. We would need to decide whether the POST request for deployment is kept alive while all the dependencies are installed, or if we give users a link to follow the deployment logs and status.

[anandology]

Separate site-packages/ for each app

Why do you want to reinvent virtualenv?

This solves for the storage space problem...

When did storage space became a problem?

Also, hamr is not-only for python. We shouldn't try to build new abstractions for deploying python apps.

The problem statement could probably be:

Provide a way to manage application dependencies easily. That could work with any language. We may provide a default behavior for popular languages, but that doesn't stop anyone from installing their own dependencies.

Also, consider the following cases and see what does it take to deploy them on hamr.

• a static website built with Jekyll
• a static website built with hugo
• docs with mkdocs
• an app that requires installing a binary (for example: graphviz-api)

[anandology]

Also, read about https://www.heroku.com/elements/buildpacks

[nikochiko]

consider the following cases and see what does it take to deploy them on hamr

OK, that's a good angle to look at it. Thanks!