Retire HomerBot and improve our private requirements installation

[OmarIthawi]

The Problem

We can't use the same SSH key for checking out multiple private git repositories on GitHub. Therefore we don't know how make Open edX install private git repositories via pip in a clean way.

The current method of requirements is to make the whole edX Platform installation to use a HomerBot key which needs decent amount of manual work and pinging others for something that should be simple.

Proposed Solution

I propose making a role for installing requirements for the Open edX, or pretty much any virtual env.

Technical Details

So our solution is to first clone those requirements into a directory, then instruct pip to get that directory.

The role configs would look like the following:

PRIVATE_PIP_REQUIREMENTS:
  - pip_bin: /edx/app/edxapp/venvs/edxapp/bin/pip
    chown:
      user: edxapp
      group: edxapp
    egg: The-Forbidden-Kingdom
    url: git@github.com:appsembler/the-forbidden-kingdom.git
    version: release-0.3.1
    git_identity: "{{ SECRET_FORBIDDEN_KINGDOM_PROJECT_SSH_KEY }}"

  - pip_bin: /edx/app/edxapp/venvs/edxapp/bin/pip
    chown:
      user: edxapp
      group: edxapp
    egg: secret_sauce
    url: git@github.com:appsembler/secret-sauce.git
    version: x.y.z
    git_identity: "{{ SECRET_SAUCE_GIT_IDENTITY }}"

  - pip_bin: /edx/app/ecommerce/venvs/ecommerce/bin/pip
    chown:
      user: ecommerce
      group: ecommerce
    egg: matej_theme
    url: git@github.com:appsembler/matej-theme.git
    version: a.b.c
    git_identity: "{{ SECRET_MATEJ_THEME_GIT_IDENTITY }}"

The role would roughly do the following:

tasks:
 - name: Create special private requirements directories

 - name: Clone the repos
   git:
    repo: url
    key_file: ...
    dest: ...
   with_items: "{{ PRIVATE_PIP_REQUIREMENTS }}"

 - name: Chown and fix permissions

 - name: Install the requirements
   virtualenv: "{{ item.virtualenv }}"
   name: "path/to/private/dir"
   editable: true
   with_items: "{{ PRIVATE_PIP_REQUIREMENTS }}"

Let's retire @HomerBot!

[OmarIthawi]

@thraxil @johnbaldwin FYI.

[johnbaldwin]

@OmarIthawi Looks very reasonable. We can then use repo deploy keys and store the required keys in each edx-config installation's secrets.yml file

I would LOVE to no longer have to use homerbot, too!

Thanks for writing this up!

[OmarIthawi]

I no longer think of this as an issue since we don't overuse privage git requirements.