search

dandi / dandi-hub

Infrastructure and code for the dandihub
https://hub.dandiarchive.org
Other
11 stars 23 forks source link

Investigate how to test the playbook #71

Open asmacdo opened 1 year ago

asmacdo commented 1 year ago

This issue is to

  1. describe the remaining problems with full automation
  2. identify components which should be separated (to simplify testing) ~3. estimate cost for full e2e tests on Amazon~ (This can be done later)
  3. identify what can be tested outside of EC2
asmacdo commented 1 year ago

Testing Plan

Not automated

The only remaining piece that is not fully automatable is the creation of the https cert.

Separate into Roles

The most difficult part of testing this is the setup of EC2. This part has to happen "for real" and we have to pay for the resources. Since this is the first thing that has to happen, it is challenging to test the whole playbook as a unit.

Instead, I propose that we break this into Ansible roles.

  1. EC2 setup (create resources, set permissions) (not yet planned)
  2. Kops setup (create Kubernetes cluster) (see also #81)
  3. Jupyterhub setup (Create Kubernetes resources and call helm chart) (WiP #76)

This can be done iteratively, and it looks like it would be simplest to do in reverse order (3, 2, 1). All of these roles can be published to ansible-galaxy and can be used with composition for various projects (hopefully removing deduplication with kb-hub)

Introducing Ansible Molecule

https://molecule.readthedocs.io/en/latest/

Ansible Molecule is a project specifically designed for testing Ansible roles. Its flexible, supports testing with matrices, and is incorporated directly into the role itself. It also tests for idempotence out of the box, which is very important for these roles.

Next step

Since jupyterhub will be be the simplest part to setup, we could begin to pull those tasks into a separate role. Rather than depend on EC2 and a kops cluster, the molecule prepare stage can setup a kind cluster and run the tests there. We can get value from this immediately by adding this portion to CI and adding k8s versions as they are released to the test matrix (so we aren't surprised later).

Risks

I'm confident jupyterhub can be pulled into a generally useful role, perhaps jupyterhub would be interested in participating in this. Setting up Kops will probably be useful for this as well, given that the zero-to-jupyterhub is an effort they have worked on. The EC2 parts can be pulled into a role, but may be difficult to make this useful for anyone other than us. It will also be the most expensive part to test.

asmacdo commented 1 year ago

When these roles are published to ansible-galaxy, they could all be packaged in the same collection so the playbook would look like:

---
- name: z2jh
  hosts: all
  collections:
    - dandi.z2jh
  tasks:
    - name: Setup EC2
      import_role:
        name: ec2  # from z2jh collection

    - name: Setup Kubernetes
      import_role:
        name: kops  # from z2jh collection

    - name: Setup Jupyterhub
      import_role:
        name: jupyterhub  # from z2jh collection

Importantly, these roles can share variables, set their own defaults, be tested separately, and can be shared.

asmacdo commented 1 year ago

Jupyterhub role: (WIP at this time) https://github.com/dandi/dandi-hub/pull/76