Identify specific VMs to action on from Semaphore GUI

[uwardlaw]

• [ ] We need instructors to be able to enter a full VM name like [u or c]/class21012/student05/windowsDC or 05-windowsDC or some combination of numbers and make is accessible in ansible playbooks when a task template is run
• [ ] Begin building Semaphore from source project
• [ ] Add an extra arguments box at task launch. It's a requested feature in the semaphore repo
  • Consider fork and submit a PR
• Optimal solution would:
  • Scan through the task template playbook and get a list of requested arguments based on defined interactive input
  • Dynamically create a number of dialog boxes for inputs with a label for the variable per box
  • Not execute unless all variables are filled
  • Pass --extra-vars to the playbook to silence the interactive prompt with the entered variables

vars_prompt example for interactive input

---
- hosts: all
  vars_prompt:

    - name: username
      prompt: What is your username?
      private: no

    - name: password
      prompt: What is your password?

Could look like:

https://github.com/ansible-semaphore/semaphore/blob/develop/web2/src/components/TaskForm.vue

    <v-text-field
        v-model="item.message"
        label="Message (Optional)"
        :disabled="formSaving"
    />

Seems to make the form

And this handles other environment data

<v-text-field
      v-model="item.name"
      label="Environment Name"
      :rules="[v => !!v || 'Name is required']"
      required
      :disabled="formSaving"
      class="mb-4"
    ></v-text-field>

    <codemirror
        :style="{ border: '1px solid lightgray' }"
        v-model="item.json"
        :options="cmOptions"
        placeholder="Enter environment JSON..."
    />

    <v-alert
        dense
        text
        type="info"
        class="mt-4"
    >
      Environment must be valid JSON. You may use the key <code>ENV</code> to pass
      environment variables to ansible-playbook.
      Example:
      <pre style="font-size: 14px;">{
  "var_available_in_playbook_1": 1245,
  "var_available_in_playbook_2": "test",
  "ENV": {
    "VAR1": "Read by lookup('env', 'VAR1')"
  }

[rylagek]

Updated tasks to note that we need to have a clear denotation of CWO vs UCWT classes (C/U)

[rmfirth]

Update: Scanning the playbook for interactive input prompts would be a fairly complex task and would probably be a little too much to be practical right now. Instead, I have forked Semaphore and implemented a solution where a Task Template can specify Dynamic Variables that should be provided at task runtime. Then, when a user goes to run a task, they will be prompted to enter a value for those variables before they are able to run the task.

For Example, we create a Create Class template that probably needs to run with a different Class # and Class Size each time it is run. In the template, we can specify these variables as pictured below:

This will result in the following prompt when using the "Run" button for tasks in the UI:

Thus, by designing a template that requires certain dynamic variables to be provided at runtime, we can design tasks that are easy and straightforward to run with different inputs each time.

Please see my next comment about how we can incorporate these changes into our tech stack

[rmfirth]

I have forked the semaphore repository and created a branch with the implementation described in the previous comment here https://github.com/rmfirth/semaphore/commits/rfirth/cherryPick.

We have a few options moving forward about how to actually use these changes in our Semaphore instance.

1) Wait for the next semaphore release: Assuming my changes are accepted into the semaphore repository (link PR here when created), we would need to wait for a new release of Semaphore, then modify our Dockerfile in images/semaphore to use the new release version instead of v2.8.21, as is specified currently in that file. This is the most straightforward and is probably the best long-term solution, so that we are relying on a vetted and publicly available semaphore release. It also has a number of drawbacks to be aware of: a) Currently, releases don't seem to happen very frequently with Semaphore. As of this writing, the last stable release was in November. b) Semaphore has undergone a few changes since the version we are currently using as of this writing (v2.8.21), and some of these changes will have nontrivial effects on our current setup (for example, I realized that a later release adds some DB encryption to User/Password key stores, which we use for our localhost inventory. This wouldn't be a problem for new Control instances that are spun up, but anyone that already has a machine build that is seeking to upgrade to the new version will find that the NoneKeyLP key will no longer be accessible and they will have to manually create a new one, then switch their localhost inventory to use that key instead. This is fairly easy to accomplish but a good thing to be aware of nonetheless)

2) Create and publish our own Docker image: This would require creating our own repository in Docker Hub, which looks fairly straightforward to do. We could then build our own semaphore docker image, publish it to our repository, then reference that image in our Dockerfile (i.e. we replace this line with something like FROM <OUR_REPOSITORY>/semaphore:latest. Now in order to even create the Docker image, one solution that would minimize the impact of other changes to semaphore since the release we are currently using (v2.8.21) could be something like the following: we could checkout the Semaphore commit associated with v2.8.21, apply my commits directly onto this new branch, then build a new semaphore Docker image (the repository has the files needed for building a new Docker image, and I have succeeded in doing this locally). We would then push the new image to our Docker Hub repository. This could be a good short-term solution if we need to get this functionality ASAP and don't have time to wait for another semaphore release.

I recommend approach 2) for the short-term, with the understanding that in the long-term we would be looking to switch to an official semaphore release that includes my changes.

@rylagek @marissaeinhorn @uwardlaw Let me know if you think there's anything I have failed to consider or if you have any questions.

[rmfirth]

UPDATE: I have included a link to my changes in the previous comment (where it said "insert here", which was in the first paragraph).

NOTE: TL;DR: This not the latest version of semaphore, just v2.8.21 with my changes applied on top. See below if you care about the explanation. Explanation: This particular branch I have linked to has my changes applied directly on top of the last commit that was apart of Semaphore v2.8.21. Originally, I built my solution on top of the current Semaphore development branch, but I didn't want to force the vater project to use other new updates to Semaphore in addition to my own, so this was an effort to minimize the impact of the change. If, for some reason, you want to apply my change on top of a different version of semaphore, all you need to do is cherry pick all my commits onto whatever branch of semaphore you want to build and then build it (assuming merge conflict resolution, etc.)

[rmfirth]

How to build a new docker container for semaphore:

1. Follow the instructions in the semaphore contribution guide to install semaphore for your local dev environment. Note that this includes cloning the original semaphore project (not my fork - we'll get to that later)
2. Once you have gotten your local installation running (i.e. "task compile" works and you are able to get a local server running), change directories into the directory where your newly cloned semaphore repository is if you aren't already there (you should be seeing subdirectories like api, db, web2, etc.)
3. Now you are going to Change the "origin" alias of your local semaphore repository to point to my fork instead of semaphore (you can change it to point back to semaphore later if, for example, you want to check out branches from the semaphore project). If you are unfamiliar with setting the value of the "origin" url, you can run the following command: git remote set-url origin git@github.com:rmfirth/semaphore.git. You can run git remote get-url origin to confirm that it took.
4. Do a git fetch and checkout the rfirth/cherryPick branch.
5. Go to your local copy of Taskfile.yml and change line 96 to have db/**/migrations/* instead of db/migrations/* (Why? See below for an explanation)
6. Run task compile to compile the code.
7. Run context=prod task docker:build
8. If all has gone well up to this point, you should have a new docker image called ansiblesemaphore/semaphore with the tag latest in your local image registry (run docker images) .
9. If you want to push this to a docker repository you have created, you can push this image to that repository (Note: you may have to change the docker namespace of your image (the part before the slash in the name, i.e. ansiblesemaphore to be the namespace of your repository. I don't know for sure as I have never had to create a docker repository and push images there). If you don't want to do that and you just want to run this new image in your local vater instance, you can change this line in your local copy of vater to point to ansiblesemaphore/semaphore:latest (or whatever your image is called, if you renamed it). Since your local docker daemon has an image called ansiblesemaphore/semaphore:latest, it should use your image, not the actual latest docker semaphore image. If you want to be sure, you could change the tag of your local docker image to something unique like my-super-unique-tag and then have your local copy of vater point to ansiblesemaphore/semaphore:my-super-unique-tag.
10. At this point, you can restart the semaphore service using vater and it should rebuild the service with the new image. The changes I added should now be reflected in your local version.

Note (If you care, an explanation of step 5): Without this change, when you run task build, your task program won't know to look for the database migration file I added, which is critical for the the feature to work properly. I believe there is a mistake in the current version of the Taskfile as it does not correctly point to the database migration folder as a dependency of the compilation steps.

[rmfirth]

ONE MORE THING:

It's worth noting that the semaphore maintainer beat me to in including the new feature (https://github.com/ansible-semaphore/semaphore/releases/tag/v2.8.35) before I could submit a PR. His version doesn't have the user-friendly UI but it at least is part of an actual release that might be coming out soon (if you are okay with the other changes that have been made to semaphore since v2.8.21)

[uwardlaw]

We ran into a few errors we'll try to reproduce, but this was history for a successful creation:

 2057  mkdir sem-build
 2060  cd sem-build/
 2063  go version
 2067  sudo rm -rf /usr/local/go && tar -C /usr/local -xzf go1.17.6.linux-amd64.tar.gz
 2068  sudo tar -C /usr/local -xzf go1.17.6.linux-amd64.tar.gz
 2071  mkdir -p src/github.com/ansible-semaphore
 2081  cd src/github.com/ansible-semaphore/semaphore/
 2074  git clone --recursive https://github.com/ansible-semaphore/semaphore.git && cd semaphore
 2083  go install github.com/go-task/task/v3/cmd/task@latest
 2084  task deps
 2115  sudo snap install task --classic
 2122  task deps
 2124  sudo apt install npm
 2125  task deps
 2126  task compile
 2141  go run cli/main.go setup
 2142  go run cli/main.go service --config ./config.json
 2147  git remote set-url origin https://github.com/rmfirth/semaphore.git
 2148  git fetch
 2154  rm web2/package-lock.json 
 2155  git checkout rfirth/cherryPick
 2157  nano Taskfile.yml 
 2158  task compile
 2159  context=prod task docker:build
 2160  sudo context=prod task docker:build
 2177  nano ~/vater/control-services/images/semaphore/Dockerfile 
 2178  vater restart

[uwardlaw]

Note. should change the tag from latest to local to make the different between upstream and the build more obvious

[rmfirth]

How to Create a task template with my changes, via the API:

use the same POST /project/{project_id}/templates endpoint already in the documentation, but add

dynamic_vars: string to the model.

Specifically, dynamic_vars must be a serialized JSON array where each item in the array is an object with the following model:

{
  "name": string, // the name of the variable
  "default": string, // a default value for the variable (this value is not required for the program to work)
  "required": boolean // true if you want the user to be required to input the variable in order to run a task
}

So, for example, if you wanted to create a task template that, when run as a task, will require the user to enter two values: class and classSize, the API call would look like the following:

POST /project/{project_id}/templates

{
  "project_id": 1,
  "inventory_id": 1,
// ... etc
  "dynamic_vars": "[{\\"name\\":\\"class\\",\\"required\\":true},{\\"name\\":\\"classSize\\",\\"required\\":true}]"
}

If you were to do a JSON.parse on that dynamic_vars string, you'd find it decodes into an array that looks like this:

[
    {
        name: 'class',
        required: true
    },
    {
        name: 'classSize',
        required: true
    }
];

========================================

How to Run a new task with my changes with dynamic variables, via the API: use the same POST /project/{project_id}/tasks endponit already in the documentation, but you can add:

dynamic_vars: string to the model.

Specifically, dynamic_vars must be a serialized JSON object of key-value pairs, where the key is the name of the variable (e.g. "classSize") and the value is the value for that variable. For example:

POST /project/{project_id}/tasks

{
  "template_id": 1,
  "dynamic_vars": "{\\"class\\":\\"21012\\",\\"classSize\\":\\"15\\"}"
}