Outstanding development tasks and issues for Dojo-CLI

[GoogleSheets]

Tasks

• Allow user to specify model image version.
• Move docker server url (on linux this is unix://var/run/docker.sock) to configuration file.
• Check model container status.
• Get model (docker container) logs.
• Get results path.
  • runmodel could probably return this and print it in the console after the run.

Issues

• Docker Hub image download needs progress bar implementation.
• The specified results folder has root:root ownership in the container when mounted. The current fix for this is to execute chown clouseau:clouseau on those folders in the container; but there should be a way to specify the folder owner when running the container.
• python -m build has been screwy on Linux; it does something weird with the path.
  • https://python-poetry.org/ might be a option for this.

[brandomr]

Looks great so far. I noticed a couple things when I tested it out:

1. There seems to be an unnecessary extra directory layer inside the run directory...can this be avoided?
2. Can we write the parameters file used back out to the run result directory for posterity's sake?
3. We should try to do some kind of model search where you can put in a word/keyword and see a list of ranked matching models
4. We should give the model container a name that is somewhat descriptive (e.g. the model name with some timestamp or something)
5. we should aim to have this pip installable so instead of python dojocli/cli.py it would be something like dojo-cli runmodel etc
6. Are accessories being sucked out of the container? If not, we should grab them and possibly to to write a captions.txt file that would have captions for them.
7. I don't think dockerhub auth is needed--all these images should be public
8. We might want to consider some kind of "interactive mode" which asks the user a set of questions such as:
   • what kind of model are you looking for? -> user enters some text
   • select model from list -> user enters a number
   • select from options [describe model, choose parameters] -> user picks from 2 options and either learns more about the model from its description/maintainer info or gets to set parameters
   • user is iteratively asked to set each parameter (or shown the default)
9. We should definitely have a mechanism to get the model description (sanitized of the UAZ ontologies). This is a describe command.
10. We should show users parameter descriptions so they can make good choices somewhere. This is a get parameters command (could name it something more elegant)
11. We should show users output variable descriptions so they can interpret the output. This is a get output variables command.

NOTE: #9-11 are probably most important.

[brandomr]

For #1 withCHIRPS-GEFS Monthly the structure is:

runs/
   CHIRPS-GEFS Monthly/
      20212404132416/
         ddd12a2c-2939-44f6-91e1-6d6146752b8a/
            chirps.csv

Where ddd12a2c-2939-44f6-91e1-6d6146752b8a is the id of the outputfile. Do we need this? I think it is going to confuse people.

[brandomr]

Here is an example of accessories for a model with captions: https://dojo-test.com/dojo/accessories/33cf1a60-2544-420f-ae08-b453a9751cfc

[brandomr]

Once Docker is installed on Linux or Mac you can add the current user to the Docker group with:

sudo groupadd docker
sudo gpasswd -a $USER docker

then log out/back in so changes can take effect. This should be done after installing Docker (include this in README, as well as link to Docker install instructions)

[GoogleSheets]

1. There seems to be an unnecessary extra directory layer inside the run directory...can this be avoided?

Changed this folder to output. Need to test a bit more.

2. Can we write the parameters file used back out to the run result directory for posterity's sake?

Done.

3. We should try to do some kind of model search where you can put in a word/keyword and see a list of ranked matching models

TODO.

4. We should give the model container a name that is somewhat descriptive (e.g. the model name with some timestamp or something)

Done. ModelNameDatetimeStamp e.g. stochasticgriddedconflictmodel20211209164335

5. we should aim to have this `pip installable` so instead of `python dojocli/cli.py` it would be something like `dojo-cli runmodel` etc

Done, got build to work on by specifying sdist and wheel: python -m build --sdist ~/source/repos/dojo-cli --wheel

6. Are accessories being sucked out of the container? If not, we should grab them and possibly to to write a `captions.txt` file that would have captions for them.

Yes, try --model="Stochastic Gridded Conflict Model". Captions written to accessories-captions.json

7. I don't think dockerhub auth is needed--all these images should be public

Correct; dockerhub auth is removed.

8. We might want to consider some kind of "interactive mode" which asks the user a set of questions such as:

   * what kind of model are you looking for? -> user enters some text
   * select model from list -> user enters a number
   * select from options `[describe model, choose parameters]` -> user picks from 2 options and either learns more about the model from its description/maintainer info or gets to set parameters
   * user is iteratively asked to set each parameter (or shown the default)

Need to decide if it is worth the effort.

9. We should definitely have a mechanism to get the model description (sanitized of the UAZ ontologies). This is a `describe` command.

describe command implemented

10. We should show users parameter descriptions so they can make good choices somewhere. This is a `get parameters` command (could name it something more elegant)

printparams implemented

11. We should show users output variable descriptions so they can interpret the output. This is a `get output variables` command.

printoutputs implemented

See the updated README.MD for examples of describe, printparams, and printoutputs.

"Stochastic Gridded Conflict Model" is a good test model because it has multiple accessory and output files. printparams does not find any example params in the command_raw, so I run it with no params it apparently uses default values.

Other items:

• progress bar on docker image download
• option to not detach container i.e. have it stop on model completion
• handle model versions
• container status?
• container logs?

[brandomr]

@GoogleSheets thanks for the update on this! Will take a look--let's plan to circle up on next steps on Monday.

[brandomr]

I think we should rename dojocli to just dojo for the install.

I think we should change printparams to parameters and printoutputs to just `outputs.

We probably should have a versions command to list versions and commit messages; then --version option for the other commands.

[brandomr]

1. Adding version flag
2. Adding listversions command
3. Write logs.txt to run directory
4. Quotes around all the things
5. Stop/remove container after run completes
6. Clean up README
7. Move repo to "dojo-modeling" organization on Github
8. attach to model (default to true)

[GoogleSheets]

Updated TODO

@brandomr

• [X] Adding version flag
• [X] Adding listversions command: versions
• [X] Write logs.txt to run directory
• [X] Quotes around all the things
• [X] Stop/remove container after run completes (attached only)
• [X] Attach to model (default to true)
• [X] Add user alerts for details of using detached containers.
• [X] Prefix container names with dojo-
• [X] Include model name and version/model_id in screen output
• [X] Fix container command formatting for model commands that include bash -c e.g. Flee and Topoflow
• [X] Return models without images to list. Direct user to versions when attempting to run model lacking image.
• [X] Clean up README
• [X] Move repo to "dojo-modeling" organization on Github (make main branch default)
• [X] Push build to PyPi
• [X] Prepare video walkthrough
• [X] Test ad nauseum
• [ ] Stream logs for detached containers.

[GoogleSheets]

@brandomr Repo is moved to https://github.com/dojo-modeling/dojo-cli but I will continue updating this issue or we can create a new issue there.

The detached container logging is outstanding.

[GoogleSheets]

@brandomr Noting a few issues or updates after seeing demo:

• [X] dojo-cli requirements.txt is missing Jinja2>=2.11.3
• [X] cite demo script in README.md
• [X] cite pypi install (pip install dojo-cli) in installation instructions
• [ ] update PyPi package

[GoogleSheets]

PyPi 0.1.3 published with --results command to check if a detached model run is complete. If it is, the outputs and logs will be copied locally and the container removed.