valohai-utils

[akx]

• [x] Parse Python source code for inputs & parameters
• [x] Implement valohai.prepare() without writing anything to disk
• [x] Downloading & caching inputs (for local runs)
• [x] Unpacking and iterating input files
• [x] Auto-argparsing parameters
• [x] Output packing
• [x] Generate valohai.yaml
• [x] Standardized logging
• [x] Default image for valohai.yaml
• [x] Valohai CLI command(s)
• [x] Refactor global state to singleton
• [x] Redesign of API (https://valohai.slack.com/archives/C3J44LTL6/p1605277828112500)
• [x] Write decent readme.md

Original Google doc: https://docs.google.com/document/d/1Jbi2PhLKgSgvjtSsk6e1FvS5eEjCXgO0LCt7oJSYDnE/edit

---

User story

• Annoying to get and manage dataset locally:
  • Heavily manual process with varying tools
  • Annoying to manually download
  • Forget to ignore data folder in .gitignore
  • Can’t remember how to use unzip/tar command
  • Forget to delete the local archive, iterate files in code expecting everything to be JPEGs, crashes on .zip
  • Write separate code for local & Valohai (different local paths)
    • Need know about VH_INPUTS_DIR
    • Need to write unzip code just for Valohai

What I wantz

• This is my input called “my_input” and this is the url
  • Classic: Write to valohai.yaml input called my_images with default url
  • Jupyhai: Write to cell tagged as inputs has variable called my_images
• One line of code that gets me a path to my files
  • If local machine, download & cache
  • If it is a zip, unzip it for me plz

Proposal

• Call get_inputfile{paths, streams, ...}(”my_images”, unarchive=True, extensions=(“.jpg”, “.jpeg”, “.gif”))
  • Does the unarchiving happen in the named input root, say, the archive name doesn’t create an extra directory? Assumption: yes, it happens in the named input root directory without an extra directory
  • What happens if there are multiple archives in the input? Assumption: unarchive all
  • What happens if the multiple archives have overwriting files? Assumption: overwrite silently, but if e.g. 2 input files have the same directories with differently named files, we should maintain all of them
  • Local:
  • 1. Figure out url
    2. Classic: Look for my_input input default url in valohai.yaml
    3. Jupyhai: globals()[”my_images”]
  • 2. Download & cache
  • 3. Auto unzip/untar/etc if possible
  • 4. Return
  • ./.valohai/inputs/my_images/some.jpg
  • ./.valohai/inputs/my_images/some2.jpg
  • ./.valohai/inputs/my_images/some3.jpg
  • Valohai:
  • 1. Already downloaded by Peon
  • 2. Return
  • /valohai/inputs/my_images/some.jpg
  • /valohai/inputs/my_images/some2.jpg
  • /valohai/inputs/my_images/some3.jpg

[akx]

Done.