Architecting data flow with Google Sheets

[johnml1135]

We need to figure out the main data flow for the orchestrating of these experiments.

Proposal:

• The initial starting is run locally in python, using a yaml file, which references a Google sheet for the data of each run
• When starting, the following happens:
  • The yaml and Google sheet are validated for having "accurate enough" data
  • A new Google drive folder is created that will hold
  • The original yaml and (copied) Google sheet
  • Any other configuration files needed (yamls for NMT building, etc.)
  • A new Google sheet that will contain all the results (raw data, charts, etc.)
  • The new folder is set to "anyone with the link can see"
  • A ClearML task will be queued to process the experiments
  • The ClearML task will start on a CPU only instance with access to the Google folder with the data (through a "anyone can read" link)
  • It will create a folder on the S drive that will in turn have a folder per experiment
  • Each experiment folder will have the appropriate data copied to it
  • All experiments will be created in ClearML in a new, unique project and queued
  • The "results" Google sheet will be updated with the status of each experiment as they progress
  • When all sub-experiments are complete, the top experiment will pull results data from the S drive and copy it to the Google drive folder
  • It will then summarize the data in the results Google sheet, along with any tables and charts needed.

[johnml1135]

How to make a chart on Google sheets: https://www.youtube.com/watch?v=xt3p5I8mNWE