Onboard: Optimus

[YannCoderre]

This ticket is related to Project Optimus

Contact is Kate Burnett-Isaccs

JIRA ticket: https://jirab.statcan.ca/browse/DACSU-1218

[YannCoderre]

Meeting notes from March 2, 2022:

Kate needs more information concerning the services between AVD and DAaaS. Kate is under the impression that the upcoming deployment (November 24, 2022) of the Statcan AVD will suffice most of the users (analyst).

We did inform Kate of the differences between AAW and AVD

• External user(s) access
• Pipelines
• Pre-defined notebooks (with all the software / languages)
• RMI(s) were already completed

I also informed Kate that the AVD (testing machine at this point) might get more restrictions at rollout.

[YannCoderre]

Brendan email March 2, 20202 - RE First steps for Optimus Project.txt

[bryanpaget]

Meeting with Kate

I met with Kate today and we discussed what would be useful to see in an AAW demonstration on March 11 @ 10AM (or March 24 @ 10AM).

Background

• It wasn't until 2017 that R and Python saw widespread adoption within Statistics Canada.
• Until very recently, R and Python were just used in research and development.
• Most analysts using R and Python have been doing so on Net A.
• Bottom line: Analysts just want a well supported R and Python environment with access to packages.
• Most analysts don’t use, need nor know about MLOps.
• Most data sets are at most 1 GB in size though some are using up to 1TB.
• Doing visualizations is important to the analysts.

Questions to Address for the AAW Demo

• How to:
  • Do regular work on the AAW?
  • Bring data into the AAW?
  • Export work from AAW (preferably using git)?
  • How to upgrade R and Python packages?

Big Question

• How do users of the AAW bring their projects from StatCan's Gitlab server (from Cloud Main) into the AAW?
  • Can't do it.
• Can AAW users git clone from StatCan's Gitlab server?
  • No.
• Can Gitea help with this?

Other Issues to Address in the Future

• How to share R Shiny and Plotly Dash dashboards?
  • Can we wrap them in containers and deploy on a CI/CD pipeline?
• How to guide on CI/CD best practices for MLOps.
• How to make R and Python packages and share them on a local repository, not through CRAN or PyPI.

Presentation Outline

From the above, I think an appropriate demonstration of the AAW should cover the following.

1. Getting code from Gitlab Main or some other source.
   • We can use R Studio and JupyterLab, they can upload using a button.
2. Run some code in R Studio or JupyterLab
   • Show how to set up environment and run the code.
3. Committing back to Gitlab or something else.
4. Talk about disks.
   • What size of disk do I recommend?
   • When to use extra storage: “if you have a big file, you can use MinIO”.
   • How do I access large disk, maybe show how you’d access the large data volume
5. Q&A portion will be very useful

[bryanpaget]

What's the status on this? @YannCoderre

[YannCoderre]

@bryanpaget , the Optimus project is ongoing. There are many aspects before we start onboarding this projet:

• One Git solution (GitLab preferred) with CICD capability: Kate understands that we cant have one repository that can hold the code from a user on Protected B and Public. GITEA is not really a viable option. We need to wait until GITEA will be in DEV and PROD
• Protected B is necessary
• Artifactory is import for Kate
• Kate is a strong supporter of AAW
• Data Hybrid Initiative needs to happen
• The platform needs to be stable
• and more to come

[YannCoderre]

This is an ongoing project that is huge. We have created a JIRA board. We are meeting with Kate weekly to discussion and requirements.