Wed Feb 16 Hackathon

[laszewsk]

• [x] license of everything we do is apache 2.0 send sattement via email. I agree that ... is done under Apache 2.0 license (see #39 for individual comments showing agreements.

  • [x] jake agrees
  • [x] thomas agrees
  • [x] robert agrees

• [x] test code with me (pycharm)

• [ ] code location:

  • [x] https://github.com/laszewsk/mlcommons/tree/main/benchmarks/earthquake
  • [x] https://github.com/laszewsk/mlcommons/tree/main/examples/mnist-tensorflow
  • [ ] rivanna tutorial https://laszewsk.github.io/mlcommons/docs/tutorials/rivanna/
  • [ ] have all tried this

• [ ] do we have install instructions for cloudmesh that work in conda?

• [ ] integrate the mnist performance data in the mlcommons laszewsk web site that records the benchmarks

  • [ ] Robert https://spring22ds6011002.slack.com/team/U030LCZ4N3V
  • [ ] Thomas
  • [ ] Jake

• [x] can pycharm code with me be used to edit code at the same time from multile people

• [x] use onlcy git cp/mv from gregors code, e.g. gregor must make first checkin.

• [ ] record things in a hackathon.md file and this issue

• [x] Plan for hackathon and prepwork:

  • [x] windows computers must have prior to 6pm (this can be in the worst case done during the hackathon:
  • [x] gitbash installed
  • [x] ssh-keygen
  • [x] id_rsa.pub uploades to your github preferences
  • [x] pycharm installed
  • [x] enough space to downoald and uncpmpress data on your machine
  • [x] ~/ENV3 environment and not in venv in the directory using python 3.10.0 (conda) or better 3.10.2 ([python.org (http://python.org/))
  • [x] activated ~/ENV3
  • [x] mac or linux (similar)
  • [x] note that i do not use vscode and recommend pycharm at this time

• [x] nstallation of cloudmesh-cms via cloudmesh-installer

• [x] if you have installed before we have an undated cloudmesh-common to do some tricks with benchmarks

• [ ]I like to add the @benchmark decorator. I have actually created a super cool extension that addresses a rename of the internal wrapper so you can refer to the wrapped function as the same name as the internal function, but I do not know if this works.

  • [x] This is a bit advanced python, so you guys may want to help and create testcases fro that

• [ ] upload of data to data dir

  • [ ]each of us compresses and does the benchmark on the local machine, add benchmark on rivanna
  • [ ] link here: on where original data is

• [ ] gregor converst to python from jupyter notebook

• [x] discuss juptext which i can not identify why this is diffeerent from nbconvert if no different use nbconvert

• [ ] use gregors libs already separated: MAIN TASK

• [x] 1. Phase one delete not needed code

• [ ] 2. Phase two simplify print statements so code is readable and we can replace and enhance the print features mor easily.

• [x] 3. Phase create condition to read daat if locally run or in collab

• [ ] identify if we run in colab and than use colab locations of data else use local data

• [ ] 4. Phase: make sure we have code that reades directly from githb if data not available (easy to develop

• [ ] 5. Phase: general code cleanup

• [ ] 6. Phase benchmark and run

• [ ] safe original code and figure oout to run with data location adapted

• [ ] run modified coed and compare result with original code

• [x] integrate cloudmesh benchmarks where needed.

• [ ] cloudmesh-data

  • [ ] if file is git download from git with requests

• [ ] improve documentation

[laszewsk]

• [x] add this is some form as tutorial to mlcommons

ssh-keygen
# 1. (return)
# 2. type string password
cat ~/.ssh/id_rsa.pub
-------------------
execute in terminal or git bash
python3.10 -m venv ~/ENV3
source ~/ENV3/bin/activate
mkdir -p ~/Desktop/github/cm
cd ~/Desktop/github/cm
pip install pip -U
pip install cloudmesh-installer
cloudmesh-installer get cms
cms help
cms debug on

[laszewsk]

• [ ] juptext: A suggestion I'd like to propose as we go forward with code edits: I think we might benefit from using an alternate ipynb format representation that works better with version control that can do round trip conversions. I've use jupytext (https://jupytext.readthedocs.io/en/latest/) with a lot of success in the past using the --format py:percent and the --format myst switches. This uses a format is similar to a something called "hydrogen", which is supported by several IDEs and keeps the cell structure in a deliberate format that can roundtrip the notebook formatting. (the Myst format is part of the executable book project that's python's answer to R's popular bookdown package). Here's an example: ipynb - https://github.com/mwouts/jupytext/blob/main/demo/World%20population.ipynb py:percent - https://github.com/mwouts/jupytext/blob/main/demo/World%20population.pct.py Now why would I suggest this? As we're looking into code review tools, formatters, and other python code analysis tools along with batch execution, the ipynb format isn't well equipped. Jupyter provides a utility to run the notebook using jupyter run file.ipynb , but this adds additional overhead to runs. Papermill gives a bit more flexibility, but it still uses the ipynb format which doesn't play well with version control or with static analysis tools. It'd be wonderful if we could use things like pep8, pyflake, prospector, black, et al, against the notebook, but they only support *.py files. It could unlock a few additional options to help when working with a notebook of this size.

[laszewsk]

• [ ] add this somewhere on mlcommens web site if it is not already

I have made major updates to the repo. it will be necessary to do a new clone as easisst for those not familiar with submodules.

git clone --recursive git@github.com:Data-ScienceHub/mlcommons-science.git
cd mlcommons-science/mlcommons
git remote set-url origin git@github.com:laszewsk/mlcommons.git
git checkout main
cd ..

[laszewsk]

• [ ] docker container with singularity https://spring22ds6011002.slack.com/team/U030HEUC30D

[laszewsk]

• [x] link I'm stepping through the earthquake notebook now and I noticed that the topearthquakes_20.csv and pix_faults_SmallJan21.csv are not in the googledrive share that has the 1950start data. Do you know where this data is located? (edited) https://drive.google.com/drive/folders/1wz7K2R4gc78fXLNZMHcaSVfQvIpIhNPi?usp=sharing

Gregor von Laszewski 10 days ago note to gregor: I have them on dgx ls /raid/colabdata/EarthquakeDec2020/ 1950start pix_faults_SmallJan21.csv ValidationEARTHQDGX-Transformer1 Outputs topearthquakes_20.csv ValidationEARTHQN-Transformer3 Will have to copy them to rivanna

[rknuu]

• [ ] If you haven't already, you can get Jetbrains Ultimate for free using your university account (and by proxy can install pycharm professional)

https://www.jetbrains.com/shop/eform/students

• [ ] @Toble007
• [ ] @jakekolessar

[laszewsk]

• [x] cahnge the color theme to light colors before you join hackathon if you can as white font on black background is difficult to see for some
• [ ] make sure if you have an external monitor to use it, we wil be sharing a big screen. Mine is 4K