Closed laszewsk closed 1 year ago
Currently an mnist test has been implemented https://github.com/cloudmesh/cloudmesh-cc/blob/main/tests/test_070_run_mnist_workflow_exec.py
This test runs this python file: https://github.com/cybertraining-dsc/reu2022/blob/main/code/deeplearning/mnist/run_all_rivanna.py
...which executes the mlp_mnist and the mnist_autoencoder Jupyter notebooks. https://github.com/cybertraining-dsc/reu2022/blob/main/code/deeplearning/mnist/mlp_mnist.ipynb https://github.com/cybertraining-dsc/reu2022/blob/main/code/deeplearning/mnist/mnist_autoencoder.ipynb
This issue needs a review to see if this has successfully completed the objective, or if there is more to be done.
+---------------------------------+----------+-----------+-----------+---------------------+----------------------------
----------+-------+---------+--------+-------+-------------------------------------+
| Name | Status | Time | Sum | Start | tag
| msg | Node | User | OS | Version |
|---------------------------------+----------+-----------+-----------+---------------------+----------------------------
----------+-------+---------+--------+-------+-------------------------------------|
| v100-total | ok | 16451.4 | 16451.4 | 2022-09-23 18:56:43 | rivanna-IntelXeonE5-
2630-v100 | | rivanna | XXXXXX | Linux | #1 SMP Wed Feb 23 16:47:03 UTC 2022 |
| mlp_mnist | ok | 0.014 | 0.014 | 2022-09-23 23:30:54 | rivanna-IntelXeonE5-
2630-v100 | | rivanna | XXXXXX | Linux | #1 SMP Wed Feb 23 16:47:03 UTC 2022 |
| mnist_autoencoder | ok | 0.007 | 0.007 | 2022-09-23 23:30:54 | rivanna-IntelXeonE5-
2630-v100 | | rivanna | XXXXXX | Linux | #1 SMP Wed Feb 23 16:47:03 UTC 2022 |
| mnist_cnn | ok | 0.006 | 0.006 | 2022-09-23 23:30:54 | rivanna-IntelXeonE5-
2630-v100 | | rivanna | XXXXXX | Linux | #1 SMP Wed Feb 23 16:47:03 UTC 2022 |
| mnist_lstm | ok | 0.006 | 0.006 | 2022-09-23 23:30:54 | rivanna-IntelXeonE5-
2630-v100 | | rivanna | XXXXXX | Linux | #1 SMP Wed Feb 23 16:47:03 UTC 2022 |
| mnist_mlp_with_lstm | ok | 0.006 | 0.006 | 2022-09-23 23:30:54 | rivanna-IntelXeonE5-
2630-v100 | | rivanna | XXXXXX | Linux | #1 SMP Wed Feb 23 16:47:03 UTC 2022 |
| mnist_rnn | ok | 0.006 | 0.006 | 2022-09-23 23:30:54 | rivanna-IntelXeonE5-
2630-v100 | | rivanna | XXXXXX | Linux | #1 SMP Wed Feb 23 16:47:03 UTC 2022 |
| mnist_with_distributed_training | ok | 0.006 | 0.006 | 2022-09-23 23:30:54 | rivanna-IntelXeonE5-
2630-v100 | | rivanna | XXXXXX | Linux | #1 SMP Wed Feb 23 16:47:03 UTC 2022 |
| mnist_with_pytorch | ok | 0.006 | 0.006 | 2022-09-23 23:30:54 | rivanna-IntelXeonE5-
2630-v100 | | rivanna | XXXXXX | Linux | #1 SMP Wed Feb 23 16:47:03 UTC 2022 |
+---------------------------------+----------+-----------+-----------+---------------------+----------------------------
----------+-------+---------+--------+-------+-------------------------------------+
it takes a long time, but it works.
pytest -v -x --capture=no tests/test_070_run_mnist_workflow_exec.py
updated times. it takes nearly 4 hours. perhaps we dont include this in the usual pytest
suite? or we lessen the number of epochs?
+---------------------------------+----------+-----------+-----------+---------------------+--------------------------------------+-------+---------+--------+-------+-----------
--------------------------+
| Name | Status | Time | Sum | Start | tag | msg | Node | User | OS | Version
|
|---------------------------------+----------+-----------+-----------+---------------------+--------------------------------------+-------+---------+--------+-------+-----------
--------------------------|
| v100-total | ok | 13137.3 | 13137.3 | 2022-09-24 00:41:05 | rivanna-XXXXXX-IntelXeonE5-2630-v100 | | rivanna | XXXXXX | Linux | #1 SMP Wed
Feb 23 16:47:03 UTC 2022 |
| mlp_mnist | ok | 178.13 | 178.13 | 2022-09-24 00:41:05 | rivanna-XXXXXX-IntelXeonE5-2630-v100 | | rivanna | XXXXXX | Linux | #1 SMP Wed
Feb 23 16:47:03 UTC 2022 |
| mnist_autoencoder | ok | 354.45 | 354.45 | 2022-09-24 00:44:03 | rivanna-XXXXXX-IntelXeonE5-2630-v100 | | rivanna | XXXXXX | Linux | #1 SMP Wed
Feb 23 16:47:03 UTC 2022 |
| mnist_cnn | ok | 1291.1 | 1291.1 | 2022-09-24 00:49:58 | rivanna-XXXXXX-IntelXeonE5-2630-v100 | | rivanna | XXXXXX | Linux | #1 SMP Wed
Feb 23 16:47:03 UTC 2022 |
| mnist_lstm | ok | 892.249 | 892.249 | 2022-09-24 01:11:29 | rivanna-XXXXXX-IntelXeonE5-2630-v100 | | rivanna | XXXXXX | Linux | #1 SMP Wed
Feb 23 16:47:03 UTC 2022 |
| mnist_mlp_with_lstm | ok | 4232.95 | 4232.95 | 2022-09-24 01:26:21 | rivanna-XXXXXX-IntelXeonE5-2630-v100 | | rivanna | XXXXXX | Linux | #1 SMP Wed
Feb 23 16:47:03 UTC 2022 |
| mnist_rnn | ok | 297.577 | 297.577 | 2022-09-24 02:36:54 | rivanna-XXXXXX-IntelXeonE5-2630-v100 | | rivanna | XXXXXX | Linux | #1 SMP Wed
Feb 23 16:47:03 UTC 2022 |
| mnist_with_distributed_training | ok | 4491.24 | 4491.24 | 2022-09-24 02:41:52 | rivanna-XXXXXX-IntelXeonE5-2630-v100 | | rivanna | XXXXXX | Linux | #1 SMP Wed
Feb 23 16:47:03 UTC 2022 |
| mnist_with_pytorch | ok | 1399.59 | 1399.59 | 2022-09-24 03:56:43 | rivanna-XXXXXX-IntelXeonE5-2630-v100 | | rivanna | XXXXXX | Linux | #1 SMP Wed
Feb 23 16:47:03 UTC 2022 |
+---------------------------------+----------+-----------+-----------+---------------------+--------------------------------------+-------+---------+--------+-------+-----------
--------------------------+
# csv,timer,status,time,sum,start,tag,msg,uname.node,user,uname.system,platform.version
# csv,v100-total,ok,13137.33,13137.33,2022-09-24 00:41:05,rivanna-XXXXXX-IntelXeonE5-2630-v100,None,rivanna,XXXXXX,Linux,#1 SMP Wed Feb 23 16:47:03 UTC 2022
# csv,mlp_mnist,ok,178.13,178.13,2022-09-24 00:41:05,rivanna-XXXXXX-IntelXeonE5-2630-v100,None,rivanna,XXXXXX,Linux,#1 SMP Wed Feb 23 16:47:03 UTC 2022
# csv,mnist_autoencoder,ok,354.45,354.45,2022-09-24 00:44:03,rivanna-XXXXXX-IntelXeonE5-2630-v100,None,rivanna,XXXXXX,Linux,#1 SMP Wed Feb 23 16:47:03 UTC 2022
# csv,mnist_cnn,ok,1291.096,1291.096,2022-09-24 00:49:58,rivanna-XXXXXX-IntelXeonE5-2630-v100,None,rivanna,XXXXXX,Linux,#1 SMP Wed Feb 23 16:47:03 UTC 2022
# csv,mnist_lstm,ok,892.249,892.249,2022-09-24 01:11:29,rivanna-XXXXXX-IntelXeonE5-2630-v100,None,rivanna,XXXXXX,Linux,#1 SMP Wed Feb 23 16:47:03 UTC 2022
# csv,mnist_mlp_with_lstm,ok,4232.946,4232.946,2022-09-24 01:26:21,rivanna-XXXXXX-IntelXeonE5-2630-v100,None,rivanna,XXXXXX,Linux,#1 SMP Wed Feb 23 16:47:03 UTC 2022
# csv,mnist_rnn,ok,297.577,297.577,2022-09-24 02:36:54,rivanna-XXXXXX-IntelXeonE5-2630-v100,None,rivanna,XXXXXX,Linux,#1 SMP Wed Feb 23 16:47:03 UTC 2022
# csv,mnist_with_distributed_training,ok,4491.236,4491.236,2022-09-24 02:41:52,rivanna-XXXXXX-IntelXeonE5-2630-v100,None,rivanna,XXXXXX,Linux,#1 SMP Wed Feb 23 16:47:03 UTC 2022
# csv,mnist_with_pytorch,ok,1399.593,1399.593,2022-09-24 03:56:43,rivanna-XXXXXX-IntelXeonE5-2630-v100,None,rivanna,XXXXXX,Linux,#1 SMP Wed Feb 23 16:47:03 UTC 2022
yes this should be not in pytest but example dire. the issue may be that if you say pytest without parameter it looks through ll dirs with test*.py and executes them, so I how we can name it example*.py and it can be started with example_ name
pytest -v -x --capture=no examples/example_run_mnist_workflow_exec.py
this test now iterates through the GPUs available on rivanna:
gpu = ['v100', 'a100', 'k80', 'p100']
and runs all mnist python scripts successfully. by submitting sbatch with the --gres=gpu parameter.
please document and add section for how to run and what output will be
i believe this is done. https://github.com/cloudmesh/cloudmesh-cc/blob/main/api/source/mnist.md
This has been partially finished in the code directory.
https://github.com/cloudmesh/cloudmesh-cc/tree/main/tests/mnist
Here ^.