search

hpcflow / matflow-new

Mozilla Public License 2.0
3 stars 1 forks source link

Demo workflow not working on CSF3 with latest matflow #256

Closed gcapes closed 3 weeks ago

gcapes commented 1 month ago

HI @aplowman I've reproduced an error that @yuchenZZheng is having using MatFlow, version 0.3.0a128 on CSF. I've installed matflow in a python venv, Yuchen has installed it following these instructions https://docs.matflow.io/stable/installation.html#matflow-cli

We're running this workflow: matflow demo-workflow go tension_DAMASK_Al, but get the same error for tension_DAMASK_Ti. The error is in the first task, which then looks to cause the rest to fail. Looking in artifacts/submissions/... at the SGE error file TypeError: Command.__init__() got an unexpected keyword argument 'rules'

Not sure where to go from here really - is this a problem with the task schema?

aplowman commented 1 month ago

This is because the same version of MatFlow needs to be installed in the "software" environment that is used for each task. In your case, there is a mismatch. If you are using the shared MatFlow conda environment matflow_full_env, this has not yet been updated (as discussed here). So, you can either update the shared conda environment, or you could help @YuchenZZheng set up their own conda environment with the current matflow version. You would need to update MatFlow's environment file so it points to the new conda environment.

gcapes commented 1 month ago

Currently trying to get the action to run (https://github.com/LightForm-group/non-repo-issues/issues/25), in order to build the environment file for Yuchen to use.

gcapes commented 1 month ago

I've been unpacking the environment file on CSF3 for a long time now (into my own user space), so just making a note in case I need to return to it another time. I've installed matflow-new using pip, and was going to look into editing a copy of the environment file once this has unpacked. This is likely duplication of work yuchen is doing, but will be useful practice for updating a shared installation in the future

gcapes commented 1 month ago

Restarted it and it went quicker. I noticed when modifying a duplicate of the csf3 env file that previously each env e.g. damask_parse_env defdap_env have different paths, but I only get the one path for the matflow_full_env conda environment I've unpacked. Does the full env include all the others, which were previously done separately?

gcapes commented 1 month ago

I am currently running the demo workflow, and get this warning

$ matflow show
/mnt/iusers01/support/mbexegc2/yuchen-zheng/.venv/lib/python3.11/site-packages/paramiko/pkey.py:100: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed
from this module in 48.0.0.
  "cipher": algorithms.TripleDES,
/mnt/iusers01/support/mbexegc2/yuchen-zheng/.venv/lib/python3.11/site-packages/paramiko/transport.py:259: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be 
removed from this module in 48.0.0.
  "class": algorithms.TripleDES,

MatFlow, version 0.3.0a129

gcapes commented 1 month ago

The demo workflow ran without errors :) Looks like I'll be able to update matflow on CSF3 following discussion

gcapes commented 3 weeks ago

In summary, this wasn't a MatFlow error at all - it was just that the environment wasn't up-to-date.