search

ufs-community / ufs-srweather-app

UFS Short-Range Weather Application
Other
55 stars 116 forks source link

Running from cron no longer possible on Derecho #998

Open mkavulich opened 7 months ago

mkavulich commented 7 months ago

Expected behavior

Previously the USE_CRON_TO_RELAUNCH=true option worked on all Tier-1 platforms (as far as I know).

Current behavior

Due to a policy change with the new machine, there are now special procedures for setting up cron tables on Derecho. These procedures are not compatible with automatic modification due to needing to log in to a separate machine, so it is not feasible to support this mode of running the workflow automatically (USE_CRON_TO_RELAUNCH=true) on Derecho.

Machines affected

Derecho

Steps To Reproduce

~~1. Generate an experiment with the USE_CRON_TO_RELAUNCH=true option on Derecho

  1. Observe that the workflow is not run.~~

So far, it looks like cron jobs do still work on Derecho. But I have been told by CISL that we need to migrate away from this system ASAP.

Detailed Description of Fix

Users guide run instructions will need to be updated with Derecho-specific instructions, as the crontab functionality is currently the recommended way to run the workflow.

Possible Implementation

One way to get around this would be to leverage the WE2E functionality currently present for running and monitoring experiments for general use. This would require some tweaking of the current setup to be more user-friendly outside of the WE2E context.

Output (optional)

Currently, the error message that appears in log.launch_FV3LAM_wflow is:

Running "module reset". Resetting modules to system default. The following $MODULEPATH directories have been removed: None
ERROR:
Loading of platform-specific module file (WFLOW_MOD_FN) for the workflow 
task failed:
  WFLOW_MOD_FN = "wflow_derecho"

But that may change in the future, as I was informed by CISL that the crontab functionality will stop working all together at some point.

SarahLu-NOAA commented 7 months ago

@mkavulich My SRW exp at Derecho sit in the queue for 12 hours since last night. The exp with USE_CRON_TO_RELAUNCH=true runs ok until last night. You posted this issue last week. Is my jobs sitting in the queue related to a very busy Derecho or with the cron option in UFS/SRW no longer working/supported?

mkavulich commented 7 months ago

@SarahLu-NOAA It looks like the cron jobs have not yet been disabled (though this change is "imminent"), so this is likely unrelated to the issues you saw.