Open mkavulich opened 7 months ago
@mkavulich My SRW exp at Derecho sit in the queue for 12 hours since last night. The exp with USE_CRON_TO_RELAUNCH=true runs ok until last night. You posted this issue last week. Is my jobs sitting in the queue related to a very busy Derecho or with the cron option in UFS/SRW no longer working/supported?
@SarahLu-NOAA It looks like the cron jobs have not yet been disabled (though this change is "imminent"), so this is likely unrelated to the issues you saw.
Expected behavior
Previously the
USE_CRON_TO_RELAUNCH=true
option worked on all Tier-1 platforms (as far as I know).Current behavior
Due to a policy change with the new machine, there are now special procedures for setting up cron tables on Derecho. These procedures are not compatible with automatic modification due to needing to log in to a separate machine, so it is not feasible to support this mode of running the workflow automatically (
USE_CRON_TO_RELAUNCH=true
) on Derecho.Machines affected
Derecho
Steps To Reproduce
~~1. Generate an experiment with the
USE_CRON_TO_RELAUNCH=true
option on DerechoSo far, it looks like cron jobs do still work on Derecho. But I have been told by CISL that we need to migrate away from this system ASAP.
Detailed Description of Fix
Users guide run instructions will need to be updated with Derecho-specific instructions, as the crontab functionality is currently the recommended way to run the workflow.
Possible Implementation
One way to get around this would be to leverage the WE2E functionality currently present for running and monitoring experiments for general use. This would require some tweaking of the current setup to be more user-friendly outside of the WE2E context.
Output (optional)
Currently, the error message that appears in
log.launch_FV3LAM_wflow
is:But that may change in the future, as I was informed by CISL that the crontab functionality will stop working all together at some point.