Watcher failing from stale filehandle

2024-11-19 12:36:41,671 INFO: Found 0 unclaimed jobs.
2024-11-19 12:37:02,979 INFO: Reading state from /global/cfs/cdirs/m3408/var/dev/agent.state
2024-11-19 12:39:10,376 INFO: Restored 0 jobs from state
2024-11-19 12:39:13,171 DEBUG: Starting new HTTPS connection (1): api-dev.microbiomedata.org:443
2024-11-19 12:39:13,405 DEBUG: https://api-dev.microbiomedata.org:443 "GET /jobs?max_page_size=100&filter=%7B%22workflow.id%22:%20%7B%22$in%22:%20%5B%22Sequencing%20Noninterleaved:%20%22,%20%22Sequencing%20Interleaved:%20%22,%20%22Reads%20QC:%20v1.0.13%22,%20%22Reads%20QC%20Interleave:%20v1.0.12%22,%20%22Metagenome%20Assembly:%20v1.0.7%22,%20%22Metagenome%20Annotation:%20v1.1.0%22,%20%22MAGs:%20v1.3.12%22,%20%22Readbased%20Analysis:%20v1.0.8%22%5D%7D,%20%22claims%22:%20%7B%22$size%22:%200%7D%7D HTTP/11" 200 16
2024-11-19 12:39:13,406 INFO: Found 0 unclaimed jobs.
2024-11-19 12:39:33,511 INFO: Reading state from /global/cfs/cdirs/m3408/var/dev/agent.state
2024-11-19 12:40:33,557 INFO: Restored 0 jobs from state
2024-11-19 12:40:33,561 ERROR: Error occurred during cycle: [Errno 116] Stale file handle: '/global/homes/n/nmdcda/nmdc_automation/dev/nmdc_automation/nmdc_automation/config/workflows/workflows.yaml'
Traceback (most recent call last):
  File "/global/homes/n/nmdcda/nmdc_automation/dev/nmdc_automation/nmdc_automation/workflow_automation/watch_nmdc.py", line 360, in watch
  File "/global/homes/n/nmdcda/nmdc_automation/dev/nmdc_automation/nmdc_automation/workflow_automation/watch_nmdc.py", line 318, in cycle
  File "/global/homes/n/nmdcda/nmdc_automation/dev/nmdc_automation/nmdc_automation/config/siteconfig.py", line 104, in allowed_workflows
OSError: [Errno 116] Stale file handle: '/global/homes/n/nmdcda/nmdc_automation/dev/nmdc_automation/nmdc_automation/config/workflows/workflows.yaml'
microbiomedata / nmdc_automation

Watcher failing from stale filehandle #289
