search

SalishSeaCast / SalishSeaNowcast

Python package associated with running the SalishSeaCast NEMO and other ocean models in near-real-time to produce daily nowcasts/forecasts
https://salishsea.eos.ubc.ca/nemo/
Apache License 2.0
4 stars 3 forks source link

FileNotFoundError during crop_gribs startup due to race condition with collect_weather startup #197

Closed douglatornell closed 9 months ago

douglatornell commented 10 months ago

Log messages just before failure:

2023-08-28 20:49:47,132 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2023-08-28 20:49:47,134 DEBUG [manager] cmd = ['/SalishSeaCast/nowcast-env/bin/python3', '-m', 'nowcast.workers.collect_weather', '/SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml', '06', '2.5km']
2023-08-28 20:49:47,134 DEBUG [collect_weather] received message from manager: (ack) message acknowledged
2023-08-28 20:49:47,135 DEBUG [collect_weather] shutting down
2023-08-28 20:49:47,138 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2023-08-29'], host='localhost')
2023-08-28 20:49:47,139 DEBUG [manager] cmd = ['/SalishSeaCast/nowcast-env/bin/python3', '-m', 'nowcast.workers.crop_gribs', '/SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml', '06', '--fcst-date', '2023-08-29']
2023-08-28 20:49:47,144 DEBUG [manager] listening...
2023-08-28 20:49:51,174 DEBUG [crop_gribs] wrote GRIB file cropped to SalishSeaCast subdomain: /results/forcing/atmospheric/continental2.5/GRIB/20230829/00/048/20230829T00Z_MSC_HRDPS_APCP_Sfc_RLatLon0.0225_PT048H_SSC.grib2
2023-08-28 20:49:51,174 DEBUG [crop_gribs] observer thread files remaining to process: 5
2023-08-28 20:49:52,831 INFO [crop_gribs] running in process 25692
2023-08-28 20:49:52,832 INFO [collect_weather] running in process 25690
2023-08-28 20:49:52,833 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2023-08-28 20:49:52,834 INFO [crop_gribs] publishing log messages to tcp://*:5562
2023-08-28 20:49:52,835 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2023-08-28 20:49:52,836 INFO [crop_gribs] cropping 2023-08-29 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2023-08-28 20:49:52,838 DEBUG [crop_gribs] creating ECCC GRIB file paths list for 20230829 06Z forecast
2023-08-28 20:49:52,838 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2023-08-28 20:49:52,839 INFO [collect_weather] publishing log messages to tcp://*:5563
2023-08-28 20:49:52,840 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2023-08-28 20:49:52,850 DEBUG [collect_weather] calculated set of expected file paths for 2.5 km 20230829/06
2023-08-28 20:49:52,851 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20230829/06/
2023-08-28 20:49:52,853 DEBUG [collect_weather] created /results/forcing/atmospheric/continental2.5/GRIB/20230829/
2023-08-28 20:49:52,853 DEBUG [collect_weather] created /results/forcing/atmospheric/continental2.5/GRIB/20230829/06/
2023-08-28 20:49:52,854 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2023-08-28 20:49:52,862 CRITICAL [crop_gribs] unhandled exception:

Note that crop_gribs starts to watch /results/forcing/atmospheric/continental2.5/GRIB/20230829/06/ before collect_weather creates it.

Traceback:

2023-08-28 20:49:52,862 CRITICAL [crop_gribs] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 391, in _do_work
    checklist = self.worker_func(
                ^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/crop_gribs.py", line 135, in crop_gribs
    observer.start()
  File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/watchdog/observers/api.py", line 261, in start
    emitter.start()
  File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/watchdog/utils/__init__.py", line 92, in start
    self.on_thread_start()
  File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/watchdog/observers/inotify.py", line 119, in on_thread_start
    self._inotify = InotifyBuffer(path, self.watch.is_recursive)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/watchdog/observers/inotify_buffer.py", line 37, in __init__
    self._inotify = Inotify(path, recursive)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/watchdog/observers/inotify_c.py", line 181, in __init__
    self._add_watch(path, event_mask)
  File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/watchdog/observers/inotify_c.py", line 416, in _add_watch
    Inotify._raise_error()
  File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/watchdog/observers/inotify_c.py", line 432, in _raise_error
    raise OSError(err, os.strerror(err))
FileNotFoundError: [Errno 2] No such file or directory