Make crop_gribs worker more robust

[douglatornell]

re:

• watchdog observer thread stalls
• HRDPS data dissemination failures
• race condition between crop_gribs and collect_weather re: creation of forcing files directory tree (issue #197)

[codecov[bot]]

Codecov Report

Merging #203 (fd565d7) into main (3b17460) will increase coverage by 0.08%. The diff coverage is 94.28%.

@@            Coverage Diff             @@
##             main     #203      +/-   ##
==========================================
+ Coverage   77.65%   77.73%   +0.08%     
==========================================
  Files         133      133              
  Lines       18655    18723      +68     
  Branches     1969     1976       +7     
==========================================
+ Hits        14486    14554      +68     
+ Misses       4111     4110       -1     
- Partials       58       59       +1     

Flag	Coverage Δ
unittests	77.73% <94.28%> (+0.08%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
tests/test_config.py	100.00% <ø> (ø)
tests/workers/test_crop_gribs.py	99.67% <100.00%> (+0.06%)	:arrow_up:
nowcast/workers/crop_gribs.py	92.48% <78.94%> (-1.44%)	:arrow_down:

... and 1 file with indirect coverage changes

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[douglatornell]

Successfully tested stall recovery in production on several occasions, most recently 25sep23 00Z forecast.

Thanks to issues arising from MSC data centre migration, also had successful test of critical log message containing list of missing files when stall recovery fails: 25sep23 12Z forecast.