pnwairfire / bluesky

BlueSky Framework rearchitected as a pipeable collection of standalone modules.
GNU General Public License v3.0
18 stars 10 forks source link

localmet: IndexError in arlprofiler module when parsing meteorology data #288

Closed daisyhjt closed 4 weeks ago

daisyhjt commented 1 month ago

Hi there,

I'm using the NAM 12 km archived meteorology data (https://www.ready.noaa.gov/archives.php) to run bluesky. However, not all cases run successfully. For some days, I encounter an error: IndexError: list index out of range in the _parse_hourlytext function of the arlprofiler module. Thus it returns exactly empty meteorology json data.

Command Used:

bsp --log-level=DEBUG -i SE2018_05_11_permit.json -c permit_config.json -o SE2018_05_11_met.json -C localmet.working_dir=my_work_dir_name -B localmet.skip_failures=False -C localmet.time_step=3 localmet

Detailed Log:

INFO: underlay of /etc/localtime required more than 50 (118) bind mounts 2024-06-17 00:18:50,831 DEBUG: filling in run_id wildcards 2024-06-17 00:18:50,868 SUMMARY: BlueSky v4.5.19 2024-06-17 00:18:50,868 SUMMARY: Input File(s): /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_input/SE2018_05_11_permit.json 2024-06-17 00:18:50,868 SUMMARY: Output File: /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_met/SE2018_05_11_met.json 2024-06-17 00:18:50,868 INFO: input files: ['/home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_input/SE2018_05_11_permit.json'] 2024-06-17 00:18:50,868 INFO: no input: False 2024-06-17 00:18:50,868 INFO: input file failure tolerance: None 2024-06-17 00:18:50,868 INFO: compress: False 2024-06-17 00:18:50,868 INFO: indent: None 2024-06-17 00:18:50,868 INFO: profile output file: None 2024-06-17 00:18:50,868 INFO: config options: {'localmet': {'working_dir': '/home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_met/SE2018_05_11', 'skip_failures': False, 'time_step': '3'}} 2024-06-17 00:18:50,868 INFO: config file options: {'skip_failed_fires': True, 'skip_failed_sources': True, 'fuelbeds': {'fccs_version': 2}, 'ecoregion': {'lookup_implementation': 'ogr', 'skip_failures': True, 'default': 'southern'}, 'consumption': {'consume_settings': {'activity': {'slope': {'default': 5}, 'windspeed': {'default': 6}, 'days_since_rain': {'default': 10}, 'fuel_moisture_10hr_pct': {'default': 50}, 'fm_type': {'default': 'MEAS-Th'}}, 'all': {'fuel_moisture_1000hr_pct': {'default': 30}, 'fuel_moisture_duff_pct': {'default': 75}, 'fuel_moisture_litter_pct': {'default': 16}, 'canopy_consumption_pct': {'default': 0}, 'shrub_blackened_pct': {'default': 50}, 'pile_blackened_pct': {'default': 0}}}}, 'emissions': {'model': 'prichard-oneill', 'include_emissions_details': False}, 'timeprofile': {'model': 'default'}, 'plumerise': {'model': 'feps', 'feps': {'plume_top_behavior': 'Briggs'}}} 2024-06-17 00:18:50,868 INFO: dump config: False 2024-06-17 00:18:50,868 INFO: version: False 2024-06-17 00:18:50,868 INFO: run id: None 2024-06-17 00:18:50,868 INFO: today: None 2024-06-17 00:18:50,868 INFO: module: ['localmet'] 2024-06-17 00:18:50,868 INFO: log level: 10 2024-06-17 00:18:50,868 INFO: log file: None 2024-06-17 00:18:50,868 INFO: log message format: None 2024-06-17 00:18:50,868 DEBUG: Loading local file: /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_input/SE2018_05_11_permit.json 2024-06-17 00:18:50,869 DEBUG: Failed to gzip.decompress /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_input/SE2018_05_11_permit.json: Not a gzipped file (b'{\n') 2024-06-17 00:18:50,869 INFO: input /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_input/SE2018_05_11_permit.json not gzip'd 2024-06-17 00:18:50,871 INFO: Loaded /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_input/SE2018_05_11_permit.json 2024-06-17 00:18:50,876 DEBUG: Status logging disabled - not submitting 'Good','Main', 'Start', {}. 2024-06-17 00:18:50,877 SUMMARY: Modules to be run: localmet 2024-06-17 00:18:50,877 DEBUG: Status logging disabled - not submitting 'Good','localmet', 'Start', {}. 2024-06-17 00:18:50,877 SUMMARY: Running module localmet 2024-06-17 00:18:50,880 DEBUG: Parsing met file specifications 2024-06-17 00:18:50,891 DEBUG: Extracting localmet data for 78 locations 2024-06-17 00:18:50,892 DEBUG: chdir to working directory /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_met/SE2018_05_11/0 2024-06-17 00:18:50,893 DEBUG: Calling 'bulk_profiler_csv -d/scratch/jh94030/WPS-input/Data/NAM/2018/ -f20180511_nam12 -w2 -t3 -i/home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_met/SE2018_05_11/0/locations.csv -p/home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_met/SE2018_05_11/0/profile.txt' 2024-06-17 00:18:51,962 DEBUG: Loading /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_met/SE2018_05_11/0/profile.txt 2024-06-17 00:19:01,863 DEBUG: chdir back to original directory /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data 2024-06-17 00:19:01,865 DEBUG: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/bluesky-4.5.19-py3.10.egg/bluesky/modules/localmet.py", line 51, in run LocalmetRunner(fires_manager).run() File "/usr/local/lib/python3.10/dist-packages/bluesky-4.5.19-py3.10.egg/bluesky/modules/localmet.py", line 99, in run localmet = arl_profiler.profile( File "/usr/local/lib/python3.10/dist-packages/met/arl/arlprofiler/profiler.py", line 162, in profile lmd = self._load(output_filename, met_file['first_hour'], File "/usr/local/lib/python3.10/dist-packages/met/arl/arlprofiler/profiler.py", line 260, in _load local_hourly_profile = profile.get_hourly_params() File "/usr/local/lib/python3.10/dist-packages/met/arl/arlprofiler/profileparser.py", line 179, in get_hourly_params self.parse_hourly_text(profile) File "/usr/local/lib/python3.10/dist-packages/met/arl/arlprofiler/profileparser.py", line 294, in parse_hourly_text vars[main_vars[j]].append(line[j]) IndexError: list index out of range

2024-06-17 00:19:01,865 ERROR: list index out of range 2024-06-17 00:19:01,866 DEBUG: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/bluesky-4.5.19-py3.10.egg/bluesky/models/fires.py", line 536, in run self._modules[i].run(self) File "/usr/local/lib/python3.10/dist-packages/bluesky-4.5.19-py3.10.egg/bluesky/modules/localmet.py", line 51, in run LocalmetRunner(fires_manager).run() File "/usr/local/lib/python3.10/dist-packages/bluesky-4.5.19-py3.10.egg/bluesky/modules/localmet.py", line 99, in run localmet = arl_profiler.profile( File "/usr/local/lib/python3.10/dist-packages/met/arl/arlprofiler/profiler.py", line 162, in profile lmd = self._load(output_filename, met_file['first_hour'], File "/usr/local/lib/python3.10/dist-packages/met/arl/arlprofiler/profiler.py", line 260, in _load local_hourly_profile = profile.get_hourly_params() File "/usr/local/lib/python3.10/dist-packages/met/arl/arlprofiler/profileparser.py", line 179, in get_hourly_params self.parse_hourly_text(profile) File "/usr/local/lib/python3.10/dist-packages/met/arl/arlprofiler/profileparser.py", line 294, in parse_hourly_text vars[main_vars[j]].append(line[j]) IndexError: list index out of range

2024-06-17 00:19:01,866 DEBUG: Status logging disabled - not submitting 'Failure','localmet', 'Die', {}. 2024-06-17 00:19:01,869 DEBUG: Status logging disabled - not submitting 'Failure','Main', 'Die', {}. 2024-06-17 00:19:01,869 DEBUG: Writing to local file: /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_met/SE2018_05_11_met.json 2024-06-17 00:19:01,870 SUMMARY: Fire counts: {'fires': 78, 'locations': 78, 'failed_fires': 0} 2024-06-17 00:19:01,872 SUMMARY: Run complete INFO: underlay of /etc/localtime required more than 50 (118) bind mounts

Attached Files for Debugging

SE2018_05_11_permit.json permit_config.json locations.csv profile.txt

I guess the error is likely from a mismatch between the expected number of columns in a few lines of the profile.txt file and the actual number of columns. Since I'm limited in my ability to modify the development code on HPC, I was wondering if there is any help to identify and fix the cause of this error?

Thank you in advance!

jdubowy commented 4 weeks ago

The problem is that profile.txt can have asterisk strings of varying lengths, with multiple asterisk strings possibly running into each other as well as into the preceding number values. e.g.

          HGTS   TEMP   UWND   VWND   WWND   RELH   TKEN     TPOT   WDIR   WSP
                   oC    m/s    m/s   mb/h      %              oK    deg    m/
   1000    166   20.6      0      0   0.79   80.8   0.30    293.8*******    0.
    975    384   21.1    2.2    2.1      0   72.8      0    296.4  225.6    3.
    950    610   22.6    6.0    1.3   -2.9   57.6      0    300.1  257.3    6.
    ...

I've updated the profile.txt parsing code with a hack to better handle the various asterisk string possibilities. A better solution would be to update the profiler code (different repo) that generates profile.txt to always guarantee spaces between values. Another solution would be to update the parsing code to split the pressure level value strings based the positions of the variable names in the header string.

I've published bluesky v4.5.20

daisyhjt commented 3 weeks ago

Hi Joel,

Thanks again for your troubleshooting.

However, there is another error related to IndexError in arlprofiler module. Perhaps it is caused by the interpolation function for hourly meteorological profiles. Could you help with this issue?

Detailed Log:

INFO: underlay of /etc/localtime required more than 50 (118) bind mounts 2024-06-24 13:46:24,495 DEBUG: filling in run_id wildcards 2024-06-24 13:46:24,536 SUMMARY: BlueSky v4.5.20 2024-06-24 13:46:24,537 SUMMARY: Input File(s): /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_input/SE2018_12_09_permit.json 2024-06-24 13:46:24,537 SUMMARY: Output File: /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_met/SE2018_12_09_met.json 2024-06-24 13:46:24,537 INFO: input files: ['/home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_input/SE2018_12_09_permit.json'] 2024-06-24 13:46:24,537 INFO: no input: False 2024-06-24 13:46:24,537 INFO: input file failure tolerance: None 2024-06-24 13:46:24,537 INFO: compress: False 2024-06-24 13:46:24,537 INFO: indent: None 2024-06-24 13:46:24,537 INFO: profile output file: None 2024-06-24 13:46:24,537 INFO: config options: {'localmet': {'working_dir': '/home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_met/SE2018_12_09', 'skip_failures': False, 'time_step': '3'}} 2024-06-24 13:46:24,537 INFO: config file options: {'skip_failed_fires': True, 'skip_failed_sources': True, 'fuelbeds': {'fccs_version': 2}, 'ecoregion': {'lookup_implementation': 'ogr', 'skip_failures': True, 'default': 'southern'}, 'consumption': {'consume_settings': {'activity': {'slope': {'default': 5}, 'windspeed': {'default': 6}, 'days_since_rain': {'default': 10}, 'fuel_moisture_10hr_pct': {'default': 50}, 'fm_type': {'default': 'MEAS-Th'}}, 'all': {'fuel_moisture_1000hr_pct': {'default': 30}, 'fuel_moisture_duff_pct': {'default': 75}, 'fuel_moisture_litter_pct': {'default': 16}, 'canopy_consumption_pct': {'default': 0}, 'shrub_blackened_pct': {'default': 50}, 'pile_blackened_pct': {'default': 0}}}}, 'emissions': {'model': 'prichard-oneill', 'include_emissions_details': False}, 'timeprofile': {'model': 'default'}, 'plumerise': {'model': 'feps', 'feps': {'plume_top_behavior': 'Briggs'}}} 2024-06-24 13:46:24,537 INFO: dump config: False 2024-06-24 13:46:24,537 INFO: version: False 2024-06-24 13:46:24,537 INFO: run id: None 2024-06-24 13:46:24,537 INFO: today: None 2024-06-24 13:46:24,537 INFO: module: ['localmet'] 2024-06-24 13:46:24,537 INFO: log level: 10 2024-06-24 13:46:24,537 INFO: log file: None 2024-06-24 13:46:24,537 INFO: log message format: None 2024-06-24 13:46:24,538 DEBUG: Loading local file: /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_input/SE2018_12_09_permit.json 2024-06-24 13:46:24,538 DEBUG: Failed to gzip.decompress /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_input/SE2018_12_09_permit.json: Not a gzipped file (b'{\n') 2024-06-24 13:46:24,538 INFO: input /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_input/SE2018_12_09_permit.json not gzip'd 2024-06-24 13:46:24,539 INFO: Loaded /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_input/SE2018_12_09_permit.json 2024-06-24 13:46:24,568 DEBUG: Status logging disabled - not submitting 'Good','Main', 'Start', {}. 2024-06-24 13:46:24,568 SUMMARY: Modules to be run: localmet 2024-06-24 13:46:24,568 DEBUG: Status logging disabled - not submitting 'Good','localmet', 'Start', {}. 2024-06-24 13:46:24,568 SUMMARY: Running module localmet 2024-06-24 13:46:24,568 DEBUG: Parsing met file specifications 2024-06-24 13:46:24,643 DEBUG: Extracting localmet data for 1 locations 2024-06-24 13:46:24,644 DEBUG: chdir to working directory /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_met/SE2018_12_09/0 2024-06-24 13:46:24,645 DEBUG: Calling 'bulk_profiler_csv -d/scratch/jh94030/WPS-input/Data/NAM/2018/ -f20181209_nam12 -w2 -t3 -i/home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_met/SE2018_12_09/0/locations.csv -p/home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_met/SE2018_12_09/0/profile.txt' 2024-06-24 13:46:25,918 DEBUG: Loading /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_met/SE2018_12_09/0/profile.txt 2024-06-24 13:46:25,985 DEBUG: chdir back to original directory /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data 2024-06-24 13:46:26,001 DEBUG: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/bluesky-4.5.20-py3.10.egg/bluesky/modules/localmet.py", line 51, in run LocalmetRunner(fires_manager).run() File "/usr/local/lib/python3.10/dist-packages/bluesky-4.5.20-py3.10.egg/bluesky/modules/localmet.py", line 99, in run localmet = arl_profiler.profile( File "/usr/local/lib/python3.10/dist-packages/met/arl/arlprofiler/profiler.py", line 162, in profile lmd = self._load(output_filename, met_file['first_hour'], File "/usr/local/lib/python3.10/dist-packages/met/arl/arlprofiler/profiler.py", line 260, in _load local_hourly_profile = profile.get_hourly_params() File "/usr/local/lib/python3.10/dist-packages/met/arl/arlprofiler/profileparser.py", line 182, in get_hourly_params self.spread_hourly_results() File "/usr/local/lib/python3.10/dist-packages/met/arl/arlprofiler/profileparser.py", line 429, in spread_hourly_results closest_date = sorted(times, key=lambda d:abs(new_datetime - d))[0] IndexError: list index out of range

2024-06-24 13:46:26,002 ERROR: list index out of range 2024-06-24 13:46:26,003 DEBUG: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/bluesky-4.5.20-py3.10.egg/bluesky/models/fires.py", line 536, in run self._modules[i].run(self) File "/usr/local/lib/python3.10/dist-packages/bluesky-4.5.20-py3.10.egg/bluesky/modules/localmet.py", line 51, in run LocalmetRunner(fires_manager).run() File "/usr/local/lib/python3.10/dist-packages/bluesky-4.5.20-py3.10.egg/bluesky/modules/localmet.py", line 99, in run localmet = arl_profiler.profile( File "/usr/local/lib/python3.10/dist-packages/met/arl/arlprofiler/profiler.py", line 162, in profile lmd = self._load(output_filename, met_file['first_hour'], File "/usr/local/lib/python3.10/dist-packages/met/arl/arlprofiler/profiler.py", line 260, in _load local_hourly_profile = profile.get_hourly_params() File "/usr/local/lib/python3.10/dist-packages/met/arl/arlprofiler/profileparser.py", line 182, in get_hourly_params self.spread_hourly_results() File "/usr/local/lib/python3.10/dist-packages/met/arl/arlprofiler/profileparser.py", line 429, in spread_hourly_results closest_date = sorted(times, key=lambda d:abs(new_datetime - d))[0] IndexError: list index out of range

2024-06-24 13:46:26,003 DEBUG: Status logging disabled - not submitting 'Failure','localmet', 'Die', {}. 2024-06-24 13:46:26,003 DEBUG: Status logging disabled - not submitting 'Failure','Main', 'Die', {}. 2024-06-24 13:46:26,003 DEBUG: Writing to local file: /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_met/SE2018_12_09_met.json 2024-06-24 13:46:26,004 SUMMARY: Fire counts: {'fires': 1, 'locations': 1, 'failed_fires': 0} 2024-06-24 13:46:26,005 SUMMARY: Run complete

Attached Files for Debugging SE2018_12_09_permit.json locations.csv profile.txt

jdubowy commented 3 weeks ago

Thanks for finding the issue in the hourly data interpolation. I've fixed it with https://github.com/pnwairfire/met/commit/b656e6e383d7206b3bd5e3952eb7462bcb6c1cc6 and published bluesky v4.5.21.