Closed daisyhjt closed 4 weeks ago
The problem is that profile.txt
can have asterisk strings of varying lengths, with multiple asterisk strings possibly running into each other as well as into the preceding number values. e.g.
HGTS TEMP UWND VWND WWND RELH TKEN TPOT WDIR WSP
oC m/s m/s mb/h % oK deg m/
1000 166 20.6 0 0 0.79 80.8 0.30 293.8******* 0.
975 384 21.1 2.2 2.1 0 72.8 0 296.4 225.6 3.
950 610 22.6 6.0 1.3 -2.9 57.6 0 300.1 257.3 6.
...
I've updated the profile.txt
parsing code with a hack to better handle the various asterisk string possibilities. A better solution would be to update the profiler code (different repo) that generates profile.txt
to always guarantee spaces between values. Another solution would be to update the parsing code to split the pressure level value strings based the positions of the variable names in the header string.
I've published bluesky v4.5.20
Hi Joel,
Thanks again for your troubleshooting.
However, there is another error related to IndexError in arlprofiler module. Perhaps it is caused by the interpolation function for hourly meteorological profiles. Could you help with this issue?
Detailed Log:
INFO: underlay of /etc/localtime required more than 50 (118) bind mounts 2024-06-24 13:46:24,495 DEBUG: filling in run_id wildcards 2024-06-24 13:46:24,536 SUMMARY: BlueSky v4.5.20 2024-06-24 13:46:24,537 SUMMARY: Input File(s): /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_input/SE2018_12_09_permit.json 2024-06-24 13:46:24,537 SUMMARY: Output File: /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_met/SE2018_12_09_met.json 2024-06-24 13:46:24,537 INFO: input files: ['/home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_input/SE2018_12_09_permit.json'] 2024-06-24 13:46:24,537 INFO: no input: False 2024-06-24 13:46:24,537 INFO: input file failure tolerance: None 2024-06-24 13:46:24,537 INFO: compress: False 2024-06-24 13:46:24,537 INFO: indent: None 2024-06-24 13:46:24,537 INFO: profile output file: None 2024-06-24 13:46:24,537 INFO: config options: {'localmet': {'working_dir': '/home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_met/SE2018_12_09', 'skip_failures': False, 'time_step': '3'}} 2024-06-24 13:46:24,537 INFO: config file options: {'skip_failed_fires': True, 'skip_failed_sources': True, 'fuelbeds': {'fccs_version': 2}, 'ecoregion': {'lookup_implementation': 'ogr', 'skip_failures': True, 'default': 'southern'}, 'consumption': {'consume_settings': {'activity': {'slope': {'default': 5}, 'windspeed': {'default': 6}, 'days_since_rain': {'default': 10}, 'fuel_moisture_10hr_pct': {'default': 50}, 'fm_type': {'default': 'MEAS-Th'}}, 'all': {'fuel_moisture_1000hr_pct': {'default': 30}, 'fuel_moisture_duff_pct': {'default': 75}, 'fuel_moisture_litter_pct': {'default': 16}, 'canopy_consumption_pct': {'default': 0}, 'shrub_blackened_pct': {'default': 50}, 'pile_blackened_pct': {'default': 0}}}}, 'emissions': {'model': 'prichard-oneill', 'include_emissions_details': False}, 'timeprofile': {'model': 'default'}, 'plumerise': {'model': 'feps', 'feps': {'plume_top_behavior': 'Briggs'}}} 2024-06-24 13:46:24,537 INFO: dump config: False 2024-06-24 13:46:24,537 INFO: version: False 2024-06-24 13:46:24,537 INFO: run id: None 2024-06-24 13:46:24,537 INFO: today: None 2024-06-24 13:46:24,537 INFO: module: ['localmet'] 2024-06-24 13:46:24,537 INFO: log level: 10 2024-06-24 13:46:24,537 INFO: log file: None 2024-06-24 13:46:24,537 INFO: log message format: None 2024-06-24 13:46:24,538 DEBUG: Loading local file: /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_input/SE2018_12_09_permit.json 2024-06-24 13:46:24,538 DEBUG: Failed to gzip.decompress /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_input/SE2018_12_09_permit.json: Not a gzipped file (b'{\n') 2024-06-24 13:46:24,538 INFO: input /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_input/SE2018_12_09_permit.json not gzip'd 2024-06-24 13:46:24,539 INFO: Loaded /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_input/SE2018_12_09_permit.json 2024-06-24 13:46:24,568 DEBUG: Status logging disabled - not submitting 'Good','Main', 'Start', {}. 2024-06-24 13:46:24,568 SUMMARY: Modules to be run: localmet 2024-06-24 13:46:24,568 DEBUG: Status logging disabled - not submitting 'Good','localmet', 'Start', {}. 2024-06-24 13:46:24,568 SUMMARY: Running module localmet 2024-06-24 13:46:24,568 DEBUG: Parsing met file specifications 2024-06-24 13:46:24,643 DEBUG: Extracting localmet data for 1 locations 2024-06-24 13:46:24,644 DEBUG: chdir to working directory /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_met/SE2018_12_09/0 2024-06-24 13:46:24,645 DEBUG: Calling 'bulk_profiler_csv -d/scratch/jh94030/WPS-input/Data/NAM/2018/ -f20181209_nam12 -w2 -t3 -i/home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_met/SE2018_12_09/0/locations.csv -p/home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_met/SE2018_12_09/0/profile.txt' 2024-06-24 13:46:25,918 DEBUG: Loading /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_met/SE2018_12_09/0/profile.txt 2024-06-24 13:46:25,985 DEBUG: chdir back to original directory /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data 2024-06-24 13:46:26,001 DEBUG: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/bluesky-4.5.20-py3.10.egg/bluesky/modules/localmet.py", line 51, in run LocalmetRunner(fires_manager).run() File "/usr/local/lib/python3.10/dist-packages/bluesky-4.5.20-py3.10.egg/bluesky/modules/localmet.py", line 99, in run localmet = arl_profiler.profile( File "/usr/local/lib/python3.10/dist-packages/met/arl/arlprofiler/profiler.py", line 162, in profile lmd = self._load(output_filename, met_file['first_hour'], File "/usr/local/lib/python3.10/dist-packages/met/arl/arlprofiler/profiler.py", line 260, in _load local_hourly_profile = profile.get_hourly_params() File "/usr/local/lib/python3.10/dist-packages/met/arl/arlprofiler/profileparser.py", line 182, in get_hourly_params self.spread_hourly_results() File "/usr/local/lib/python3.10/dist-packages/met/arl/arlprofiler/profileparser.py", line 429, in spread_hourly_results closest_date = sorted(times, key=lambda d:abs(new_datetime - d))[0] IndexError: list index out of range
2024-06-24 13:46:26,002 ERROR: list index out of range 2024-06-24 13:46:26,003 DEBUG: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/bluesky-4.5.20-py3.10.egg/bluesky/models/fires.py", line 536, in run self._modules[i].run(self) File "/usr/local/lib/python3.10/dist-packages/bluesky-4.5.20-py3.10.egg/bluesky/modules/localmet.py", line 51, in run LocalmetRunner(fires_manager).run() File "/usr/local/lib/python3.10/dist-packages/bluesky-4.5.20-py3.10.egg/bluesky/modules/localmet.py", line 99, in run localmet = arl_profiler.profile( File "/usr/local/lib/python3.10/dist-packages/met/arl/arlprofiler/profiler.py", line 162, in profile lmd = self._load(output_filename, met_file['first_hour'], File "/usr/local/lib/python3.10/dist-packages/met/arl/arlprofiler/profiler.py", line 260, in _load local_hourly_profile = profile.get_hourly_params() File "/usr/local/lib/python3.10/dist-packages/met/arl/arlprofiler/profileparser.py", line 182, in get_hourly_params self.spread_hourly_results() File "/usr/local/lib/python3.10/dist-packages/met/arl/arlprofiler/profileparser.py", line 429, in spread_hourly_results closest_date = sorted(times, key=lambda d:abs(new_datetime - d))[0] IndexError: list index out of range
2024-06-24 13:46:26,003 DEBUG: Status logging disabled - not submitting 'Failure','localmet', 'Die', {}. 2024-06-24 13:46:26,003 DEBUG: Status logging disabled - not submitting 'Failure','Main', 'Die', {}. 2024-06-24 13:46:26,003 DEBUG: Writing to local file: /home/jh94030/scripts/python/rxfire2018/RxFireEmissionCode/RxFireEmission/permit_data/bsp_met/SE2018_12_09_met.json 2024-06-24 13:46:26,004 SUMMARY: Fire counts: {'fires': 1, 'locations': 1, 'failed_fires': 0} 2024-06-24 13:46:26,005 SUMMARY: Run complete
Attached Files for Debugging SE2018_12_09_permit.json locations.csv profile.txt
Thanks for finding the issue in the hourly data interpolation. I've fixed it with https://github.com/pnwairfire/met/commit/b656e6e383d7206b3bd5e3952eb7462bcb6c1cc6 and published bluesky v4.5.21.
Hi there,
I'm using the NAM 12 km archived meteorology data (https://www.ready.noaa.gov/archives.php) to run bluesky. However, not all cases run successfully. For some days, I encounter an error: IndexError: list index out of range in the _parse_hourlytext function of the arlprofiler module. Thus it returns exactly empty meteorology json data.
Command Used:
Detailed Log:
Attached Files for Debugging
SE2018_05_11_permit.json permit_config.json locations.csv profile.txt
I guess the error is likely from a mismatch between the expected number of columns in a few lines of the profile.txt file and the actual number of columns. Since I'm limited in my ability to modify the development code on HPC, I was wondering if there is any help to identify and fix the cause of this error?
Thank you in advance!