GHGRP 2017 is not working with stewi or stewi combo.

[tjlca]

Detailed error is here -

runfile('/Users/tghosh/OneDrive - NREL/work_NREL/FECM/Industrial-Emissions-Inventory/fecm_data_exploration.py', wdir='/Users/tghosh/OneDrive - NREL/work_NREL/FECM/Industrial-Emissions-Inventory', current_namespace=True) /Users/tghosh/miniconda3/envs/stewi/lib/python3.9/site-packages/requests/init.py:109: RequestsDependencyWarning: urllib3 (1.26.9) or chardet (5.0.0)/charset_normalizer (2.0.12) doesn't match a supported version! warnings.warn( INFO GHGRP_2017 not found in /Users/tghosh/Library/Application Support/stewi/flowbyfacility INFO requested inventory does not exist in local directory, it will be generated... INFO downloading and processing GHGRP data to /Users/tghosh/Library/Application Support/stewi/GHGRP Data Files/tables/2017 ERROR error in url request Traceback (most recent call last): File "/Users/tghosh/OneDrive - NREL/work_NREL/FECM/Industrial-Emissions-Inventory/stewi/GHGRP.py", line 130, in get_row_count table_count = int(table_count[0].firstChild.nodeValue) IndexError: list index out of range --- Logging error --- Traceback (most recent call last): File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/logging/init.py", line 1083, in emit msg = self.format(record) File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/logging/init.py", line 927, in format return fmt.format(record) File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/logging/init.py", line 663, in format record.message = record.getMessage() File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/logging/init.py", line 367, in getMessage msg = msg % self.args TypeError: %i format: a number is required, not NodeList Call stack: File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/site-packages/spyder_kernels/console/main.py", line 24, in start.main() File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/site-packages/spyder_kernels/console/start.py", line 332, in main kernel.start() File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/site-packages/ipykernel/kernelapp.py", line 712, in start self.io_loop.start() File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 199, in start self.asyncio_loop.run_forever() File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/asyncio/base_events.py", line 601, in run_forever self._run_once() File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/asyncio/base_events.py", line 1905, in _run_once handle._run() File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/asyncio/events.py", line 80, in _run self._context.run(self._callback, self._args) File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 510, in dispatch_queue await self.process_one() File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 499, in process_one await dispatch(args) File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 406, in dispatch_shell await result File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 730, in execute_request reply_content = await reply_content File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/site-packages/ipykernel/ipkernel.py", line 383, in do_execute res = shell.run_cell( File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/site-packages/ipykernel/zmqshell.py", line 528, in run_cell return super().run_cell(*args, **kwargs) File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2975, in run_cell result = self._run_cell( File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3030, in _run_cell return runner(coro) File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/site-packages/IPython/core/async_helpers.py", line 78, in _pseudo_sync_runner coro.send(None) File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3257, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3473, in run_ast_nodes if (await self.runcode(code, result, async=asy)): File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3553, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/var/folders/sm/spdh5zkx26v6vh_fk8w7p8456l6sp0/T/ipykernel_18885/414800848.py", line 1, in <cell line: 1> runfile('/Users/tghosh/OneDrive - NREL/work_NREL/FECM/Industrial-Emissions-Inventory/fecm_data_exploration.py', wdir='/Users/tghosh/OneDrive - NREL/work_NREL/FECM/Industrial-Emissions-Inventory', current_namespace=True) File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/site-packages/spyder_kernels/customize/spydercustomize.py", line 585, in runfile exec_code(file_code, filename, ns_globals, ns_locals, File "/Users/tghosh/miniconda3/envs/stewi/lib/python3.9/site-packages/spyder_kernels/customize/spydercustomize.py", line 465, in exec_code exec(compiled, ns_globals, ns_locals) File "/Users/tghosh/OneDrive - NREL/work_NREL/FECM/Industrial-Emissions-Inventory/fecm_data_exploration.py", line 38, in save_data(inventory,year) File "/Users/tghosh/OneDrive - NREL/work_NREL/FECM/Industrial-Emissions-Inventory/fecm_data_exploration.py", line 14, in save_data flow_by_facility = stewi.getInventory(inventory, year, 'flowbyfacility',filters=['filter_for_LCI']) File "/Users/tghosh/OneDrive - NREL/work_NREL/FECM/Industrial-Emissions-Inventory/stewi/init.py", line 80, in getInventory inventory = read_inventory(inventory_acronym, year, f, File "/Users/tghosh/OneDrive - NREL/work_NREL/FECM/Industrial-Emissions-Inventory/stewi/globals.py", line 322, in read_inventory generate_inventory(inventory_acronym, year) File "/Users/tghosh/OneDrive - NREL/work_NREL/FECM/Industrial-Emissions-Inventory/stewi/globals.py", line 356, in generate_inventory GHGRP.main(Option = 'A', Year = [year]) File "/Users/tghosh/OneDrive - NREL/work_NREL/FECM/Industrial-Emissions-Inventory/stewi/GHGRP.py", line 696, in main ghgrp1 = download_and_parse_subpart_tables(year, m) File "/Users/tghosh/OneDrive - NREL/work_NREL/FECM/Industrial-Emissions-Inventory/stewi/GHGRP.py", line 286, in download_and_parse_subpart_tables table_df = import_or_download_table(filepath, subpart_emissions_table, File "/Users/tghosh/OneDrive - NREL/work_NREL/FECM/Industrial-Emissions-Inventory/stewi/GHGRP.py", line 240, in import_or_download_table log.info('Downloading %s (rows: %i)', table, row_count) Message: 'Downloading %s (rows: %i)' Arguments: ('C_CONFIGURATION_LEVEL_INFO', []) Traceback (most recent call last):

File "/Users/tghosh/OneDrive - NREL/work_NREL/FECM/Industrial-Emissions-Inventory/fecm_data_exploration.py", line 38, in save_data(inventory,year)

File "/Users/tghosh/OneDrive - NREL/work_NREL/FECM/Industrial-Emissions-Inventory/fecm_data_exploration.py", line 14, in save_data flow_by_facility = stewi.getInventory(inventory, year, 'flowbyfacility',filters=['filter_for_LCI'])

File "/Users/tghosh/OneDrive - NREL/work_NREL/FECM/Industrial-Emissions-Inventory/stewi/init.py", line 80, in getInventory inventory = read_inventory(inventory_acronym, year, f,

File "/Users/tghosh/OneDrive - NREL/work_NREL/FECM/Industrial-Emissions-Inventory/stewi/globals.py", line 322, in read_inventory generate_inventory(inventory_acronym, year)

File "/Users/tghosh/OneDrive - NREL/work_NREL/FECM/Industrial-Emissions-Inventory/stewi/globals.py", line 356, in generate_inventory GHGRP.main(Option = 'A', Year = [year])

File "/Users/tghosh/OneDrive - NREL/work_NREL/FECM/Industrial-Emissions-Inventory/stewi/GHGRP.py", line 696, in main ghgrp1 = download_and_parse_subpart_tables(year, m)

File "/Users/tghosh/OneDrive - NREL/work_NREL/FECM/Industrial-Emissions-Inventory/stewi/GHGRP.py", line 286, in download_and_parse_subpart_tables table_df = import_or_download_table(filepath, subpart_emissions_table,

File "/Users/tghosh/OneDrive - NREL/work_NREL/FECM/Industrial-Emissions-Inventory/stewi/GHGRP.py", line 242, in import_or_download_table table_df = download_chunks(table=table, table_count=row_count, m=m,

File "/Users/tghosh/OneDrive - NREL/work_NREL/FECM/Industrial-Emissions-Inventory/stewi/GHGRP.py", line 141, in download_chunks while row_start <= table_count:

TypeError: '<=' not supported between instances of 'int' and 'NodeList'

[tjlca]

2018 and 2019 is working without any issues.

[bl-young]

It looks like 2017 was running fine last checked around two week ago. https://github.com/USEPA/standardizedinventories/runs/6874609232?check_suite_focus=true#step:6:251

I will check on my end when I get a chance.

[bl-young]

In the meantime 2017 files can be downloaded here: https://edap-ord-data-commons.s3.amazonaws.com/index.html?prefix=stewi/ or with stewi.getInventory('GHGRP', 2017, download_if_missing=True)

https://github.com/USEPA/standardizedinventories/blob/698a0b316f5cd999f33ae8ddffe0a2fca94f2bf9/stewi/__init__.py#L62-L64

[tjlca]

@bl-young We had it running fine 2 weeks ago too.

[tjlca]

stewi.getInventory('GHGRP', 2017, download_if_missing=True)

Same error with this command.

[bl-young]

confirming that I can replicate your error. Shows up for 2017 but not for 2018. This suggests to me that something is wrong with the data source itself.

Same error with this command.

Depending on which datasets you need, go ahead and run this command for all of the types 'facility', 'flow', etc and that will store them locally then you can run the rest of your code with already downloaded datasets instead of having to generate them yourself

[tjlca]

Currently TRI data source for 2017 is showing issues also.

[bl-young]

@TJTapajyoti the error you raised seems to have been resolved by the data provider. A new GHGRP error (#122) has also been fixed. I assume the TRI issue you were having is the same as in #121. Please let me know if any issues remain as far as you know.