dthierry
commented
12 months ago @calmc I tried the manual download with this branch but I've experienced issues with the FRS part and then the GHGRP.
INFO:root:Combined zip file does not exist. Downloading...
INFO:root:Combined file unzipped.
Traceback (most recent call last):
File "/Users/dthierry/Projects/cm_debug_test/fied_compilation.py", line 1315, in <module>
frs_data = pd.read_csv(
^^^^^^^^^^^^
File "/Users/dthierry/mambaforge/envs/found/lib/python3.11/site-packages/pandas/io/parsers/readers.py", line 948, in read_csv
return _read(filepath_or_buffer, kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/dthierry/mambaforge/envs/found/lib/python3.11/site-packages/pandas/io/parsers/readers.py", line 611, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/dthierry/mambaforge/envs/found/lib/python3.11/site-packages/pandas/io/parsers/readers.py", line 1448, in __init__
self._engine = self._make_engine(f, self.engine)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/dthierry/mambaforge/envs/found/lib/python3.11/site-packages/pandas/io/parsers/readers.py", line 1705, in _make_engine
self.handles = get_handle(
^^^^^^^^^^^
File "/Users/dthierry/mambaforge/envs/found/lib/python3.11/site-packages/pandas/io/common.py", line 863, in get_handle
handle = open(
^^^^^
FileNotFoundError: [Errno 2] No such file or directory: './data/FRS/frs_data_formatted.csv'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/dthierry/mambaforge/envs/found/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 3790, in get_loc
return self._engine.get_loc(casted_key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "index.pyx", line 152, in pandas._libs.index.IndexEngine.get_loc
File "index.pyx", line 158, in pandas._libs.index.IndexEngine.get_loc
TypeError: 'Index([51], dtype='int64')' is an invalid key
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/dthierry/mambaforge/envs/found/lib/python3.11/site-packages/pandas/core/frame.py", line 4338, in _set_value
iindex = self.index.get_loc(index)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/dthierry/mambaforge/envs/found/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
self._check_indexing_error(key)
File "/Users/dthierry/mambaforge/envs/found/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 5974, in _check_indexing_error
raise InvalidIndexError(key)
pandas.errors.InvalidIndexError: Index([51], dtype='int64')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/dthierry/Projects/cm_debug_test/fied_compilation.py", line 1323, in <module>
frs_data = frs_methods.import_format_frs(combined=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/dthierry/Projects/cm_debug_test/frs/frs_extraction.py", line 503, in import_format_frs
pgm_data = self.read_frs_csv(
^^^^^^^^^^^^^^^^^^
File "/Users/dthierry/Projects/cm_debug_test/frs/frs_extraction.py", line 378, in read_frs_csv
data = self.format_program_csv(data, programs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/dthierry/Projects/cm_debug_test/frs/frs_extraction.py", line 255, in format_program_csv
data_dict[a].at[use_index, f'PGM_SYS_ID_{a}_additional'] =\
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/dthierry/mambaforge/envs/found/lib/python3.11/site-packages/pandas/core/indexing.py", line 2499, in __setitem__
return super().__setitem__(key, value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/dthierry/mambaforge/envs/found/lib/python3.11/site-packages/pandas/core/indexing.py", line 2455, in __setitem__
self.obj._set_value(*key, value=value, takeable=self._takeable)
File "/Users/dthierry/mambaforge/envs/found/lib/python3.11/site-packages/pandas/core/frame.py", line 4357, in _set_value
raise InvalidIndexError(
pandas.errors.InvalidIndexError: You can only assign a scalar value not a <class 'str'> ```
calmc
Addressed bug in downloading and unzipping GHGRP unit data. Also fixed a few other problems associated with reconciling energy estimates from GHGRP and NEI data. All of the calculations and dataset assembly specified in
fied_compilation.pyshould now run, given the environment and manually-downloaded data specified in the repo's README.@dthierry are you able to review this and compare what changes you made in your branch? Or, maybe switch to this branch and try to run
fied_compilation.pywith the manually-downloaded datasets to see if there are any errors?