Gailin/evaluate 2021 lags

[gailin-p]

Notebook update

identify_eia930_time_lags.ipynb is:

• updated to use current OGE output formats
• Added documentation describing steps for manual checking/cleaning aided by lagged correlation
• Removed unused code

Code update

eia930.manual_930_adjust is fixed to not adjust SC data after 2020, since the EIA BALANCE files are fixed after Dec 31, 2020.

[gailin-p]

Draft because still need to test pipeline

[grgmiller]

I changed the base to merge into my patch_negative_efs branch. I'm going to test this overnight

[grgmiller]

I'm getting the following error when running the data cleaning:

12. Cleaning EIA-930 data
    Adjusting EIA-930 time stamps
A:\miniconda3\envs\open_grid_emissions\lib\site-packages\pandas\core\arrays\datetimelike.py:1189: PerformanceWarning: Adding/subtracting object-dtype array to DatetimeArray not vectorized.
  warnings.warn(
Traceback (most recent call last):
  File "a:\GitHub\open-grid-emissions\src\data_pipeline.py", line 539, in <module>
    main()
  File "a:\GitHub\open-grid-emissions\src\data_pipeline.py", line 334, in main
    eia930.clean_930(year, small=args.small, path_prefix=path_prefix)
  File "a:\GitHub\open-grid-emissions\src\eia930.py", line 146, in clean_930
    df = manual_930_adjust(df)
  File "a:\GitHub\open-grid-emissions\src\eia930.py", line 423, in manual_930_adjust
    sc_idx = pd.DatetimeIndex(sc_dat.index + sc_offsets)  # make shifted dates
  File "A:\miniconda3\envs\open_grid_emissions\lib\site-packages\pandas\core\ops\common.py", line 70, in new_method
    return method(self, other)
  File "A:\miniconda3\envs\open_grid_emissions\lib\site-packages\pandas\core\arraylike.py", line 104, in __radd__
    return self._arith_method(other, roperator.radd)
  File "A:\miniconda3\envs\open_grid_emissions\lib\site-packages\pandas\core\series.py", line 5639, in _arith_method
    return base.IndexOpsMixin._arith_method(self, other, op)
  File "A:\miniconda3\envs\open_grid_emissions\lib\site-packages\pandas\core\base.py", line 1295, in _arith_method
    result = ops.arithmetic_op(lvalues, rvalues, op)
  File "A:\miniconda3\envs\open_grid_emissions\lib\site-packages\pandas\core\ops\array_ops.py", line 216, in arithmetic_op
    res_values = op(left, right)
  File "A:\miniconda3\envs\open_grid_emissions\lib\site-packages\pandas\core\roperator.py", line 9, in radd
    return right + left
  File "A:\miniconda3\envs\open_grid_emissions\lib\site-packages\pandas\core\ops\common.py", line 70, in new_method
    return method(self, other)
  File "A:\miniconda3\envs\open_grid_emissions\lib\site-packages\pandas\core\arrays\datetimelike.py", line 1280, in __add__
    result = self._addsub_object_array(other, operator.add)
  File "A:\miniconda3\envs\open_grid_emissions\lib\site-packages\pandas\core\arrays\datetimelike.py", line 1201, in _addsub_object_array
    res_values = op(self.astype("O"), np.asarray(other))
  File "pandas\_libs\tslibs\timestamps.pyx", line 296, in pandas._libs.tslibs.timestamps._Timestamp.__add__
TypeError: Addition/subtraction of integers and integer-arrays with Timestamp is no longer supported.  Instead of adding/subtracting `n`, use `n * obj.freq`

[grgmiller]

It looks like the issue here was that instead of: sc_offsets["2021-01-01T00:00":] = 0

it needs to be: sc_offsets["2021-01-01T00:00":] = timedelta(0)