search

NREL / flasc

A rich floris-driven suite for SCADA analysis
https://nrel.github.io/flasc/
BSD 3-Clause "New" or "Revised" License
30 stars 18 forks source link

[BUG] Example 01 is crashing #149

Closed paulf81 closed 8 months ago

paulf81 commented 8 months ago

Is there an existing issue for this?

Current Behavior

The example is examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb is crashing on my machine when I try to run it. The error occurs in cell 5, pasting error below.

Think on the one hand should fix the issue and then also think a little bit more about including some example running within CI a la Issue #109. What do you think @misi9170 ?

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
[/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb) Cell 9 line 2
    [248](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=247)     df_pow_curve = ws_pow_filtering.pw_curve_df
    [250](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=249)     return df, df_pow_curve
--> [253](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=252) df_scada_filtered, df_pow_curve = filter_by_ws_pow_curve(
    [254](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=253)     df=df_scada_formatted.copy(),
    [255](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=254)     plot_figures=plot_figures,
    [256](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=255)     save_figures=save_figures,
    [257](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=256)     use_bokeh_for_plots=use_bokeh_plotting_library,
    [258](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=257) )

[/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb) Cell 9 line 1
     [93](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=92) ws_pow_filtering.filter_by_condition(
     [94](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=93)     condition=(ws_pow_filtering.df["ws_{:03d}".format(ti)] > 4.0)
     [95](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=94)     & (ws_pow_filtering.df["pow_{:03d}".format(ti)] < 1.0),
   (...)
     [98](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=97)     verbose=True,
     [99](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=98) )
    [101](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=100) # Other common filters here are based on turbine-specific/OEM-specific flags. For example,
    [102](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=101) # a 'run counter' may indicate how many seconds of a 10-minute period the turbine was
    [103](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=102) # reporting valid measurements. By filtering for the condition that that value is 600, or
   (...)
    [117](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=116) # In this case, we have an operational_status flag. If that has a 'False' value, then we
    [118](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=117) # mark those measurements as faulty.
--> [119](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=118) ws_pow_filtering.filter_by_condition(
    [120](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=119)     condition=(ws_pow_filtering.df["is_operation_normal_{:03d}".format(ti)] is False),
    [121](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=120)     label="Self-flagged (is_operation_normal==False)",
    [122](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=121)     ti=ti,
    [123](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=122)     verbose=True,
    [124](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=123) )
    [126](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=125) # Filter for sensor-stuck faults
    [127](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=126) ws_pow_filtering.filter_by_sensor_stuck_faults(
    [128](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=127)     columns=["wd_{:03d}".format(ti), "ws_{:03d}".format(ti)],
    [129](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=128)     ti=ti,
   (...)
    [132](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=131)     plot=False,
    [133](vscode-notebook-cell:/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/00_filter_ws_power_curves.ipynb#X11sZmlsZQ%3D%3D?line=132) )

File [~/Projects/FLORIS/flasc/flasc/turbine_analysis/ws_pow_filtering.py:244](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/Projects/FLORIS/flasc/flasc/turbine_analysis/ws_pow_filtering.py:244), in ws_pw_curve_filtering.filter_by_condition(self, condition, label, ti, verbose, apply_filters_to_df)
    [242](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/Projects/FLORIS/flasc/flasc/turbine_analysis/ws_pow_filtering.py:242) # Mark data as faulty on the dataframe
    [243](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/Projects/FLORIS/flasc/flasc/turbine_analysis/ws_pow_filtering.py:243) N_pre = [dff.df_get_no_faulty_measurements(df_in, tii) for tii in ti]
--> [244](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/Projects/FLORIS/flasc/flasc/turbine_analysis/ws_pow_filtering.py:244) df_out = dff.df_mark_turbdata_as_faulty(df=df_in, cond=condition, turbine_list=ti)
    [246](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/Projects/FLORIS/flasc/flasc/turbine_analysis/ws_pow_filtering.py:246) # Print the reduction in useful data to the console, if verbose
    [247](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/Projects/FLORIS/flasc/flasc/turbine_analysis/ws_pow_filtering.py:247) if verbose:

File [~/Projects/FLORIS/flasc/flasc/dataframe_operations/dataframe_filtering.py:138](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/Projects/FLORIS/flasc/flasc/dataframe_operations/dataframe_filtering.py:138), in df_mark_turbdata_as_faulty(df, cond, turbine_list, exclude_columns)
    [135](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/Projects/FLORIS/flasc/flasc/dataframe_operations/dataframe_filtering.py:135) for ti in turbine_list:
    [136](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/Projects/FLORIS/flasc/flasc/dataframe_operations/dataframe_filtering.py:136)     cols = [s for s in df.columns if s[-4::] == ('_%03d' % ti)
    [137](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/Projects/FLORIS/flasc/flasc/dataframe_operations/dataframe_filtering.py:137)             and s not in exclude_columns]
--> [138](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/Projects/FLORIS/flasc/flasc/dataframe_operations/dataframe_filtering.py:138)     df.loc[cond, cols] = None  # Delete measurements
    [140](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/Projects/FLORIS/flasc/flasc/dataframe_operations/dataframe_filtering.py:140) return df

File [~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:885](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:885), in _LocationIndexer.__setitem__(self, key, value)
    [882](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:882) self._has_valid_setitem_indexer(key)
    [884](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:884) iloc = self if self.name == "iloc" else self.obj.iloc
--> [885](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:885) iloc._setitem_with_indexer(indexer, value, self.name)

File [~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:1795](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:1795), in _iLocIndexer._setitem_with_indexer(self, indexer, value, name)
   [1791](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:1791) for i, idx in enumerate(indexer):
   [1792](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:1792)     if isinstance(idx, dict):
   [1793](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:1793)         # reindex the axis to the new value
   [1794](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:1794)         # and set inplace
-> [1795](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:1795)         key, _ = convert_missing_indexer(idx)
   [1797](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:1797)         # if this is the items axes, then take the main missing
   [1798](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:1798)         # path first
   [1799](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:1799)         # this correctly sets the dtype and avoids cache issues
   [1800](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:1800)         # essentially this separates out the block that is needed
   [1801](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:1801)         # to possibly be modified
   [1802](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:1802)         if self.ndim > 1 and i == info_axis:
   [1803](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:1803)             # add the new item, and set the value
   [1804](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:1804)             # must have all defined axes if we have a scalar
   [1805](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:1805)             # or a list-like on the non-info axes if we have a
   [1806](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:1806)             # list-like

File [~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:2607](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:2607), in convert_missing_indexer(indexer)
   [2604](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:2604)     indexer = indexer["key"]
   [2606](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:2606)     if isinstance(indexer, bool):
-> [2607](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:2607)         raise KeyError("cannot use a single bool to index into setitem")
   [2608](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:2608)     return indexer, True
   [2610](https://file+.vscode-resource.vscode-cdn.net/Users/pfleming/Projects/FLORIS/flasc/examples_artificial_data/01_raw_data_processing/~/opt/anaconda3/envs/floris/lib/python3.10/site-packages/pandas/core/indexing.py:2610) return indexer, False

KeyError: 'cannot use a single bool to index into setitem'

Expected Behavior

No response

Steps To Reproduce

No response

Environment

- OS:
- pip environment (can be retrieved with `pip list`):

Anything else?

No response

paulf81 commented 8 months ago

Addressed by #154