SolarArbiter / solarforecastarbiter-core

Core data gathering, validation, processing, and reporting package for the Solar Forecast Arbiter
https://solarforecastarbiter-core.readthedocs.io
MIT License
33 stars 21 forks source link

replace ARM GHI with Best Estimate GHI #716

Closed wholmgren closed 2 years ago

wholmgren commented 2 years ago
wholmgren commented 2 years ago

Warning added in 32ceb99 fired as expected:

2021-09-07 16:38:44,046 WARNING Duplicate index values in sgpmetE9.b1. Keeping last.

The deduplicator is needed at this level of the code because reference_observations/arm.py concatenates the irradiance and the meteorological data along the columns, and this only works if there are no duplicates in either dataframe. Debatable whether the deduplicator should be here or in reference_observations/arm.py. Going with the most straightforward approach for now.

The first deduplicator added in a268a94 did solve the problem for ARM sites that have duplicates in both the irradiance and the met data. Sentry logs indicate that this problem has also occurred with at least a MIDC site, so let's keep these lines too.