search

singularity-energy / open-grid-emissions

Tools for producing high-quality hourly generation and emissions data for U.S. electric grids
MIT License
67 stars 4 forks source link

Fix incomplete timeseries warnings #316

Open grgmiller opened 7 months ago

grgmiller commented 7 months ago

When running the data pipeline for 2021, there are several warnings appearing for various data outputs regarding incomplete timeseries data:

Data quality metric export:

2023-06-09 20:20:56,968 [INFO] oge.output_data:158 Exporting hourly_profile_method to data/results/2021/data_quality_metrics
2023-06-09 20:20:58,949 [WARNING] oge.validation:792 There are incomplete timeseries for the following ['plant_id_eia'] groups in shaped_eia_data
2023-06-09 20:20:58,949 [WARNING] oge.validation:795 
                          count                       min                       max  expected_num_hours
plant_id_eia report_date                                                                               
903212       2021-11-01     719 2021-11-01 08:00:00+00:00 2021-12-01 08:00:00+00:00                 721
909312       2021-11-01     719 2021-11-01 05:00:00+00:00 2021-12-01 05:00:00+00:00                 721
913612       2021-11-01     719 2021-11-01 03:00:00+00:00 2021-12-01 03:00:00+00:00                 721
[...]

Upon combining hourly plant data:

2023-06-09 20:21:59,875 [INFO] oge.data_pipeline:542 16. Combining plant-level hourly data
2023-06-09 20:24:04,908 [WARNING] oge.validation:773 There are incomplete timeseries for the following ['plant_id_eia'] groups in combined_plant_data
2023-06-09 20:24:04,970 [WARNING] oge.validation:776 
              datetime_utc  expected_num_hours
plant_id_eia                                  
9                     5136                8760
47                    6553                8760
51                    6529                8760
59                    7296                8760
124                   5808                8760
170                   5064                8760
201                   8017                8760
203                   6552                8760
[...]

For certain BA-level results:

2023-06-09 20:24:22,416 [INFO] oge.output_data:131 Exporting AECI to data/results/2021/power_sector_data/annual/
2023-06-09 20:24:22,448 [INFO] oge.validation:149 Checking that fuel and emissions values are positive...  
2023-06-09 20:24:22,463 [INFO] oge.validation:175 OK
2023-06-09 20:24:22,463 [INFO] oge.validation:181 Checking that no values are missing...  
2023-06-09 20:24:22,463 [INFO] oge.validation:198 OK
2023-06-09 20:24:22,683 [WARNING] oge.validation:773 There are incomplete timeseries for the following ['fuel_category'] groups in power sector hourly ba table
2023-06-09 20:24:22,683 [WARNING] oge.validation:776 
               datetime_utc  expected_num_hours
fuel_category                                  
wind                   8758                8760
2023-06-09 20:26:00,361 [INFO] oge.output_data:131 Exporting LDWP to data/results/2021/power_sector_data/annual/
2023-06-09 20:26:00,392 [INFO] oge.validation:149 Checking that fuel and emissions values are positive...  
2023-06-09 20:26:00,408 [INFO] oge.validation:175 OK
2023-06-09 20:26:00,408 [INFO] oge.validation:181 Checking that no values are missing...  
2023-06-09 20:26:00,408 [INFO] oge.validation:198 OK
2023-06-09 20:26:00,612 [WARNING] oge.validation:773 There are incomplete timeseries for the following ['fuel_category'] groups in power sector hourly ba table
2023-06-09 20:26:00,612 [WARNING] oge.validation:776 
               datetime_utc  expected_num_hours
fuel_category                                  
natural_gas            8761                8760
total                  8761                8760