singularity-energy / open-grid-emissions

Tools for producing high-quality hourly generation and emissions data for U.S. electric grids
MIT License
72 stars 5 forks source link

Small logging updates #289

Closed grgmiller closed 1 year ago

grgmiller commented 1 year ago

Currently, the pipeline outputs the log file to outputs_folder("data_pipeline.log"), but it would be better if the log were saved to the directory for the specific year for which the pipeline was run, e.g. outputs_folder("{year}/data_pipeline.log"). However, the root logger is configured outside of main() where the year was defined. To get around this, I just made the final step of the data pipeline shutil.move(outputs_folder("data_pipeline.log"), outputs_folder(f"{year}/data_pipeline.log")). However, I'm not sure if this is a good way to do this. @miloknowles would there be a better / easy way to write directly to the specific year folder?

This PR also makes a few small formatting changes:

miloknowles commented 1 year ago

I would just move the configure_root_logger call inside of main then, so that you can include the year in the filepath. No harm in doing that.

Btw, the only reason this wouldn't work is if configure_root_logger was called somewhere else first with a different filepath (e.g in one of the imported modules). That isn't the case currently, but I would remove the file src/__init__.py just to be safe.

grgmiller commented 1 year ago

Hi @miloknowles I updated the location of the logger configuration to be under main() in data_pipeline. Could you please make a look and make sure this looks ok?