Keep exceptions logs

[ekatef]

Closes #897

Changes proposed in this Pull Request

Add an exception hook to stream exception traceback into the logs.

Checklist

• [x] I consent to the release of this PR's code under the AGPLv3 license and non-code contributions under CC0-1.0 and CC-BY-4.0.
• [x] I tested my contribution locally and it seems to work fine.
• [x] Code and workflow changes are sufficiently documented.
• [ ] Newly introduced dependencies are added to envs/environment.yaml and doc/requirements.txt.
• [ ] Changes in configuration options are added in all of config.default.yaml and config.tutorial.yaml.
• [ ] Add a test config or line additions to test/ (note tests are changing the config.tutorial.yaml)
• [ ] Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
• [x] A note for the release notes doc/release_notes.rst is amended in the format of previous release notes, including reference to the requested PR.

[ekatef]

Playing with structuring the code. It appears that python logging is a bit tricky to deal with due logger hierarchy,. The current implementation results in an empty log, but works if definition of create_logger() is moved to build_shapes module itself.

[ekatef]

Cool @ekatef :D Few minor comments, I think we are almost there :D

Agree, it looks like the issue has been fixed :) Thanks a lot for support, as always.

Have added an exception hook to all the scripts, except build_test_configs.py, as it doesn't contained any logging settings. Not sure if it was intentional, and happy to fix in case it is needed.

[ekatef]

We are there :) May you share few logs to me? I'd like to see a build_shapes (example) and build_renewable_profile if possible.

Many thanks!

Have improved a definition of create_logger(), as you suggested. As the next step, I'll update logging setup in all the scripts (which now can be as short as it was initially). Thanks a lot for supervision! :)

Sharing the logs bellow.

build_shapes

WARNING:snakemake.logging:Stage 1 of 4: Create country shapes
WARNING:snakemake.logging:Stage 2 of 4: Create offshore shapes
WARNING:snakemake.logging:Stage 4/4: Creation GADM GeoDataFrame
WARNING:snakemake.logging:Stage 4/4 POP: Add population data to GADM GeoDataFrame
WARNING:snakemake.logging:Stage 4/4: Add gdp data to GADM GeoDataFrame
WARNING:snakemake.logging:Stage 4/4: Access to GDP raster data
CRITICAL:snakemake.logging:Stage 4/4: File GDP_PPP_1990_2015_5arcmin_v2.tif not found, the file will be produced by processing GDP_PPP_1990_2015_5arcmin_v2.nc
WARNING:snakemake.logging:Stage 4/4: Access to GDP raster data
CRITICAL:snakemake.logging:Stage 3/4 GDP data of year False not found, selected the most recent data (2015)

build_renewable_profiles (onwind)

INFO:distributed.http.proxy:To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
INFO:distributed.scheduler:State start
INFO:distributed.scheduler:  Scheduler at:     tcp://127.0.0.1:50156
INFO:distributed.scheduler:  dashboard at:  http://127.0.0.1:8787/status
INFO:distributed.nanny:        Start Nanny at: 'tcp://127.0.0.1:50159'
INFO:distributed.scheduler:Register worker <WorkerState 'tcp://127.0.0.1:50162', name: 0, status: init, memory: 0, processing: 0>
INFO:distributed.scheduler:Starting worker compute stream, tcp://127.0.0.1:50162
INFO:distributed.core:Starting established connection to tcp://127.0.0.1:50164
INFO:distributed.scheduler:Receive client connection: Client-38bafc6c-75e8-11ee-a0a5-f64e6159bb86
INFO:distributed.core:Starting established connection to tcp://127.0.0.1:50165
WARNING:snakemake.logging:Calculate landuse availabilities...
WARNING:snakemake.logging:Completed availability calculation (22.84s)
INFO:atlite.convert:Convert and aggregate 'wind'.
INFO:atlite.convert:Convert and aggregate 'wind'.
WARNING:snakemake.logging:Calculating maximal capacity per bus (method 'simple')
WARNING:snakemake.logging:Calculate average distances.
INFO:distributed.nanny:Closing Nanny at 'tcp://127.0.0.1:50159'. Reason: nanny-close
INFO:distributed.nanny:Nanny asking worker to close. Reason: nanny-close
INFO:distributed.core:Received 'close-stream' from tcp://127.0.0.1:50164; closing.
INFO:distributed.scheduler:Remove worker <WorkerState 'tcp://127.0.0.1:50162', name: 0, status: closing, memory: 0, processing: 0> (stimulus_id='handle-worker-cleanup-1698535104.803447')
INFO:distributed.scheduler:Lost all workers
INFO:distributed.scheduler:Scheduler closing...
INFO:distributed.scheduler:Scheduler closing all comms

[ekatef]

Uncaught exceptions are also captured by the logs. For example, there is a log after an error provoked in build_shapes by referring to undefined variables x and y in print(x + y):

ERROR:_helpers:An error happened in module '~/_github_/pypsa-earth/scripts/build_shapes.py', function '<module>': name 'x' is not defined
Traceback (most recent call last):
  File "~/_github_/pypsa-earth/.snakemake/scripts/tmph3o6bx36.build_shapes.py", line 925, in <module>
    print(x + y)
NameError: name 'x' is not defined

[ekatef]

Fantastic @ekatef :D From now on, logging will be easier! :D Ready to merge!

Thanks a lot for amazing support @davide-f 🙂