google / weather-tools

Tools to make weather data accessible and useful.
https://weather-tools.readthedocs.io/
Apache License 2.0
215 stars 40 forks source link

Beam NameError in `weather-dl`: save_main_session not applied #123

Closed alxmrs closed 2 years ago

alxmrs commented 2 years ago

When running the weather downloader as usual on the main branch:

weather-dl ./configs/era5_example_config_local_run.cfg --local-run

The following error occurs:

...
File "/Users/alxrsngrtn/homebrew/Caskroom/miniconda/base/lib/python3.8/site-packages/future/utils/__init__.py", line 446, in raise_with_traceback
    raise exc.with_traceback(traceback)
  File "apache_beam/runners/common.py", line 1213, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 570, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "/Users/alxrsngrtn/homebrew/Caskroom/miniconda/base/lib/python3.8/site-packages/apache_beam/transforms/core.py", line 1544, in <lambda>
    wrapper = lambda x: [fn(x)]
  File "/Users/alxrsngrtn/homebrew/Caskroom/miniconda/base/lib/python3.8/site-packages/apache_beam/transforms/core.py", line 2385, in <lambda>
    return pcoll | Map(lambda x: (self._key_func()(x), x)) | GroupByKey()
  File "/Users/alxrsngrtn/Github/weather-tools/weather_dl/download_pipeline/pipeline.py", line 86, in subsection_and_request
    subsection = t.cast(str, it.get('parameters', {}).get('__subsection__', 'default'))
NameError: name 't' is not defined [while running 'GroupBy request limits/Map(<lambda at core.py:2385>)'

Troubleshooting this bug is documented here: https://cloud.google.com/dataflow/docs/resources/faq#how_do_i_handle_nameerrors

PR #120 attempted to fix this; however, the error seems to have persisted.

alxmrs commented 2 years ago

I've tested the fix in #124 both on a local and remote run. It seems to have fixed the issue.