search

PyPSA / atlite

Atlite: A Lightweight Python Package for Calculating Renewable Power Potentials and Time Series
https://atlite.readthedocs.io
255 stars 87 forks source link

Cutout.prepare() no reponse #56

Closed fang1103 closed 4 years ago

fang1103 commented 4 years ago

Code

When running the standard pattern for creating an ERA5 cutout (https://atlite.readthedocs.io/en/latest/examples/plotting_with_atlite.html)

import geopandas as gpd
import cartopy.io.shapereader as shpreader
import atlite
import logging

logging.basicConfig(level=logging.INFO)
shpfilename = shpreader.natural_earth(resolution='10m',
                                      category='cultural',
                                      name='admin_0_countries')
reader = shpreader.Reader(shpfilename)
UkIr = gpd.GeoSeries({r.attributes['NAME_EN']: r.geometry
                      for r in reader.records()},
                     crs={'init': 'epsg:4326'}
                     ).reindex(['United Kingdom', 'Ireland'])

# Define the cutout; this will not yet trigger any major operations
cutout = atlite.Cutout(name="uk-2011-01",
                       cutout_dir="./",
                       module="era5",
                       bounds=UkIr.unary_union.bounds,
                       years=slice(2011,2011),
                       months=slice(1,1))

# This is where all the work happens.
cutout.prepare()

Error

I got the following message when I run the code in Spyder:


> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs\crs.py:55: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6
>   return _prepare_from_string(" ".join(pjargs))
> INFO:atlite.cutout:Cutout uk-2011-01 not found in directory ./, building new one
> INFO:cdsapi:Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels
> INFO:cdsapi:Request is completed
> INFO:atlite.datasets.era5:Downloading request for 9 variables to C:\Users\Yiping\AppData\Local\Temp\tmpqrvzeks3.nc
> INFO:cdsapi:Downloading http://136.156.132.235/cache-compute-0006/cache/data7/adaptor.mars.internal-1583414461.1578808-27797-3-b18c0f99-b457-440f-990b-0daa709defa9.nc to C:\Users\Yiping\AppData\Local\Temp\tmpqrvzeks3.nc (126.6K)
> INFO:cdsapi:Download rate 846.1K/s        
> INFO:atlite.preparation:Starting preparation of cutout 'uk-2011-01'
> INFO:atlite.preparation:1 tasks have been collected. Starting running them on all processors.

It seems to be working, but the problem is: my computer CPU keeps running with full speed but without any output for 2 hours.

Then, I run the code in the command line and get the following error. It seems it is the problem of the Initialization of Cutout with "months = slice(1,1)". But it is exactly the same as the given example (see examples/create_cutout.py). Can anyone help with this? Thanks a lot!**

> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
>   return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
>   return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
>   return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
>   return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
>   return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
>   return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
>   return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
>   return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
>   return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
>   return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
>   return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
>   return _prepare_from_string(" ".join(pjargs))
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
>     exitcode = _main(fd)
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
>     prepare(preparation_data)
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>     _fixup_main_from_path(data['init_main_from_path'])
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
>         exitcode = _main(fd)run_name="__mp_main__")
> 
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
>         prepare(preparation_data)
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
> pkg_name=pkg_name, script_name=fname)
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
>         _fixup_main_from_path(data['init_main_from_path'])mod_name, mod_spec, pkg_name, script_name)
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
> 
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
>         exec(code, run_globals)run_name="__mp_main__")
>   File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
> 
> months=slice(1,1))  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
> 
>   File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
>     pkg_name=pkg_name, script_name=fname)    assert False
> AssertionError
> 
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
>     Traceback (most recent call last):
>   File "<string>", line 1, in <module>
> mod_name, mod_spec, pkg_name, script_name)  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
> 
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
>     exitcode = _main(fd)
>       File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
> exec(code, run_globals)
> Traceback (most recent call last):
>   File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
>       File "<string>", line 1, in <module>
> prepare(preparation_data)      File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
> 
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
> months=slice(1,1))
>   File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
> exitcode = _main(fd)    Traceback (most recent call last):
> _fixup_main_from_path(data['init_main_from_path'])
> 
> Traceback (most recent call last):
>       File "<string>", line 1, in <module>
> Traceback (most recent call last):
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
>   File "<string>", line 1, in <module>
> assert False  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
>   File "<string>", line 1, in <module>
>         Traceback (most recent call last):
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
> 
>     prepare(preparation_data)  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
> 
>   File "<string>", line 1, in <module>
> AssertionErrorexitcode = _main(fd)run_name="__mp_main__")      File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
> 
> exitcode = _main(fd)  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
> 
> exitcode = _main(fd)Traceback (most recent call last):
> 
> 
>       File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
> 
>       File "<string>", line 1, in <module>
> exitcode = _main(fd)  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
> _fixup_main_from_path(data['init_main_from_path'])  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
>       File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
> Traceback (most recent call last):
> 
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
> 
>     pkg_name=pkg_name, script_name=fname)      File "<string>", line 1, in <module>
> prepare(preparation_data)      File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
>     prepare(preparation_data)
> Traceback (most recent call last):
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
> 
> prepare(preparation_data)    exitcode = _main(fd)
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
>       File "<string>", line 1, in <module>
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
> 
> exitcode = _main(fd)
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
>     run_name="__mp_main__")  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
> prepare(preparation_data)  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
> 
> _fixup_main_from_path(data['init_main_from_path'])    mod_name, mod_spec, pkg_name, script_name)
> 
>       File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
> 
> _fixup_main_from_path(data['init_main_from_path'])
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
> exitcode = _main(fd)Traceback (most recent call last):
>     _fixup_main_from_path(data['init_main_from_path'])  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
> 
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
> 
> pkg_name=pkg_name, script_name=fname)    prepare(preparation_data)
> 
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
>           File "<string>", line 1, in <module>
> 
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
> _fixup_main_from_path(data['init_main_from_path'])prepare(preparation_data)
> 
> run_name="__mp_main__")exec(code, run_globals)  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
>       File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
>       File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
> 
> 
>         run_name="__mp_main__")        prepare(preparation_data)      File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
>       File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
> mod_name, mod_spec, pkg_name, script_name)exitcode = _main(fd)
> 
> _fixup_main_from_path(data['init_main_from_path'])
> run_name="__mp_main__")    _fixup_main_from_path(data['init_main_from_path'])
> run_name="__mp_main__")  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
> 
> 
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
> 
> pkg_name=pkg_name, script_name=fname)
> months=slice(1,1))  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
>       File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
> 
>       File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
> 
>     pkg_name=pkg_name, script_name=fname)
>           File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
> _fixup_main_from_path(data['init_main_from_path'])  File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
>     exec(code, run_globals)    prepare(preparation_data)  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
> run_name="__mp_main__")pkg_name=pkg_name, script_name=fname)
> 
>     run_name="__mp_main__")
> pkg_name=pkg_name, script_name=fname)
> 
> 
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
> assert False
>   File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
>       File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
> mod_name, mod_spec, pkg_name, script_name)  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
> 
> pkg_name=pkg_name, script_name=fname)        mod_name, mod_spec, pkg_name, script_name)months=slice(1,1))
>       File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
> AssertionError
> run_name="__mp_main__")
> 
> _fixup_main_from_path(data['init_main_from_path'])  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
> mod_name, mod_spec, pkg_name, script_name)
> pkg_name=pkg_name, script_name=fname)
> mod_name, mod_spec, pkg_name, script_name)  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
>   File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
> 
> 
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
> 
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
> 
>           File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
> exec(code, run_globals)  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
>       File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
> mod_name, mod_spec, pkg_name, script_name)  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
> pkg_name=pkg_name, script_name=fname)assert False
> 
> 
> exec(code, run_globals)  File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
> exec(code, run_globals)exec(code, run_globals)
> mod_name, mod_spec, pkg_name, script_name)  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
> 
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
> 
>     run_name="__mp_main__")  File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
> 
> 
> AssertionError          File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
> months=slice(1,1))
> 
>   File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
> 
> exec(code, run_globals)mod_name, mod_spec, pkg_name, script_name)          File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
>   File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
> 
> months=slice(1,1))months=slice(1,1))
> months=slice(1,1))exec(code, run_globals)assert False  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
> 
> 
> pkg_name=pkg_name, script_name=fname)  File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
> 
> 
>       File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
>   File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
> 
> 
>   File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
> months=slice(1,1))AssertionErrorexec(code, run_globals)          File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
>   File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
> 
> 
> 
> assert Falseassert False          File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
> assert False  File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
> 
> 
> mod_name, mod_spec, pkg_name, script_name)months=slice(1,1))
>     AssertionErrorAssertionError
> 
> assert FalseAssertionErrormonths=slice(1,1))
> 
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
>   File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
> 
> 
> 
> AssertionErrorassert False  File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
> exec(code, run_globals)
> 
> 
> assert False  File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
> AssertionError
> 
>     AssertionErrormonths=slice(1,1))
> 
>   File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
>     assert False
> AssertionError

[edit by @euronion : Formatting. Surronding your output with ``` ... ``` preserves the error message formatting and makes it easier to read.]

euronion commented 4 years ago
  1. The documentation on https://atlite.readthedocs.io is for the upcoming version of atlite which we have yet to finish in the next weeks. You seem to be running on another version. Which version do you have installed?
  2. Preparing the cutout does take rather long and uses all CPU ressources it can get its hands on. This will get better with the new version.
  3. The code you posted seems incomplete. Is this the same code that produces the error when you run the script from your command line?
fang1103 commented 4 years ago

@euronion Thank you for the comment!

  1. Thanks for the information. I am using version 0.0.2.
  2. I have updated the original problem description with the complete code and error log. Yes, there is no error reported when I run it in Spyder, but the problem is that it keeps runs for at least 2hrs without any response/output.
  3. I run the same code in the command line, and it firstly shows the same message ( "...Starting running them on all processors") and then keeps printing repeatedly the error I reported. Sorry for the formatting.
euronion commented 4 years ago

Hi @fang1103 ,

Thanks for the code update. Are you 100% sure it is the one you are running? Your output contains DEBUG output from the logging package, which is only configured for INFO in your code above.

I tried reproducing the problem on windows and on linux. On both plattforms the code above runs without problems and finishes within a minute or so on my laptop.

INFO:atlite.preparation:Starting preparation of cutout 'ukie-2011-01'
INFO:atlite.preparation:1 tasks have been collected. Starting running them on all processors.
INFO:atlite.preparation:Merging variables into monthly compound files
DEBUG:atlite.preparation:Completed file 201101.nc
INFO:atlite.preparation:Cutout 'ukie-2011-01' has been successfully prepared

Unfortunatley it is not clear to me, how your error above comes to be.

fang1103 commented 4 years ago

Hi @euronion

Sorry for the original misleading post, the output containing DEBUG output was when I used the following code

import warnings
import logging

warnings.simplefilter('default', DeprecationWarning)
logging.captureWarnings(True)
logging.basicConfig(level=logging.INFO)

I updated the message log after suppressing the DEBUG output in my original post. But the problem is the same, my computer keeps running for hours without any output after INFO:atlite.preparation:1 tasks have been collected. Starting running them on all processors. with Spyder, and it keeps reporting the error repeatedly with the command line.

It is not normal, right? I've tested the CDSAPI with the following code, and it works perfectly with a download rate 4.7M/s. I've also removed and reinstalled atlite, but didn't help...I will try to run it on Linux to see what happens. 

import cdsapi
c = cdsapi.Client()
c.retrieve("reanalysis-era5-single-levels",
{
"variable": "runoff",
"product_type": "reanalysis",
"year": "2008",
"month": "01",
"day": "01",
"time": "12:00",
"format": "netcdf"
},
"runoff.nc")
fang1103 commented 4 years ago

@euronion Hi, I come back to you after I set nprocesses=1 in cutout.prepare(), which can help to see more clearly the error bug: The following message keeps showing repeatedly when I am running the code in the command line:

C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs\crs.py:55: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6
  return _prepare_from_string(" ".join(pjargs))
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 30, in <module>
    months=slice(1,1))
  File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
    assert False
AssertionError
euronion commented 4 years ago

Thanks for the updates.

The code modifications were not significant, but sometimes it can happen that one accidentally uses different code than the one that is posted - just making sure that's not the case :)

Setting nprocesses=1 really helped making things more clear.

You are running into these lines https://github.com/PyPSA/atlite/blob/7d60f68e9043cecd7f65c77b7e0f480c1f22d4b0/atlite/cutout.py#L57-L60

Can you check whether the directory $cutout_dir/uk-2011-01 for storing the cutout files exists? If it does, delete or rename it, including all contents, and try again.

fang1103 commented 4 years ago

@euronion Thanks for the quick message! I've tracked to this part, but still haven't figured out what's the problem.

The directory $cutout_dir/uk-2011-01 does not exist when I was running the code and having the above-reported message, then I have this directory created by the code after I interrupted its running with Ctrl+C (since it keeps reporting the AssertionError continuously)

If the directory $cutout_dir/uk-2011-01 exists, I got the following error and the code stops runing:

C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs\crs.py:55: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6
  return _prepare_from_string(" ".join(pjargs))
Traceback (most recent call last):
  File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
    months=slice(1,1))
  File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
    assert False
AssertionError
euronion commented 4 years ago

That's weird, it should stop at this point if the directories it wants to write only partially exist.

From which source are you installing? PyPI, conda-forge or GitHub? Could you retry with a fresh conda environment based on the environment.yaml file + conda install -c conda-forge cartopy atlite cdsapi ?

fang1103 commented 4 years ago

@euronion I installed atlite from the conda-forge using conda install atlite with:

> channels:
>   - conda-forge
>   - defaults
> channel_priority: flexible

Following your suggestion, I created a fresh conda environment using the environment.yaml file by conda env create -f environment.yaml and installed all the packages conda install -c conda-forge cartopy atlite cdsapi and run the code there, unfortunately it is the same result, very weird...

euronion commented 4 years ago

Ok, so it doesn't seem to be a package / dependency-specific issue. Let's wait and see what your results from testing it on linux are.

fang1103 commented 4 years ago

Hi @euronion, it works when I set up the condaenvironment and run the code in the VM Virtualbox-based Ubuntu system. I had the cutout successfully. Then, I re-install the anaconda and re-setup all the things in my windows but still has the same problem, it just doesn't work...very weird Which setting are you using in your windows?

I looked at the code again, and suspect it might be the problem of the multiprocessing, for which Windows and Linux have different mechanisms?

coroa commented 4 years ago

hi @fang1103, in a sense you are right that it is multiprocessing causing the problem here and it would be interesting to compare how each of you start their code.

How multiprocessing in windows works is that it starts a new python interpreter and makes it load the same module/script again. Up to now, I always thought, it would only use the module from which one sets up the multiprocessing pool but in your case the stacktrace shows clearly that multiprocessing starts your script again. So to fix your problem, move your logic into a main function. I would wagger that @euronion runs your code from within an ipykernel either through ipython or jupyter in which case multiprocessing will not re-run the script in the newly spawned python engines, so it goes through fine. On linux you don't have the problem since you can just fork instead of start a new python interpreter. Multiprocessing is weird. Let's get v0.2 out and get rid of that bit of code!

import geopandas as gpd
import cartopy.io.shapereader as shpreader
import atlite
import logging

def main():
    shpfilename = shpreader.natural_earth(resolution='10m',
                                        category='cultural',
                                        name='admin_0_countries')
    reader = shpreader.Reader(shpfilename)
    UkIr = gpd.GeoSeries({r.attributes['NAME_EN']: r.geometry
                        for r in reader.records()},
                        crs={'init': 'epsg:4326'}
                        ).reindex(['United Kingdom', 'Ireland'])

    # Define the cutout; this will not yet trigger any major operations
    cutout = atlite.Cutout(name="uk-2011-01",
                        cutout_dir="./",
                        module="era5",
                        bounds=UkIr.unary_union.bounds,
                        years=slice(2011,2011),
                        months=slice(1,1))

    # This is where all the work happens.
    cutout.prepare()

if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    main()
fang1103 commented 4 years ago

@coroa Thank you very much for the clear explanation!! Now it works, seems we need to always protect the multiprocessing code.

fang1103 commented 4 years ago

Hi, Jonas

Does Atlite (the current version V0.0.2 or the upcoming V0.2) work with the Cordex data? I am a bit lost in creating a cutout from the Cordex dataset, can you provide a simple example, or just tell the general procedures of doing it? Thanks!

Kind regards,

Yiping

On Mon, 9 Mar 2020 at 18:06, Jonas Hörsch notifications@github.com wrote:

Closed #56 https://github.com/PyPSA/atlite/issues/56.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/PyPSA/atlite/issues/56?email_source=notifications&email_token=AFUQPVEKDXZYQYHV5TYKF4DRGUOZNA5CNFSM4LCM4WT2YY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOXF3NXHI#event-3111574429, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFUQPVA6AOZOJTPA34GPIDTRGUOZNANCNFSM4LCM4WTQ .

euronion commented 4 years ago

[...] I would wagger that @euronion runs your code from within an ipykernel either through ipython or jupyter [...]

Correct, I was testing it in jupyter lab.

Thanks for the help @coroa !