Error in building the base_network, tutorial, release 0.61

[luigimoccia]

Hi,

I am using the current master branch or the latest release 0.61

I am running on an up-to-date pypsa-eur environment. The env. is updated via conda env update -f envs/environment.yaml.

I have followed the tutorial for a new installation and I'm using the tutorial configuration 'cp config.tutorial.yaml config.yaml'

Bug

I tried to run the tutorial example with: 'snakemake -j 1 results/networks/elec_s_6_ec_lcopt_Co2L-24H.nc'

It fails at the first step, base_network, with the following log

LOG

Building DAG of jobs... Using shell: /bin/bash Provided cores: 1 (use --cores to define parallelism) Rules claiming more threads will be scaled down. Job stats: job count min threads max threads

---

add_electricity 1 1 1 add_extra_components 1 1 1 base_network 1 1 1 build_bus_regions 1 1 1 build_powerplants 1 1 1 build_renewable_profiles 4 1 1 cluster_network 1 1 1 prepare_network 1 1 1 simplify_network 1 1 1 solve_network 1 1 1 total 13 1 1

Select jobs to execute...

[Thu Dec 22 11:56:44 2022] rule base_network: input: data/entsoegridkit/buses.csv, data/entsoegridkit/lines.csv, data/entsoegridkit/links.csv, data/entsoegridkit/converters.csv, data/entsoegridkit/transformers.csv, data/parameter_corrections.yaml, data/links_p_nom.csv, data/links_tyndp.csv, resources/country_shapes.geojson, resources/offshore_shapes.geojson, resources/europe_shape.geojson output: networks/base.nc log: logs/base_network.log jobid: 7 benchmark: benchmarks/base_network reason: Missing output files: networks/base.nc resources: tmpdir=/var/folders/yf/pz_s8_n93b58fdcf_snyw1nr0000gn/T, mem_mb=500, mem_mib=477

[Thu Dec 22 11:56:46 2022] Error in rule base_network: jobid: 7 input: data/entsoegridkit/buses.csv, data/entsoegridkit/lines.csv, data/entsoegridkit/links.csv, data/entsoegridkit/converters.csv, data/entsoegridkit/transformers.csv, data/parameter_corrections.yaml, data/links_p_nom.csv, data/links_tyndp.csv, resources/country_shapes.geojson, resources/offshore_shapes.geojson, resources/europe_shape.geojson output: networks/base.nc log: logs/base_network.log (check log file(s) for error details)

RuleException: CalledProcessError in file /Users/luigimoccia2022/pypsa-eur/Snakefile, line 172: Command 'set -euo pipefail; /Users/luigimoccia2022/opt/miniconda3/envs/pypsa-eur/bin/python3.10 /Users/luigimoccia2022/pypsa-eur/.snakemake/scripts/tmpdb4cilr8.base_network.py' returned non-zero exit status 1. File "/Users/luigimoccia2022/pypsa-eur/Snakefile", line 172, in __rule_base_network File "/Users/luigimoccia2022/opt/miniconda3/envs/pypsa-eur/lib/python3.10/concurrent/futures/thread.py", line 52, in run Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message

List of installed packages

_ipython_minor_entry_point 8.7.0 hb6b4a82_0 conda-forge affine 2.3.1 pyhd8ed1ab_0 conda-forge ampl-mp 3.1.0 hbec66e7_1006 conda-forge amply 0.1.5 pyhd8ed1ab_0 conda-forge appdirs 1.4.4 pyh9f0ad1d_0 conda-forge appnope 0.1.3 pyhd8ed1ab_0 conda-forge asttokens 2.2.1 pyhd8ed1ab_0 conda-forge atlite 0.2.9 pyhd8ed1ab_1 conda-forge attrs 22.2.0 pyh71513ae_0 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 pyhd8ed1ab_3 conda-forge backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge beautifulsoup4 4.11.1 pyha770c72_0 conda-forge blosc 1.21.3 h1d6ff8b_0 conda-forge bokeh 2.4.3 pyhd8ed1ab_3 conda-forge boost-cpp 1.78.0 h1cb353e_1 conda-forge bottleneck 1.3.5 py310hf1a086a_1 conda-forge branca 0.6.0 pyhd8ed1ab_0 conda-forge brotli 1.0.9 h1a8c8d9_8 conda-forge brotli-bin 1.0.9 h1a8c8d9_8 conda-forge brotlipy 0.7.0 py310h8e9501a_1005 conda-forge bzip2 1.0.8 h3422bc3_4 conda-forge c-ares 1.18.1 h3422bc3_0 conda-forge ca-certificates 2022.12.7 h4653dfc_0 conda-forge cairo 1.16.0 h73a0509_1014 conda-forge cartopy 0.21.1 py310h473893f_0 conda-forge cdsapi 0.5.1 pyhd8ed1ab_0 conda-forge certifi 2022.12.7 pyhd8ed1ab_0 conda-forge cffi 1.15.1 py310h2399d43_3 conda-forge cfitsio 4.2.0 h2f961c4_0 conda-forge cftime 1.6.2 py310hf1a086a_1 conda-forge charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge click 8.1.3 unix_pyhd8ed1ab_2 conda-forge click-plugins 1.1.1 py_0 conda-forge cligj 0.7.2 pyhd8ed1ab_1 conda-forge cloudpickle 2.2.0 pyhd8ed1ab_0 conda-forge coin-or-cbc 2.10.8 h1ce7d08_0 conda-forge coin-or-cgl 0.60.6 hf050ae7_2 conda-forge coin-or-clp 1.17.7 h65c2c7c_2 conda-forge coin-or-osi 0.108.7 h19bcb3e_2 conda-forge coin-or-utils 2.11.6 h8717b36_2 conda-forge coincbc 2.10.8 0_metapackage conda-forge colorama 0.4.6 pyhd8ed1ab_0 conda-forge conda 22.11.1 py310hbe9552e_1 conda-forge conda-package-handling 2.0.2 pyh38be061_0 conda-forge conda-package-streaming 0.7.0 pyhd8ed1ab_1 conda-forge configargparse 1.5.3 pyhd8ed1ab_0 conda-forge connection_pool 0.0.3 pyhd3deb0d_0 conda-forge contourpy 1.0.6 py310h2887b22_0 conda-forge country_converter 0.8.0 pyhd8ed1ab_0 conda-forge countrycode 0.2 pypi_0 pypi cryptography 38.0.4 py310hfc83b78_0 conda-forge curl 7.87.0 h9049daf_0 conda-forge cycler 0.11.0 pyhd8ed1ab_0 conda-forge cytoolz 0.12.0 py310h8e9501a_1 conda-forge dask 2022.12.1 pyhd8ed1ab_0 conda-forge dask-core 2022.12.1 pyhd8ed1ab_0 conda-forge datrie 0.8.2 py310h8e9501a_6 conda-forge decorator 5.1.1 pyhd8ed1ab_0 conda-forge deprecation 2.1.0 pyh9f0ad1d_0 conda-forge descartes 1.1.0 py_4 conda-forge distributed 2022.12.1 pyhd8ed1ab_0 conda-forge docutils 0.19 py310hbe9552e_1 conda-forge dpath 2.1.3 py310hbe9552e_0 conda-forge entsoe-py 0.5.8 pyhd8ed1ab_0 conda-forge et_xmlfile 1.0.1 py_1001 conda-forge exceptiongroup 1.0.4 pyhd8ed1ab_0 conda-forge executing 1.2.0 pyhd8ed1ab_0 conda-forge expat 2.5.0 hb7217d7_0 conda-forge filelock 3.8.2 pyhd8ed1ab_0 conda-forge fiona 1.8.22 py310h2e51ddd_5 conda-forge fmt 9.1.0 hffc8910_0 conda-forge folium 0.14.0 pyhd8ed1ab_0 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 hab24e00_0 conda-forge fontconfig 2.14.1 h82840c6_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.38.0 py310h8e9501a_1 conda-forge freetype 2.12.1 hd633e50_1 conda-forge freexl 1.0.6 h1a8c8d9_1 conda-forge fsspec 2022.11.0 pyhd8ed1ab_0 conda-forge gdal 3.6.1 py310h0b34360_1 conda-forge geographiclib 1.52 pyhd8ed1ab_0 conda-forge geojson-rewind 1.0.2 pyhd8ed1ab_0 conda-forge geopandas 0.12.2 pyhd8ed1ab_0 conda-forge geopandas-base 0.12.2 pyha770c72_0 conda-forge geopy 2.3.0 pyhd8ed1ab_0 conda-forge geos 3.11.1 hb7217d7_0 conda-forge geotiff 1.7.1 h90559a4_4 conda-forge gettext 0.21.1 h0186832_0 conda-forge giflib 5.2.1 h27ca646_2 conda-forge gitdb 4.0.10 pyhd8ed1ab_0 conda-forge gitpython 3.1.29 pyhd8ed1ab_0 conda-forge glpk 5.0 h6d7a090_0 conda-forge gmp 6.2.1 h9f76cd9_0 conda-forge gurobi 10.0.0 py310_0 gurobi hdf4 4.2.15 h1a38d6a_5 conda-forge hdf5 1.12.2 nompi_ha7af310_101 conda-forge heapdict 1.0.1 py_0 conda-forge highspy 1.5.0.dev0 pypi_0 pypi icu 70.1 h6b3803e_0 conda-forge idna 3.4 pyhd8ed1ab_0 conda-forge importlib-metadata 5.2.0 pyha770c72_0 conda-forge importlib_metadata 5.2.0 hd8ed1ab_0 conda-forge importlib_resources 5.10.1 pyhd8ed1ab_1 conda-forge iniconfig 1.1.1 pyh9f0ad1d_0 conda-forge ipopt 3.14.10 h843a782_0 conda-forge ipython 8.7.0 pyhd1c38e8_0 conda-forge jedi 0.18.2 pyhd8ed1ab_0 conda-forge jinja2 3.1.2 pyhd8ed1ab_1 conda-forge joblib 1.2.0 pyhd8ed1ab_0 conda-forge jpeg 9e he4db4b2_2 conda-forge json-c 0.16 hc449e50_0 conda-forge jsonschema 4.17.3 pyhd8ed1ab_0 conda-forge jupyter_core 5.1.0 py310hbe9552e_0 conda-forge kealib 1.5.0 hfd766a6_0 conda-forge kiwisolver 1.4.4 py310h2887b22_1 conda-forge krb5 1.20.1 h69eda48_0 conda-forge lcms2 2.14 h8193b64_0 conda-forge lerc 4.0.0 h9a09cb3_0 conda-forge libaec 1.0.6 hbdafb3b_0 conda-forge libarchive 3.6.2 h83f22c9_0 conda-forge libblas 3.9.0 16_osxarm64_openblas conda-forge libbrotlicommon 1.0.9 h1a8c8d9_8 conda-forge libbrotlidec 1.0.9 h1a8c8d9_8 conda-forge libbrotlienc 1.0.9 h1a8c8d9_8 conda-forge libcblas 3.9.0 16_osxarm64_openblas conda-forge libcurl 7.87.0 h9049daf_0 conda-forge libcxx 14.0.6 h2692d47_0 conda-forge libdeflate 1.14 h1a8c8d9_0 conda-forge libedit 3.1.20191231 hc8eb9b7_2 conda-forge libev 4.33 h642e427_1 conda-forge libffi 3.4.2 h3422bc3_5 conda-forge libgdal 3.6.1 hf393d26_1 conda-forge libgfortran 5.0.0 11_3_0_hd922786_27 conda-forge libgfortran5 11.3.0 hdaf2cc0_27 conda-forge libglib 2.74.1 h4646484_1 conda-forge libiconv 1.17 he4db4b2_0 conda-forge libkml 1.3.0 h41464e4_1015 conda-forge liblapack 3.9.0 16_osxarm64_openblas conda-forge liblapacke 3.9.0 16_osxarm64_openblas conda-forge libmamba 1.1.0 h588180f_3 conda-forge libmambapy 1.1.0 py310h34b6e76_3 conda-forge libnetcdf 4.8.1 nompi_h2510be2_106 conda-forge libnghttp2 1.47.0 h519802c_1 conda-forge libopenblas 0.3.21 openmp_hc731615_3 conda-forge libpng 1.6.39 h76d750c_0 conda-forge libpq 15.1 h1a28acd_2 conda-forge librttopo 1.1.0 h844f84d_12 conda-forge libsolv 0.7.23 hb5ab8b9_0 conda-forge libspatialindex 1.9.3 hbdafb3b_4 conda-forge libspatialite 5.0.1 h558e587_22 conda-forge libsqlite 3.40.0 h76d750c_0 conda-forge libssh2 1.10.0 h7a5bd25_3 conda-forge libtiff 4.4.0 heb92581_5 conda-forge libwebp-base 1.2.4 h57fd34a_0 conda-forge libxcb 1.13 h9b22ae9_1004 conda-forge libxml2 2.10.3 h87b0503_0 conda-forge libxslt 1.1.37 h1bd8bc4_0 conda-forge libzip 1.9.2 h76ab92c_1 conda-forge libzlib 1.2.13 h03a7124_4 conda-forge linopy 0.0.14 pyhd8ed1ab_0 conda-forge llvm-openmp 15.0.6 h7cfbb63_0 conda-forge locket 1.0.0 pyhd8ed1ab_0 conda-forge lxml 4.9.2 py310h85b680a_0 conda-forge lz4 4.0.2 py310ha6df754_0 conda-forge lz4-c 1.9.3 hbdafb3b_1 conda-forge lzo 2.10 h642e427_1000 conda-forge mamba 1.1.0 py310ha5d4528_3 conda-forge mapclassify 2.4.3 pyhd8ed1ab_0 conda-forge markupsafe 2.1.1 py310h8e9501a_2 conda-forge matplotlib 3.6.2 py310hb6292c7_0 conda-forge matplotlib-base 3.6.2 py310h78c5c2f_0 conda-forge matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge memory_profiler 0.61.0 pyhd8ed1ab_0 conda-forge metis 5.1.0 h9f76cd9_1006 conda-forge msgpack-python 1.0.4 py310h2887b22_1 conda-forge mumps-include 5.2.1 hce30654_11 conda-forge mumps-seq 5.2.1 h0066fe1_11 conda-forge munch 2.5.0 py_0 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge nbformat 5.7.1 pyhd8ed1ab_0 conda-forge ncurses 6.3 h07bb92c_1 conda-forge netcdf4 1.6.2 nompi_py310haaa361f_100 conda-forge networkx 2.8.8 pyhd8ed1ab_0 conda-forge nspr 4.35 hb7217d7_0 conda-forge nss 3.78 h1483a63_0 conda-forge numexpr 2.8.3 py310h2b830bf_1 conda-forge numpy 1.24.0 py310h5d7c261_0 conda-forge openjdk 17.0.3 hf913c23_5 conda-forge openjpeg 2.5.0 h5d4e404_1 conda-forge openpyxl 3.0.10 py310h8e9501a_2 conda-forge openssl 3.0.7 h03a7124_1 conda-forge packaging 22.0 pyhd8ed1ab_0 conda-forge pandas 1.5.2 py310h2b830bf_0 conda-forge parso 0.8.3 pyhd8ed1ab_0 conda-forge partd 1.3.0 pyhd8ed1ab_0 conda-forge patsy 0.5.3 pyhd8ed1ab_0 conda-forge pcre2 10.40 hb34f9b4_0 conda-forge pexpect 4.8.0 pyh1a96a4e_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 9.2.0 py310h9337a76_3 conda-forge pip 22.3.1 pyhd8ed1ab_0 conda-forge pixman 0.40.0 h27ca646_0 conda-forge pkgutil-resolve-name 1.3.10 pyhd8ed1ab_0 conda-forge plac 1.3.5 pyhd8ed1ab_0 conda-forge platformdirs 2.6.0 pyhd8ed1ab_0 conda-forge pluggy 1.0.0 pyhd8ed1ab_5 conda-forge ply 3.11 py_1 conda-forge poppler 22.12.0 hae7f5f0_0 conda-forge poppler-data 0.4.11 hd8ed1ab_0 conda-forge postgresql 15.1 h45c140d_2 conda-forge powerplantmatching 0.5.4 pyhd8ed1ab_0 conda-forge progressbar2 4.2.0 pyhd8ed1ab_0 conda-forge proj 9.1.0 h3bdf472_0 conda-forge prompt-toolkit 3.0.36 pyha770c72_0 conda-forge psutil 5.9.4 py310h8e9501a_0 conda-forge pthread-stubs 0.4 h27ca646_1001 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pulp 2.7.0 py310hbe9552e_0 conda-forge pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge pybind11 2.10.2 pypi_0 pypi pybind11-abi 4 hd8ed1ab_3 conda-forge pycosat 0.6.4 py310h8e9501a_1 conda-forge pycountry 22.3.5 pyhd8ed1ab_0 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pygments 2.13.0 pyhd8ed1ab_0 conda-forge pyomo 6.4.4 py310h0f1eb42_0 conda-forge pyopenssl 22.1.0 pyhd8ed1ab_0 conda-forge pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge pyproj 3.4.1 py310hce8d790_0 conda-forge pypsa 0.21.3 pyhd8ed1ab_0 conda-forge pyrsistent 0.19.2 py310h8e9501a_0 conda-forge pyshp 2.3.1 pyhd8ed1ab_0 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge pytables 3.7.0 py310h838cff0_3 conda-forge pytest 7.2.0 pyhd8ed1ab_2 conda-forge python 3.10.0 h43b31ca_3_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-fastjsonschema 2.16.2 pyhd8ed1ab_0 conda-forge python-utils 3.4.5 pyhd8ed1ab_0 conda-forge python_abi 3.10 3_cp310 conda-forge pytz 2022.7 pyhd8ed1ab_0 conda-forge pyxlsb 1.0.10 pyhd8ed1ab_0 conda-forge pyyaml 6.0 py310h8e9501a_5 conda-forge rasterio 1.3.4 py310h2a1f202_0 conda-forge readline 8.1.2 h46ed386_0 conda-forge reproc 14.2.4 h1a8c8d9_0 conda-forge reproc-cpp 14.2.4 hb7217d7_0 conda-forge requests 2.28.1 pyhd8ed1ab_1 conda-forge reretry 0.11.8 pyhd8ed1ab_0 conda-forge rtree 1.0.1 py310ha3239f5_1 conda-forge ruamel.yaml 0.17.21 py310h8e9501a_2 conda-forge ruamel.yaml.clib 0.2.7 py310h8e9501a_1 conda-forge scikit-learn 1.2.0 py310ha00a7cd_0 conda-forge scipy 1.9.3 py310ha0d8a01_2 conda-forge scotch 6.0.9 h7537618_2 conda-forge seaborn 0.12.1 hd8ed1ab_0 conda-forge seaborn-base 0.12.1 pyhd8ed1ab_0 conda-forge setuptools 65.6.3 pyhd8ed1ab_0 conda-forge setuptools-scm 7.1.0 pyhd8ed1ab_0 conda-forge setuptools_scm 7.1.0 hd8ed1ab_0 conda-forge shapely 1.8.5 py310h9356385_2 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge smart_open 6.3.0 pyha770c72_0 conda-forge smmap 3.0.5 pyh44b312d_0 conda-forge snakemake-minimal 7.19.1 pyhdfd78af_0 bioconda snappy 1.1.9 h17c5cce_2 conda-forge snuggs 1.4.7 py_0 conda-forge sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge soupsieve 2.3.2.post1 pyhd8ed1ab_0 conda-forge sqlite 3.40.0 h2229b38_0 conda-forge stack_data 0.6.2 pyhd8ed1ab_0 conda-forge statsmodels 0.13.5 py310hf1a086a_2 conda-forge stopit 1.1.2 py_0 conda-forge tabula-py 2.6.0 py310hbe9552e_0 conda-forge tabulate 0.9.0 pyhd8ed1ab_1 conda-forge tblib 1.7.0 pyhd8ed1ab_0 conda-forge threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge throttler 1.2.1 pyhd8ed1ab_0 conda-forge tiledb 2.13.0 h9bd36d0_1 conda-forge tk 8.6.12 he1e0b03_0 conda-forge tomli 2.0.1 pyhd8ed1ab_0 conda-forge toolz 0.12.0 pyhd8ed1ab_0 conda-forge toposort 1.7 pyhd8ed1ab_0 conda-forge tornado 6.2 py310h8e9501a_1 conda-forge tqdm 4.64.1 pyhd8ed1ab_0 conda-forge traitlets 5.8.0 pyhd8ed1ab_0 conda-forge tsam 2.2.2 pypi_0 pypi typing-extensions 4.4.0 hd8ed1ab_0 conda-forge typing_extensions 4.4.0 pyha770c72_0 conda-forge tzcode 2022g h1a8c8d9_0 conda-forge tzdata 2022g h191b570_0 conda-forge unicodedata2 15.0.0 py310h8e9501a_0 conda-forge unidecode 1.3.6 pyhd8ed1ab_0 conda-forge unixodbc 2.3.10 hb39ff51_0 conda-forge urllib3 1.26.13 pyhd8ed1ab_0 conda-forge vresutils 0.3.1 pypi_0 pypi wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge wheel 0.38.4 pyhd8ed1ab_0 conda-forge wrapt 1.14.1 py310h8e9501a_1 conda-forge xarray 2022.3.0 pyhd8ed1ab_0 conda-forge xerces-c 3.2.4 h627aa08_1 conda-forge xlrd 2.0.1 pyhd8ed1ab_3 conda-forge xorg-libxau 1.0.9 h27ca646_0 conda-forge xorg-libxdmcp 1.1.3 h27ca646_0 conda-forge xyzservices 2022.9.0 pyhd8ed1ab_0 conda-forge xz 5.2.6 h57fd34a_0 conda-forge yaml 0.2.5 h3422bc3_2 conda-forge yaml-cpp 0.7.0 hb7217d7_2 conda-forge yte 1.5.1 py310hbe9552e_1 conda-forge zict 2.2.0 pyhd8ed1ab_0 conda-forge zipp 3.11.0 pyhd8ed1ab_0 conda-forge zlib 1.2.13 h03a7124_4 conda-forge zstandard 0.19.0 py310had9512b_1 conda-forge zstd 1.5.2 h8128057_4 conda-forge

[luigimoccia]

I've solved this problem by reverting to a previous version of numpy

I suggest to include in the envs/environment.yaml the following dependency:

• numpy<=1.23.5

[euronion]

Hi,

I can't reproduce the issue with the tutorial config. It also works correctly for me with numpy=1.24.0. Something else must be amiss there.

[fneum]

Please re-open if issue persists.

[luigimoccia]

In a few days I'll try to reproduce it on a different machine (the computer where it failed with numpy 1.24.0 is a mac mini M1 2020 with MacOS 12.6.1

[pz-max]

Hi @luigimoccia , a couple of days ago we experienced in PyPSA-Earth issues as well https://github.com/pypsa-meets-earth/pypsa-earth/pull/538#issuecomment-1364700612. We had to restrict the numpy version below 'numpy<1.24' which made the workflow work.

The CI couldn't detect the issue because it was solving with 'numpy<1.24' anyways.