search

pypsa-meets-earth / pypsa-earth

PyPSA-Earth: A flexible Python-based open optimisation model to study energy system futures around the world.
https://pypsa-earth.readthedocs.io/en/latest/
235 stars 196 forks source link

Issue with build_powerplants #391

Closed hazemakhalek closed 2 years ago

hazemakhalek commented 2 years ago

Checklist

Describe the Bug

The workflow crashes when trying to index certain rows from a dataframe using the query()

Error Message

If applicable, paste any terminal output to help illustrating your problem. In some cases it may also be useful to share your list of installed packages: conda list.

Select jobs to execute...
WARNING:snakemake.logging:Select jobs to execute...

INFO:snakemake.logging:
[Tue Jun 21 15:26:30 2022]
INFO:snakemake.logging:[Tue Jun 21 15:26:30 2022]
rule build_powerplants:
    input: networks/base.nc, configs/powerplantmatching_config.yaml, data/custom_powerplants.csv, data/clean/africa_all_generators.csv
    output: resources/powerplants.csv, resources/powerplants_osm2pm.csv
    log: logs/build_powerplants.log
    jobid: 15
    reason: Missing output files: resources/powerplants.csv; Code has changed since last execution
    resources: tmpdir=/tmp, mem=500
INFO:snakemake.logging:rule build_powerplants:
    input: networks/base.nc, configs/powerplantmatching_config.yaml, data/custom_powerplants.csv, data/clean/africa_all_generators.csv
    output: resources/powerplants.csv, resources/powerplants_osm2pm.csv
    log: logs/build_powerplants.log
    jobid: 15
    reason: Missing output files: resources/powerplants.csv; Code has changed since last execution
    resources: tmpdir=/tmp, mem=500

INFO:snakemake.logging:
INFO:pypsa.io:Imported network base.nc has buses, lines
INFO:powerplantmatching.collection:Create combined dataset for GEO, GPD
INFO:powerplantmatching.cleaning:Aggregating blocks in data source 'GEO'.
INFO:powerplantmatching.cleaning:Aggregating blocks in data source 'GPD'.
INFO:powerplantmatching.matching:Comparing data sources `GEO` and `GPD`
INFO:powerplantmatching.cleaning:Aggregating blocks in data source 'GEO'.
INFO:powerplantmatching.cleaning:Aggregating blocks in data source 'GPD'.
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/pypsa-africa/lib/python3.10/site-packages/numexpr/necompiler.py", line 820, in evaluate
    compiled_ex = _numexpr_cache[numexpr_key]
KeyError: ('(substation_lv) & (Series_1_0x400xad0x200xd90x5e0x7f0x00x0)', (('optimization', 'aggressive'), ('truediv', False)), (('Series_1_0x400xad0x200xd90x5e0x7f0x00x0', <class 'bool'>), ('substation_lv', <class 'numpy.float64'>)))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/PyPSA_models/pypsa-africa/.snakemake/scripts/tmpz9xl39nc.build_powerplants.py", line 284, in <module>
    substation_i = n.buses.query("substation_lv and country == @c").index
  File "/home/user/anaconda3/envs/pypsa-africa/lib/python3.10/site-packages/pandas/core/frame.py", line 4111, in query
    res = self.eval(expr, **kwargs)
  File "/home/user/anaconda3/envs/pypsa-africa/lib/python3.10/site-packages/pandas/core/frame.py", line 4240, in eval
    return _eval(expr, inplace=inplace, **kwargs)
  File "/home/user/anaconda3/envs/pypsa-africa/lib/python3.10/site-packages/pandas/core/computation/eval.py", line 355, in eval
    ret = eng_inst.evaluate()
  File "/home/user/anaconda3/envs/pypsa-africa/lib/python3.10/site-packages/pandas/core/computation/engines.py", line 80, in evaluate
    res = self._evaluate()
  File "/home/user/anaconda3/envs/pypsa-africa/lib/python3.10/site-packages/pandas/core/computation/engines.py", line 121, in _evaluate
    return ne.evaluate(s, local_dict=scope)
  File "/home/user/anaconda3/envs/pypsa-africa/lib/python3.10/site-packages/numexpr/necompiler.py", line 822, in evaluate
    compiled_ex = _numexpr_cache[numexpr_key] = NumExpr(ex, signature, **context)
  File "/home/user/anaconda3/envs/pypsa-africa/lib/python3.10/site-packages/numexpr/necompiler.py", line 621, in NumExpr
    threeAddrProgram, inputsig, tempsig, constants, input_names = precompile(ex, signature, context)
  File "/home/user/anaconda3/envs/pypsa-africa/lib/python3.10/site-packages/numexpr/necompiler.py", line 566, in precompile
    ast = typeCompileAst(ast)
  File "/home/user/anaconda3/envs/pypsa-africa/lib/python3.10/site-packages/numexpr/necompiler.py", line 202, in typeCompileAst
    raise NotImplementedError(
NotImplementedError: couldn't find matching opcode for 'and_bdb'
[Tue Jun 21 15:26:42 2022]
INFO:snakemake.logging:[Tue Jun 21 15:26:42 2022]
Error in rule build_powerplants:
    jobid: 15
    output: resources/powerplants.csv, resources/powerplants_osm2pm.csv
    log: logs/build_powerplants.log (check log file(s) for error message)

ERROR:snakemake.logging:Error in rule build_powerplants:
    jobid: 15
    output: resources/powerplants.csv, resources/powerplants_osm2pm.csv
    log: logs/build_powerplants.log (check log file(s) for error message)

RuleException:
CalledProcessError in line 329 of /home/user/PyPSA_models/pypsa-africa/Snakefile:
Command 'set -euo pipefail;  /home/user/anaconda3/envs/pypsa-africa/bin/python3.10 /home/user/PyPSA_models/pypsa-africa/.snakemake/scripts/tmpz9xl39nc.build_powerplants.py' returned non-zero exit status 1.
  File "/home/user/PyPSA_models/pypsa-africa/Snakefile", line 329, in __rule_build_powerplants
  File "/home/user/anaconda3/envs/pypsa-africa/lib/python3.10/concurrent/futures/thread.py", line 58, in run
ERROR:snakemake.logging:RuleException:
CalledProcessError in line 329 of /home/user/PyPSA_models/pypsa-africa/Snakefile:
Command 'set -euo pipefail;  /home/user/anaconda3/envs/pypsa-africa/bin/python3.10 /home/user/PyPSA_models/pypsa-africa/.snakemake/scripts/tmpz9xl39nc.build_powerplants.py' returned non-zero exit status 1.
  File "/home/user/PyPSA_models/pypsa-africa/Snakefile", line 329, in __rule_build_powerplants
  File "/home/user/anaconda3/envs/pypsa-africa/lib/python3.10/concurrent/futures/thread.py", line 58, in run
Removing output files of failed job build_powerplants since they might be corrupted:
resources/powerplants_osm2pm.csv
WARNING:snakemake.logging:Removing output files of failed job build_powerplants since they might be corrupted:
resources/powerplants_osm2pm.csv
Shutting down, this might take some time.
WARNING:snakemake.logging:Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
ERROR:snakemake.logging:Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2022-06-21T152628.595878.snakemake.log
WARNING:snakemake.logging:Complete log: .snakemake/log/2022-06-21T152628.595878.snakemake.log
davide-f commented 2 years ago

As mentioned, this error sounds very weird. I'd suggest you to verify the specific "countries" vector that is used in the workflow to check whether there may be weird values. Moreover, I'd suggest also to be sure to have resetted the powerplantmatching, both in terms of package and data, as in the other issue (https://github.com/pypsa-meets-africa/pypsa-africa/issues/358)

hazemakhalek commented 2 years ago

I think it has to do more with the operand in the query function more than the country. Which python version do you use?

davide-f commented 2 years ago

Currently I have the environment down; it is reinstalling. Python was 3.10 I think. I think it may be a problem of:

The CI constantly works and runs also that code, so that lines should work

davide-f commented 2 years ago

I've started a new fresh environment and deleted the data as described in the issue mentioned before; MA worked. In the night, I plan to run the Africa model to test the new branch normalize-hydro, which should further validate the issue. That may help crosschecking this issue as well.

What changes did you make with respect to the last time you executed the code?

davide-f commented 2 years ago

@Hazem-IEG , just as a confirmation, have you solved this issue with the above-mentioned techniques? HAve you identified the root cause?

davide-f commented 2 years ago

Closing this issue as it seems it is no more an issue