Closed Tomkourou closed 2 years ago
pz-max
commented
2 years ago Could you try to run the same thing with CBC or glpk? The clustering requires some special solving approach (I believe the code pivots to CBC for that part) Maybe for highs this is not perfectly integrated by default...
Tomkourou
commented
2 years ago When using glpk or CBC it defaults back to ipopt which I so far haven't been able to install on Ubuntu . I could explore this in more detail. Here is the error code for glpk but its exactly the same whne using CBC
INFO:pypsa.io:Imported network elec_s.nc has buses, carriers, generators, lines, loads
WARNING:__main__:The configured solver `glpk` does not support quadratic objectives. Falling back to `ipopt`.
WARNING:pyomo.solvers:Could not locate the 'ipopt' executable, which is required for solver ipopt
Traceback (most recent call last):
File "/home/thomas/Personal_projects/pypsa-africa/.snakemake/scripts/tmppq5unxd0.cluster_network.py", line 552, in <module>
clustering = clustering_for_n_clusters(
File "/home/thomas/Personal_projects/pypsa-africa/.snakemake/scripts/tmppq5unxd0.cluster_network.py", line 426, in clustering_for_n_clusters
busmap = busmap_for_n_clusters(n, n_clusters, solver_name,
File "/home/thomas/Personal_projects/pypsa-africa/.snakemake/scripts/tmppq5unxd0.cluster_network.py", line 344, in busmap_for_n_clusters
n_clusters = distribute_clusters(n,
File "/home/thomas/Personal_projects/pypsa-africa/.snakemake/scripts/tmppq5unxd0.cluster_network.py", line 293, in distribute_clusters
results = opt.solve(m)
File "/home/thomas/anaconda3/envs/pypsa-africa/lib/python3.9/site-packages/pyomo/opt/base/solvers.py", line 512, in solve
self.available(exception_flag=True)
File "/home/thomas/anaconda3/envs/pypsa-africa/lib/python3.9/site-packages/pyomo/opt/solver/shellcmd.py", line 128, in available
raise ApplicationError(msg % self.name)
pyomo.common.errors.ApplicationError: No executable found for solver 'ipopt'
[Tue Mar 22 13:13:01 2022]
INFO:snakemake.logging:[Tue Mar 22 13:13:01 2022]
Error in rule cluster_network:
jobid: 2
output: networks/elec_s_10_pre_augmentation.nc, resources/regions_onshore_elec_s_10.geojson, resources/regions_offshore_elec_s_10.geojson, resources/busmap_elec_s_10.csv, resources/linemap_elec_s_10.csv
log: logs/cluster_network/elec_s_10.log (check log file(s) for error message)
ERROR:snakemake.logging:Error in rule cluster_network:
jobid: 2
output: networks/elec_s_10_pre_augmentation.nc, resources/regions_onshore_elec_s_10.geojson, resources/regions_offshore_elec_s_10.geojson, resources/busmap_elec_s_10.csv, resources/linemap_elec_s_10.csv
log: logs/cluster_network/elec_s_10.log (check log file(s) for error message)
RuleException:
CalledProcessError in line 302 of /home/thomas/Personal_projects/pypsa-africa/Snakefile:
Command 'set -euo pipefail; /home/thomas/anaconda3/envs/pypsa-africa/bin/python3.9 /home/thomas/Personal_projects/pypsa-africa/.snakemake/scripts/tmppq5unxd0.cluster_network.py' returned non-zero exit status 1.
File "/home/thomas/Personal_projects/pypsa-africa/Snakefile", line 302, in __rule_cluster_network
File "/home/thomas/anaconda3/envs/pypsa-africa/lib/python3.9/concurrent/futures/thread.py", line 58, in run
ERROR:snakemake.logging:RuleException:
CalledProcessError in line 302 of /home/thomas/Personal_projects/pypsa-africa/Snakefile:
Command 'set -euo pipefail; /home/thomas/anaconda3/envs/pypsa-africa/bin/python3.9 /home/thomas/Personal_projects/pypsa-africa/.snakemake/scripts/tmppq5unxd0.cluster_network.py' returned non-zero exit status 1.
File "/home/thomas/Personal_projects/pypsa-africa/Snakefile", line 302, in __rule_cluster_network
File "/home/thomas/anaconda3/envs/pypsa-africa/lib/python3.9/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
WARNING:snakemake.logging:Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
ERROR:snakemake.logging:Exiting because a job execution failed. Look above for error messageOh yes, Ipopt is the right thing! You see in the CI that we are adding it to the env.yaml when running with glpk https://github.com/pypsa-meets-africa/pypsa-africa/blob/c994217e2c5fff891e39dde5fe6da6d27757c498/.github/workflows/ci.yaml#L48-L50
So running highs with ipopt will work as well. You can try two things: First install ipopt with pip like pip install ipopt (if that does not work google for ipopt pip installation). As alternative, when setting up the conda environment you should uncomment out ipopt from here https://github.com/pypsa-meets-africa/pypsa-africa/blob/c994217e2c5fff891e39dde5fe6da6d27757c498/envs/environment.yaml#L65-L67
Tomkourou
commented
2 years ago Thanks for the tip @pz-max. Somehow I tried pip install ipopt before but on the ubuntu at least it did not install an executable in my environment. However conda install -c conda-forge cyipopt did the job. Running snakemake cluster rule now works as expected. Thanks!
Checklist
mainbranch or the latest release.pypsa-africaenvironment.Describe the Bug
Running the cluster_network snakemake workflow using snakemake -c1 networks/elec_s_10.nc for "NG" and "BJ" returns a CalledProcessError, looks like something to do with running multiple threads although I really don't have experience here.
Solver: HiGHS.
IDE: Visual Studio Code.
OS: Ubuntu 20.04.4 LTS on Windows 10 x86_64
Error Message
And here is my environment:
Let me know if I can add any further information.