search

OSeMOSYS / osemosys_global

A global power system model generator for OSeMOSYS
https://osemosys-global.readthedocs.io/
GNU Affero General Public License v3.0
27 stars 17 forks source link

[BUG]: Tranmission fails if no user defined capacity defined #215

Closed trevorb1 closed 1 month ago

trevorb1 commented 1 month ago

Conda environment check

Current Behavior

If you do not specify user defined transmission capacity, the workflow fails. This is due to min_cap_invest_trn only being created if user defined capacity is supplied.

https://github.com/OSeMOSYS/osemosys_global/blob/2a3e961382604c492d28acd43150bd8005c84c50/workflow/scripts/osemosys_global/transmission/main.py#L127-L151

https://github.com/OSeMOSYS/osemosys_global/blob/2a3e961382604c492d28acd43150bd8005c84c50/workflow/scripts/osemosys_global/transmission/main.py#L198-L200

Expected Behavior

No response

Steps To Reproduce

  1. Remove data from user_defined_capacity_transmission
  2. Run the workflow

Operating System

Linux

Log output

host: Trevor
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 32
Rules claiming more threads will be scaled down.
Job stats:
job                          count
-------------------------  -------
all                              1
copy_otoole_confg                1
create_lp_file                   1
create_missing_csv               1
demand_projection_figures        1
demand_projections               1
emissions                        1
geographic_filter                1
max_capacity                     1
otoole_convert                   1
otoole_results                   1
powerplant                       1
preprocess_data_file             1
solve_lp                         1
summarise_results                1
timeslice                        1
transmission                     1
variable_costs                   1
visualisation                    1
total                           19

Select jobs to execute...
Execute 5 jobs...

[Thu Oct 24 14:56:40 2024]
Job 9: Copying otoole configuration file...
Reason: Missing output files: results/India/otoole.yaml

[Thu Oct 24 14:56:40 2024]
Job 13: Generating powerplant data...
Reason: Missing output files: results/data/powerplant/CapitalCost.csv, results/data/powerplant/OperationalLife.csv, results/data/powerplant/TotalAnnualMaxCapacityInvestment.csv, results/data/powerplant/FixedCost.csv, results/data/YEAR.csv, results/data/powerplant/OutputActivityRatio.csv, results/data/powerplant/FUEL.csv, results/data/powerplant/InputActivityRatio.csv, results/data/powerplant/TotalAnnualMinCapacityInvestment.csv, results/data/AvailabilityFactor.csv, results/data/MODE_OF_OPERATION.csv, results/data/powerplant/TECHNOLOGY.csv, results/data/powerplant/CapacityToActivityUnit.csv, results/data/REGION.csv, results/data/powerplant/ResidualCapacity.csv

[Thu Oct 24 14:56:40 2024]
Job 17: Generating timeslice data...
Reason: Missing output files: results/data/CapitalCostStorage.csv, results/data/TechnologyFromStorage.csv, results/data/Conversionls.csv, results/data/SEASON.csv, results/data/Conversionld.csv, results/data/ReserveMarginTagFuel.csv, results/data/DAILYTIMEBRACKET.csv, results/data/TechnologyToStorage.csv, results/data/TIMESLICE.csv, results/data/CapacityFactor.csv, results/data/ReserveMarginTagTechnology.csv, results/data/DaySplit.csv, results/data/Conversionlh.csv, results/data/YearSplit.csv, results/data/SpecifiedDemandProfile.csv, results/data/DAYTYPE.csv, results/data/ReserveMargin.csv, results/data/STORAGE.csv

[Thu Oct 24 14:56:40 2024]
Job 14: Generating demand data...
Reason: Missing output files: results/data/SpecifiedAnnualDemand.csv

[Thu Oct 24 14:56:40 2024]
Job 1: Generating demand figures...
Reason: Missing output files: results/figs/regression.png, results/figs/projection.png

[Thu Oct 24 14:56:41 2024]
Finished job 9.
1 of 19 steps (5%) done
[Thu Oct 24 14:57:03 2024]
Finished job 14.
2 of 19 steps (11%) done
[Thu Oct 24 14:57:07 2024]
Finished job 1.
3 of 19 steps (16%) done
[Thu Oct 24 14:57:25 2024]
Finished job 13.
4 of 19 steps (21%) done
Select jobs to execute...
Execute 1 jobs...

[Thu Oct 24 14:57:25 2024]
Job 12: Generating transmission data...
Reason: Missing output files: results/data/CapitalCost.csv, results/data/InputActivityRatio.csv, results/data/FixedCost.csv, results/data/OperationalLife.csv, results/data/TECHNOLOGY.csv, results/data/FUEL.csv, results/data/ResidualCapacity.csv, results/data/TotalAnnualMinCapacityInvestment.csv, results/data/CapacityToActivityUnit.csv, results/data/TotalTechnologyModelPeriodActivityUpperLimit.csv, results/data/TotalAnnualMaxCapacityInvestment.csv, results/data/OutputActivityRatio.csv; Input files updated by another job: results/data/powerplant/CapitalCost.csv, results/data/powerplant/OperationalLife.csv, results/data/powerplant/TotalAnnualMaxCapacityInvestment.csv, results/data/powerplant/ResidualCapacity.csv, results/data/powerplant/FixedCost.csv, results/data/YEAR.csv, results/data/powerplant/OutputActivityRatio.csv, results/data/powerplant/FUEL.csv, results/data/powerplant/InputActivityRatio.csv, results/data/powerplant/TotalAnnualMinCapacityInvestment.csv, results/data/AvailabilityFactor.csv, results/data/MODE_OF_OPERATION.csv, results/data/powerplant/TECHNOLOGY.csv, results/data/powerplant/CapacityToActivityUnit.csv, results/data/REGION.csv; Params have changed since last execution: Union of exclusive params before and now across all output: before: {'trn1': ['TRNINDEAINDNE', 5, 1975, 2030, 10, 861, 30, 95], 'trn2': ['TRNINDEAINDNE', 1, 1990, 2030, 10, 861, 30, 95], 'trn3': ['TRNINDEAINDNE', 2, 2035, 2030, 10, 861, 30, 95], 'trn4': ['TRNINDNOINDSO', 0, 2020, 2025, 0.5, 900, 32, 92]} now: <nothing exclusive> 

Waiting at most 5 seconds for missing files.
MissingOutputException in rule transmission in file /home/trevor/repos/osemosys_global/workflow/rules/preprocess.smk, line 147:
Job 12  completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait:
results/data/TotalAnnualMinCapacityInvestment.csv (missing locally, parent dir contents: TotalTechnologyModelPeriodActivityUpperLimit.csv, FUEL.csv, FixedCost.csv, powerplant, REGION.csv, TotalAnnualMaxCapacityInvestment.csv, SpecifiedAnnualDemand.csv, MODE_OF_OPERATION.csv, YEAR.csv, CapitalCost.csv, InputActivityRatio.csv, ResidualCapacity.csv, CapacityFactor.csv, SpecifiedDemandProfile.csv, YearSplit.csv, CapacityToActivityUnit.csv, OutputActivityRatio.csv, TIMESLICE.csv, TECHNOLOGY.csv, AvailabilityFactor.csv, STORAGE.csv, OperationalLife.csv)
Removing output files of failed job transmission since they might be corrupted:
results/data/CapitalCost.csv, results/data/FixedCost.csv, results/data/CapacityToActivityUnit.csv, results/data/OperationalLife.csv, results/data/TotalAnnualMaxCapacityInvestment.csv, results/data/TotalTechnologyModelPeriodActivityUpperLimit.csv, results/data/InputActivityRatio.csv, results/data/OutputActivityRatio.csv, results/data/ResidualCapacity.csv, results/data/TECHNOLOGY.csv, results/data/FUEL.csv
[Thu Oct 24 14:59:28 2024]
Error in rule timeslice:
    jobid: 17
    input: resources/data/All_Demand_UTC_2015.csv, resources/data/CSP 2015.csv, resources/data/SolarPV 2015.csv, resources/data/Hydro_Monthly_Profiles (15 year average).csv, resources/data/Won 2015.csv, resources/data/Woff 2015.csv
    output: results/data/CapacityFactor.csv, results/data/TIMESLICE.csv, results/data/SpecifiedDemandProfile.csv, results/data/YearSplit.csv, results/data/STORAGE.csv, results/data/TechnologyToStorage.csv, results/data/TechnologyFromStorage.csv, results/data/Conversionls.csv, results/data/Conversionld.csv, results/data/Conversionlh.csv, results/data/SEASON.csv, results/data/DAYTYPE.csv, results/data/DAILYTIMEBRACKET.csv, results/data/CapitalCostStorage.csv, results/data/DaySplit.csv, results/data/ReserveMargin.csv, results/data/ReserveMarginTagTechnology.csv, results/data/ReserveMarginTagFuel.csv
    log: results/logs/timeslice.log (check log file(s) for error details)
    shell:
        python workflow/scripts/osemosys_global/TS_data.py 2> results/logs/timeslice.log
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Removing output files of failed job timeslice since they might be corrupted:
results/data/CapacityFactor.csv, results/data/TIMESLICE.csv, results/data/SpecifiedDemandProfile.csv, results/data/YearSplit.csv, results/data/STORAGE.csv
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-10-24T145640.190494.snakemake.log
WorkflowError:
At least one job did not complete successfully.

Anything else?

No response

maartenbrinkerink commented 1 month ago

Fixed with commit a3bec745c8e233660b6e516f6a8dbf6b5e48f25a