Post-Discretization of network infrastructure

JulianGeis commented 5 months ago

problem: electricity transmission grids and hydrogen pipelines are optimized continuously. However, in reality, grid infrastructure can only be build in discrete steps determined by cable or pipeline diameter. A continuous optimization underestimates the investment costs needed for transmission expansion.

solution: for post-discretisation of transmission lines (power, gas, hydrogen etc.), you can use the function n.optimize_transmission_expansion_iteratively() as a starting point (https://github.com/PyPSA/PyPSA/blob/7f7da29a9fd94e6ff08776d7b2f64197a2ab1c7e/pypsa/optimization/abstract.py#L20-L155). I think it could make sense to implement a configurable discretisation routine directly in this PyPSA function and then use it in PyPSA-Eur based on config.yaml settings.

Implementation according to Fabians paper https://ieeexplore.ieee.org/abstract/document/8916411

and post-facto discretization of line investment (Section III-C).
"The optimal capacities of continuously expanded lines are directly rounded to an integer value using a discretization threshold (z = 0.3) on the fractional investment (tagged postdisc) [17]. The discretized line capacities are then fixed and line impedances are adapted accordingly for a final round of generation expansion only, to find the optimal mix of generator and link capacities given the discretized line capacities."

Related PRs:

https://github.com/PyPSA/pypsa-eur/pull/1033 (pypsa-eur @ariadne2)
https://github.com/PyPSA/PyPSA/pull/871 (PyPSA)

Testing:

3h resolution solved on the cluster
memory:

What I tested: 365H network locally

all carriers are discretised: ✔
the capacity of the discrete network and a non-discrete network do not differ significantly: ✔
plots do not differ significantly: visual check ✔
difference in capacity per carrier per year: discretised / not-discretised

lindnemi commented 4 months ago

please save the continuous network and change add_investments in the exporter

JulianGeis commented 4 months ago

@lindnemi this PR can now be tested / merged. The continuous network is not saved, but you can track the continuous values in every optimization iteration via "track_iterations" in the config. The investments calculation via s_nom_min / p_nom_min are still working as this value is not overwritten in the iterative optimization. For testing you need the lastes PyPSA version (0.28) and atm the easiest way is to use ariadne2 as submodules and change solve_network.py according to this PR: https://github.com/PyPSA/pypsa-eur/pull/1064/files

lindnemi commented 4 months ago

OUTDATED: ~~Something is a bit strange with this PR. Merging was terrible and caused many unncessary conflicts. And still, there are 6 files which contain old changes. I decided to create a fresh PR. @JulianGeis please check if everything is as it should be in #96 ~~

lindnemi commented 4 months ago

I am seeing a lot of warnings about unknown variables from linopy:

INFO:additional_functionality:For DE adding following link carriers to port 4 CO2 constraint: []
INFO:linopy.model: Solve problem using Gurobi solver
INFO:linopy.model:Solver logs written to `results/20240603postdiscretize/KN2045_Bal_v4/logs/elec_s_22_lvopt__none_2030_solver.log`.
INFO:linopy.model:Solver options:
 - threads: 4
 - method: 2
 - crossover: 0
 - BarConvTol: 1e-06
 - Seed: 123
 - AggFill: 0
 - PreDual: 0
 - GURO_PAR_BARDENSETHRESH: 200
Set parameter Username
Academic license - for non-commercial use only - expires 2025-02-01
INFO:linopy.io:Writing objective.
Writing constraints.: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 54/54 [00:00<00:00, 63.28it/s]
Writing continuous variables.: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:00<00:00, 134.12it/s]
INFO:linopy.io: Writing time: 1.08s
Warning: unknown variable 'x147186' in bounds section (line 1909776)
Warning: unknown variable 'x147188' in bounds section (line 1909778)
Warning: unknown variable 'x147190' in bounds section (line 1909780)
Warning: unknown variable 'x147192' in bounds section (line 1909782)
Warning: unknown variable 'x147194' in bounds section (line 1909784)
Warning: unknown variable 'x147196' in bounds section (line 1909786)
Warning: unknown variable 'x147198' in bounds section (line 1909788)
Warning: unknown variable 'x147200' in bounds section (line 1909790)
Warning: unknown variable 'x147202' in bounds section (line 1909792)
Warning: unknown variable 'x147204' in bounds section (line 1909794)
Warning: unknown variable 'x147215' in bounds section (line 1909805)
Warning: unknown variable 'x147217' in bounds section (line 1909807)
Warning: unknown variable 'x147220' in bounds section (line 1909810)
Warning: unknown variable 'x147222' in bounds section (line 1909812)
Warning: unknown variable 'x147225' in bounds section (line 1909815)
Warning: unknown variable 'x147227' in bounds section (line 1909817)
Warning: unknown variable 'x147230' in bounds section (line 1909820)
Warning: unknown variable 'x147232' in bounds section (line 1909822)
Warning: unknown variable 'x147235' in bounds section (line 1909825)
Warning: unknown variable 'x147237' in bounds section (line 1909827)
Warning: unknown variable 'x147240' in bounds section (line 1909830)
Warning: unknown variable 'x147242' in bounds section (line 1909832)
Warning: unknown variable 'x147245' in bounds section (line 1909835)
Warning: unknown variable 'x147247' in bounds section (line 1909837)
Warning: unknown variable 'x147250' in bounds section (line 1909840)
Warning: unknown variable 'x147252' in bounds section (line 1909842)
Warning: unknown variable 'x147255' in bounds section (line 1909845)
Warning: unknown variable 'x147257' in bounds section (line 1909847)
Warning: unknown variable 'x147260' in bounds section (line 1909850)
Warning: unknown variable 'x147262' in bounds section (line 1909852)
Warning: unknown variable 'x147265' in bounds section (line 1909855)
Warning: unknown variable 'x147267' in bounds section (line 1909857)
Warning: unknown variable 'x147270' in bounds section (line 1909860)
Warning: unknown variable 'x147272' in bounds section (line 1909862)
Warning: unknown variable 'x147275' in bounds section (line 1909865)
Warning: unknown variable 'x147277' in bounds section (line 1909867)
Warning: unknown variable 'x147280' in bounds section (line 1909870)
Warning: unknown variable 'x147282' in bounds section (line 1909872)
Warning: unknown variable 'x147285' in bounds section (line 1909875)
Warning: unknown variable 'x147287' in bounds section (line 1909877)
Warning: unknown variable 'x147290' in bounds section (line 1909880)
Warning: unknown variable 'x147292' in bounds section (line 1909882)
Warning: unknown variable 'x147298' in bounds section (line 1909888)
Warning: unknown variable 'x147300' in bounds section (line 1909890)
Warning: unknown variable 'x147304' in bounds section (line 1909894)
Warning: unknown variable 'x147306' in bounds section (line 1909896)
Warning: unknown variable 'x147308' in bounds section (line 1909898)
Warning: unknown variable 'x147310' in bounds section (line 1909900)
Warning: unknown variable 'x147312' in bounds section (line 1909902)
Warning: unknown variable 'x147314' in bounds section (line 1909904)
Warning: unknown variable 'x147316' in bounds section (line 1909906)
Warning: unknown variable 'x147318' in bounds section (line 1909908)
Warning: unknown variable 'x147320' in bounds section (line 1909910)
Warning: unknown variable 'x147322' in bounds section (line 1909912)
Warning: unknown variable 'x147324' in bounds section (line 1909914)
Warning: unknown variable 'x147326' in bounds section (line 1909916)
Warning: unknown variable 'x147328' in bounds section (line 1909918)
Warning: unknown variable 'x147330' in bounds section (line 1909920)
Warning: unknown variable 'x147332' in bounds section (line 1909922)
Warning: unknown variable 'x147340' in bounds section (line 1909930)
Warning: unknown variable 'x147342' in bounds section (line 1909932)
Warning: unknown variable 'x147343' in bounds section (line 1909933)
Warning: unknown variable 'x147345' in bounds section (line 1909935)
Warning: unknown variable 'x147347' in bounds section (line 1909937)
Warning: unknown variable 'x147349' in bounds section (line 1909939)
Warning: unknown variable 'x147353' in bounds section (line 1909943)
Warning: unknown variable 'x147357' in bounds section (line 1909947)
Warning: unknown variable 'x147359' in bounds section (line 1909949)
Warning: unknown variable 'x147361' in bounds section (line 1909951)
Warning: unknown variable 'x147363' in bounds section (line 1909953)
Read LP format model from file /tmp/linopy-problem-1a1mqd33.lp
Reading time = 0.31 seconds
obj: 303031 rows, 146959 columns, 724269 nonzeros
Set parameter Threads to value 4
Set parameter Method to value 2
Set parameter Crossover to value 0
Set parameter BarConvTol to value 1e-06
Set parameter Seed to value 123
Set parameter AggFill to value 0
Set parameter PreDual to value 0
Set parameter GURO_PAR_BARDENSETHRESH to value 200
Set parameter LogFile to value "results/20240603postdiscretize/KN2045_Bal_v4/logs/elec_s_22_lvopt__none_2030_solver.log"
Gurobi Optimizer version 11.0.1 build v11.0.1rc0 (linux64 - "openSUSE Tumbleweed-Slowroll")

CPU model: AMD Ryzen 7 PRO 7840U w/ Radeon 780M Graphics, instruction set [SSE2|AVX|AVX2|AVX512]
Thread count: 8 physical cores, 16 logical processors, using up to 4 threads

Optimize a model with 303031 rows, 146959 columns and 724269 nonzeros
Model fingerprint: 0x80783bfd
Coefficient statistics:
  Matrix range     [1e-10, 5e+04]
  Objective range  [2e-02, 2e+06]
  Bounds range     [1e+00, 1e+10]
  RHS range        [2e-16, 1e+09]
Warning: Model contains large matrix coefficient range
Warning: Model contains large rhs
Warning: Model contains large bounds
         Consider reformulating model or setting NumericFocus parameter
         to avoid numerical issues.
Presolve removed 250025 rows and 52861 columns
Presolve time: 0.26s
Presolved: 53006 rows, 94098 columns, 343036 nonzeros
Ordering time: 0.50s

Barrier statistics:
 Free vars  : 217
 AA' NZ     : 9.152e+05
 Factor NZ  : 4.275e+06 (roughly 100 MB of memory)
 Factor Ops : 7.729e+08 (less than 1 second per iteration)
 Threads    : 4

                  Objective                Residual
Iter       Primal          Dual         Primal    Dual     Compl     Time
   0   3.48647078e+15 -5.13344335e+14  5.02e+10 7.66e+03  7.29e+12     1s
   1   4.04633922e+15 -5.61264761e+14  3.57e+10 4.24e+04  4.96e+12     1s
   2   4.18452766e+15 -7.47437615e+14  1.41e+10 2.07e+04  1.99e+12     1s
   3   3.52897969e+15 -7.45845178e+14  5.48e+09 7.30e+03  7.88e+11     1s
   4   2.52024232e+15 -6.60506142e+14  2.23e+09 1.43e+03  3.09e+11     1s
   5   1.05611298e+15 -5.37436061e+14  6.79e+08 2.90e+02  9.44e+10     1s
   6   3.24798365e+14 -3.77199591e+14  1.60e+08 3.37e+01  2.30e+10     1s
   7   1.76397159e+14 -2.33936254e+14  7.83e+07 1.08e+01  1.10e+10     1s
   8   4.45151644e+13 -1.33402455e+14  1.44e+07 2.92e+00  2.45e+09     1s
   9   1.84149430e+13 -4.25195795e+13  4.22e+06 3.08e+00  7.06e+08     1s
  10   9.52383650e+12 -2.31233329e+13  1.67e+06 1.48e+00  3.14e+08     1s
  11   9.05255046e+12 -2.09201677e+13  1.54e+06 1.33e+00  2.87e+08     1s
  12   7.14825120e+12 -1.60094438e+13  1.03e+06 1.00e+00  2.06e+08     1s
  13   6.37299994e+12 -1.47992005e+13  8.58e+05 9.22e-01  1.81e+08     1s
  14   5.65576459e+12 -1.29894718e+13  7.34e+05 7.84e-01  1.56e+08     1s
  15   5.26957655e+12 -1.06098909e+13  6.57e+05 6.19e-01  1.34e+08     2s
  16   4.28001392e+12 -7.51494547e+12  4.98e+05 5.32e-01  9.91e+07     2s
  17   3.39885649e+12 -4.25662505e+12  3.61e+05 5.76e-01  6.59e+07     2s
  18   2.40541892e+12 -2.95534991e+12  2.22e+05 4.98e-01  4.36e+07     2s
  19   2.20995971e+12 -2.12091887e+12  1.97e+05 3.51e-01  3.62e+07     2s
  20   1.93321811e+12 -1.40071595e+12  1.62e+05 2.23e-01  2.84e+07     2s
  21   1.31325040e+12 -8.62429009e+11  8.97e+04 1.75e-01  1.73e+07     2s
  22   1.09269682e+12 -6.81736228e+11  6.73e+04 1.54e-01  1.38e+07     2s
  23   9.32928258e+11 -4.44990532e+11  5.21e+04 1.18e-01  1.06e+07     2s
  24   8.68213145e+11 -3.00025025e+11  4.63e+04 1.02e-01  9.13e+06     2s
  25   7.41875899e+11 -2.45532514e+11  3.51e+04 9.48e-02  7.47e+06     2s
  26   6.91261774e+11 -1.11327832e+11  3.07e+04 8.25e-02  6.19e+06     2s
  27   6.34946917e+11 -2.71037216e+10  2.59e+04 7.83e-02  5.14e+06     2s
  28   5.96095887e+11  4.48325963e+10  2.26e+04 7.59e-02  4.34e+06     2s
  29   5.42697898e+11  8.38452936e+10  1.82e+04 7.43e-02  3.57e+06     2s
  30   4.94580614e+11  1.84309306e+11  1.42e+04 7.27e-02  2.53e+06     2s
  31   4.31608519e+11  2.14182807e+11  9.35e+03 6.76e-02  1.73e+06     2s
  32   4.14590216e+11  2.36612528e+11  8.07e+03 5.95e-02  1.44e+06     2s
  33   4.00162390e+11  2.56181395e+11  6.99e+03 5.05e-02  1.19e+06     2s
  34   3.90326092e+11  2.66730024e+11  6.27e+03 4.42e-02  1.04e+06     2s
  35   3.78896787e+11  2.80452144e+11  5.44e+03 4.28e-02  8.56e+05     2s
  36   3.56185723e+11  2.90831216e+11  3.81e+03 3.81e-02  5.80e+05     2s
  37   3.37034340e+11  2.94385607e+11  2.42e+03 3.06e-02  3.76e+05     2s
  38   3.16110963e+11  2.97132307e+11  9.43e+02 2.32e-02  1.59e+05     2s
  39   3.13964578e+11  2.98401922e+11  7.98e+02 1.70e-02  1.32e+05     2s
  40   3.09682907e+11  2.99288910e+11  5.07e+02 1.29e-02  8.66e+04     3s
  41   3.08598646e+11  3.00163218e+11  4.34e+02 8.88e-03  7.14e+04     3s
  42   3.07360052e+11  3.00525911e+11  3.49e+02 7.92e-03  5.75e+04     3s
  43   3.05306752e+11  3.01235226e+11  2.11e+02 5.79e-03  3.41e+04     3s
  44   3.03517785e+11  3.01497126e+11  9.40e+01 4.51e-03  1.62e+04     3s
  45   3.03234499e+11  3.01641869e+11  7.52e+01 3.44e-03  1.26e+04     3s
  46   3.02825282e+11  3.01761203e+11  4.81e+01 2.48e-03  8.21e+03     3s
  47   3.02644420e+11  3.01836931e+11  3.68e+01 2.26e-03  6.15e+03     3s
  48   3.02403569e+11  3.01864663e+11  2.13e+01 2.22e-03  3.91e+03     3s
  49   3.02301382e+11  3.01927951e+11  1.48e+01 1.57e-03  2.64e+03     3s
  50   3.02157702e+11  3.01949493e+11  6.09e+00 1.75e-03  1.37e+03     3s
  51   3.02121413e+11  3.01995720e+11  3.94e+00 9.50e-04  8.12e+02     3s
  52   3.02097805e+11  3.02007781e+11  2.57e+00 1.22e-03  5.66e+02     3s
  53   3.02079554e+11  3.02019756e+11  1.55e+00 1.16e-03  3.66e+02     3s
  54   3.02070237e+11  3.02032502e+11  1.03e+00 9.29e-04  2.28e+02     3s
  55   3.02061764e+11  3.02041272e+11  5.85e-01 6.92e-04  1.22e+02     3s
  56   3.02057758e+11  3.02043105e+11  3.83e-01 6.03e-04  8.64e+01     3s
  57   3.02054374e+11  3.02046927e+11  2.08e-01 3.70e-04  4.37e+01     3s
  58   3.02052916e+11  3.02048241e+11  1.35e-01 2.58e-04  2.74e+01     3s
  59   3.02051892e+11  3.02048941e+11  8.43e-02 1.84e-04  1.72e+01     3s
  60   3.02051534e+11  3.02049132e+11  6.74e-02 1.60e-04  1.40e+01     3s
  61   3.02051086e+11  3.02049250e+11  4.63e-02 1.44e-04  1.06e+01     3s
  62   3.02050864e+11  3.02049695e+11  3.58e-02 7.90e-05  6.82e+00     3s
  63   3.02050624e+11  3.02049776e+11  2.48e-02 6.56e-05  4.93e+00     3s
  64   3.02050374e+11  3.02049851e+11  1.32e-02 5.11e-05  3.01e+00     3s
  65   3.02050248e+11  3.02049936e+11  7.53e-03 3.33e-05  1.79e+00     4s
  66   3.02050223e+11  3.02049951e+11  6.46e-03 2.99e-05  1.56e+00     4s

Barrier solved model in 66 iterations and 3.55 seconds (4.86 work units)
Optimal objective 3.02050223e+11

INFO:linopy.constants: Optimization successful: 
Status: ok
Termination condition: optimal
Solution: 146959 primals, 303031 duals
Objective: 3.02e+11
Solver model: available
Solver message: 2

PyPSA / pypsa-ariadne

Post-Discretization of network infrastructure #62