Specify "ordered sets" to simplify synthax

[irm-codebase]

What can be improved?

Calliope makes use of roll() and get_val_at_index() in cases were the position of a set matters. This is often used for clustering and storage balancing constraints.

Although this works, I think the approach has two weaknesses:

• For sets defined by names, like nodes and techs, using these functions would not make sense. We currently do not protect against it.
• It's very wordy, and sort of contradicts the "simple to read" approach of Calliope

Proposed improvement

Some optimization libraries, like pyomo, allow you to specify "ordered sets". I think this makes a lot of sense in our case, and would enable easier syntax and checks.

We currently do not define the dimensions directly, but I think something like this would be good:

dimensions:
  nodes:
  techs:
  carriers:
  timesteps:
    ordered: true
  investsteps:
    ordered: true

Syntax: allowed ONLY for ordered dimensions

• 0: first position
• -1: last position
• << n: roll / shift n backwards
• >> n: roll / shift n forward

Then, a constraint like

link_storage_level:
  description: Link the storage level at the end of one investment step to the start of the next.
  foreach: [nodes, techs, investsteps]
  where: storage AND NOT investsteps=get_val_at_index(investsteps=0)
  equations:
    - expression: >-
        storage[timesteps=$initial_step] == roll(
          storage[timesteps=$final_step] * (
            (1 - storage_loss) ** timestep_resolution[timesteps=$final_step]
          ),
          investsteps=1
        )
  slices:
    initial_step:
      - expression: get_val_at_index(timesteps=0)
    final_step:
      - expression: get_val_at_index(timesteps=-1)

May turn to:

link_storage_level:
  description: Link storage level... but easier this time!
  foreach: [nodes, techs, investsteps]
  where: storage AND NOT investsteps == investsteps[0]
  equations:
  - expressions: >-
      storage[timesteps[0]] == storage[timesteps[-1], investsteps<<1] * (
        (1-  storage_loss)**timestep_resolution[timesteps[-1]]
      )

or (my least favourite)

link_storage_level:
  description: Link storage level... but easier this time!
  foreach: [nodes, techs, investsteps]
  where: storage AND NOT investsteps[0]
  equations:
    - expression: >-
        storage[timesteps=$initial_step] == storage[timesteps=$final_step, investsteps<<1] * (
            (1 - storage_loss) ** timestep_resolution[timesteps=$final_step]
          )
  slices:
    initial_step:
      - expression: timesteps[0]
    final_step:
      - expression: timesteps[-1]

Version

v0.6.10

[irm-codebase]

Defining ordered sets in theory should also allow us to perform more complicated summations with conditionals (like "sum new capacity installed for all investsteps lower than this one").

I've yet to land on an approach that is easy to read for these cases, though...

[brynpickering]

I'm not keen to add all this extra core syntax math - the helper functions are there to extend the math in a way that remains modular. Although they are wordy, they are relatively explicit (they can certainly be given better names!). "<<" and ">>" are creating a whole new syntax, and "[-1]" and "[0]" overlap with existing slicing syntax which will make them difficult to parse.

You're right that those two helpers are specific to ordered sets and we have no way of controlling for someone applying them in the wrong context. We could have some check in the helpers for either user-defined metadata (ordered: true) or it being a datetime/numeric dtype. As a user, you could feasibly ensure the ordering of your sets by reorganising the input data dimensions, we just need to provide an API for that.

[irm-codebase]

I'm fine with not implementing the extra syntax :) However, a personal wish is to use ordered sets for numeric comparisons. Kind of: sum over this dimension while it is lower than [some position]. Would be a god send for pathways math.

I will close this issue as "not planned", and rewrite another with your feedback in mind.