Price devices distinctively in scheduling

[nhoening]

Based on #615, this feature allows specifying price sensors per device, flexible or unflexible.

In the flex-contextparameter, we already have consumption-price-sensor and production-price-sensor. They will remain the default (see https://flexmeasures.readthedocs.io/en/latest/api/notation.html#flex-context).

New parameters consumption-price-sensors-per-device and production-price-sensors-per-device can optionally be added to overwrite this default, so that some devices are priced by other price sensors. These parameters are dicts, mapping the power sensor IDs of devices (assets) to price sensor IDs.

TODO:

• [ ] Adapt FlexContextSchema (in data.schemas.scheduling.__init__.py). This should suffice to make the API parse these new parameters and pass them on.
• [ ] Also add these parameters to the CLI command flexmeasures add schedule for-storage (in cli/data_add.py), so they end up in flex_context (line 1054ff). This should help with testing locally.
• [ ] Get prices for these distinct price sensors from the database in data.models.planning.storage.py::StorageScheduler.compute_schedule() (also add the price slope trick for these) and pass them on to device_scheduler(). I'd say we keep them as a dict until here
• [ ] In device_scheduler, we need to adapt the model so that per device (denoted by "d" in the ConcreteModel), the correct price is applied. I believe the best places are price_down_select() and price_up_select().
• [ ] Add a scheduling unit test. Idea: In data.models.planning.test_solver.py::test_building_solver_day_2(), we could add a market_scenario parameter for the case that the price for solar (feed-in tariff) is much higher than the tariff used by the battery (the default). Then we can expect the battery to never charge. The pytest fixture create_test_tariffs might be a good place to add the extra price sensor and data.
• [ ] Add documentation for the new parameters, in the flex context description.
• [ ] Bonus: Adding an integration test with the API (api.v3_0.tests.test_sensor_schedules.py), which is crucial for the actual operation. However, the existing tests are not including the flex context a lot, anyway, so I can't demand it from you now.

[Flix6x]

I have only an implementation detail / clarification to add. I don't think the new prices per device should overwrite the price that can be set currently.

The current price is being applied to the aggregate power flow of the whole portfolio of devices being scheduled. This represents, for example, a building's net power flow to/from the grid.

The new price per device should be applied to the power flow to/from a single device and should be implemented as an additional costs, imo. This represents, for example, a device's usage-dependent depreciation costs.

[nhoening]

I see how you came to this, but I guess one does not have to view the portfolio as aggregated (on one contract). That seems to be our default view (like, for a building or industrial site), but maybe it does not apply always. In fact, the intention here seems to be that the individually-priced devices are not part of the portfolio. Maybe that can be clarified by stakeholders.

If the two signals are vastly different (two different price curves), the original intention would become harder to maintain (one would need to compute the difference between these two independent price curves).

[nhoening]

I added a TODO item, so this feature is also available through the CLI.

[Srieon]

hi @nhoening I was trying to implement the toy example using the API endpoints so that i can extend it to do the above changes discussed in the issue. All the endpoints are working fine except for the /api/v3_0/sensors/(id)/schedules/trigger which is used to create the schedule. It gives an "[Error 99 connecting to localhost:6379. Cannot assign requested address]" This seems to be an error regarding the redis configuration ( judging by the port number) but the toy example is working fine with CLI .

[nhoening]

Hi @Srieon, the toy example using the CLI would by default not require Redis, as the --as-job parameter is False per default.

The API expects to be able to queue jobs for execution, so a Redis node is needed. You can also run one via Docker, btw, see the Docker-compose file.

[Srieon]

hey @nhoening I was trying to access the trigger endpoint "/api/v3_0/sensors/(id)/schedules/trigger" with the request body consisting of both consumption-price-sensor and production-price-sensor. It throws an error if one of them is missing . Ideally shouldn't they both be same. I can see that at various places you have implemented that if consumption-price-sensor is missing consumption-price-sensor =production-price-sensor. But this endpoint doesn't have that . Is there a reason for it ? Could I just add that line so that my request body only requires one price sensor?

[nhoening]

Hi @Srieon, I agree that often they are the same. I see that in the CLI command production prices default to consumption prices, but right now I don't see where else we do that. The endpoint code would not be the best place to do that, as it doesn't parse the flex context. This point where we actually do that would be better. (and then the line in the CLI command might be able to go).

And we should document this defaulting better in the relevant documentation.

Supporting this defaulting better could be a small PR in itself, actually.

Does this help?

[anirudh-ramesh]

Hi @nhoening, the first couple of items to modify i.e. adding parameters in order for the API and CLI to parse the new fields for the sensor dictionary. We're stuck in knowing if this makes sense to you. Perhaps I could issue a PR into which we can keep adding commits and you could keep reviewing until all of these criteria are met?

Cc @Srieon

[nhoening]

Yes a PR might be a good idea. You can set it to draft status if it's not ready to be merged, but still ask for a review.

P.S. the first sentence of your last post seems to be missing a verb, so I did not get what you were saying there 😃

[anirudh-ramesh]

Ah, yes, sorry, I didn't proofread it. I meant to say that we'd implemented the first two items on the list.

[rajath-09]

Hello @nhoening

@anirudh-ramesh recently submitted a PR that made changes to parse new prices from distinct sensors to the concrete model. However, I've encountered an issue where the model is currently infeasible because the power selection is not being done. I understand that the objective function with sense=minimise should take care of the price selection, but I'm not sure how to proceed with the power selection to ensure that it happens according to the price selection using the sensor dictionary that associates the power and price sensors.

Can you please provide guidance on how I can make the model feasible and ensure that the power selection happens based on the price selection using the newly added sensor dictionary parameters? I would greatly appreciate any help or advice on this matter. Thank you."

[nhoening]

Hi @rajath-09 ― glad to help, but "the power selection is not being done" is for me not specific enough. Maybe you can share excerpts from a log file, a screenshot or (best) reproducible steps (e.g. a script constructing and running an example), so we can see the problem first hand. And when you say "currently", what do you mean? In the current state of this PR?

[rajath-09]

Hi @nhoening By "the power selection is not being done" ,what i mean is that ,isn't the model supposed to be scheduled according to both the power and price sensor data provided and the association between them?For instance,if for a particular date-time frame,price from particular sensor is selected,isn't the associated power sensor be only selected? Yes by "currently" i mean the current state of this PR.

[rajath-09]

Hi @nhoening I wanted to understand what the "planned_costs" imply in the scheduling algorithm. For instance if I have solar plants,load,battery and grid, what all costs does the "planned_costs" involve?

[rajath-09]

Also is there a feature which could be used to get the planned costs for individual assets?

[nhoening]

Tagging @Flix6x and @victorgarcia98 here who might get you an answer quicker than I can.

[rajath-09]

Could you help me out here? @Flix6x @victorgarcia98

[victorgarcia98]

Hello Rajath!

The planned cost per device is returned by the function device_scheduler in line 196. You can see how it is computed here. As far as I can see, these planned costs are not saved.

[Flix6x]

Also is there a feature which could be used to get the planned costs for individual assets?

No. This in itself would be a great feature request, imo, though. It would require something like passing a cost_sensor_id to the scheduler and using it to save the computed costs as well (for total costs). The total costs can be gotten from the model results after it solved. Costs per device are not stored on the model yet, but can be gotten by multiplying the power of each device (which is stored on the model) with the price (also stored). Saving these would require passing some mapping of devices (columns passed to the device_scheduler) to cost_sensor_ids.

If you need to compute these costs with features currently available, you might run a reporter after the scheduler has saved its results. The reporter_config should multiply each schedule with the prices. Bare in mind that reporters are kind of a new feature. A dedicated CLI command for running reporters will be released in v0.14 (it has recently become available on main already). @victorgarcia98 can sensor_search_configs be used to select only sensor data sourced by the scheduler?

[victorgarcia98]

@victorgarcia98 can sensor_search_configs be used to select only sensor data sourced by the scheduler?

At the moment, the sensor_search_configs can take a source id in its values so you could get that.

This could be done using the PandasReporter with a reporter_config similar to the following:

{
    "beliefs_search_configs": [
        {
            "sensor": 1,
            "source" :  1,
            "alias" : "power",
            "resolution" : "PT1H"
        },
        {
            "sensor": 2,
            "alias": "price",
            "resolution" : "PT1H"
        }
    ],
    "transformations": [
        {
            "df_input": "price",
            "method": "droplevel",
            "args": [
                [
                    1,
                    2,
                    3
                ]
            ]
        },
        {
            "df_input": "power",
            "method": "droplevel",
            "args": [
                [
                    1,
                    2,
                    3
                ]
            ]
        },
        {
            "df_output" : "energy_cost_df",
            "df_input" : "power",
            "method" : "multiply",
            "args" : [ "@price" ]
        }
    ],
    "final_df_output": "energy_cost_df"
}