Move featured scenarios from pro to beta

mabijkerk commented 1 month ago

In the PR for Heat improvements (https://github.com/quintel/etengine/pull/1386), a migration was run on an outdated nl2019 dataset. The dataset was fixed before deploying to pro. This means that the scenarios on pro have been migrated properly, but the scenarios on beta haven't. The goal therefore is to move the II3050v2 and IP2024 scenarios from pro to beta.

[ ] II3050v2 scenarios for 2050
[ ] II3050v2 scenarios for 2040
[ ] IP2024 scenarios for 2035
[ ] IP2024 scenarios for 2030
[ ] KEV scenarios for 2030
[ ] IP2024 scenarios for 2025

@kndehaan is this something you could pick up? We should aim to do so before beta gets to far ahead of pro again.

@noracato could you list the instructions on how to do so?

noracato commented 1 month ago

This is done, right @kndehaan? Can we close?

kndehaan commented 1 month ago

@noracato unfortunately not yet.. It turns out that I still don't get the same values from the featured scenarios on pro to beta. I think obtaining balanced values could be a piece of the puzzle. I get back to you on how to obtain balanced values from scenarios through the API.

kndehaan commented 3 weeks ago

I'm quite stuck here. I've updated the featured scenarios on beta with the scenario user_values, custom curves and custom orders. However, there's still a difference in the dashboard item values between the pro and beta.

I added functionalities to scenario-tools to get the custom curves and custom orders (https://github.com/quintel/scenario-tools/pull/40), as well as balanced user values (still work in progress on upload-improvements branche). I get the correct output here.

However, when trying to upload the same settings, the following don't seem to work:

set custom orders (excluding heat network orders which is already working on production) through upload_custom_order in the ETM_API
set balanced values through update_inputs_balanced_values in the ETM_API The following DO work:
set custom curves
set user values (except for flh_xxx_user_curve

So the not-working components is something where @noracato or @louispt1 could have a look some time? I think something goes wrong in the upload_xxx functions, see the working branche upload-improvements (it's quite in draft so ignore all comments and unfinished functions).

However, when I manually copy paste the user values, balanced values and custom orders from the pro to the beta scenarios in ETEngine, and when the custom curves are all uploaded, I would expect the same scenario outcomes. Is there something that I'm missing? @noracato can you have a look at this..? We could also have a look together.

noracato commented 3 weeks ago

Sorry to hear @kndehaan! We'll do our best to unstuck you!

I think the balanced values are not processed very well by the API. But as they are auto balanced, passing the argument autobalance: true to the request where you set the sliders should do the trick as well. (At least hypothetically)

The custom order API did change end of last year. I remember updating it for heat networks indeed. Let's check if the other custom orders also need a similar update!

quintel / etengine

Move featured scenarios from pro to beta #1431