No longer compatible with consumption data in the `demand_path` format [BUG]

[tsmbland]

Describe the bug

I tried running the model from #338 with the current dev version of MUSE, and faced a number of issues.

The first is an issue with how timeslices are defined in settings.toml. MUSE expects three levels for timeslice, but only two levels were defined for each of the timeslices in the file (e.g. winter.night should be winter.all-week.night etc). I'm not sure why this error is permitted in old versions of MUSE and not the dev version. This is a good thing and not a problem with MUSE, I'm just flagging this first as it's relevant for tacking this issue.

Anyway, when I fix this issue, the model still fails to run with the following error:

ValueError: These variables cannot be found in this dataset: ['day', 'timeslice', 'month', 'hour']

which relates to drop in line 118 of preset_sectors.py.

This seems to relate to the format that the consumption data takes in the input files. All the models we've been working with use the consumption_path format which includes timeslice data, however this model uses the demand_path format which doesn't include any timeslice data (see https://muse-os.readthedocs.io/en/latest/inputs/toml.html#preset-sectors)

I guess this is why it's complaining about trying to drop the timeslice variables, as these variables don't exist in the input data, but I'd have to dig into it a bit further to find the exact problem

To Reproduce

Try running the model from #338, fixing the error in settings.toml that I described above

Expected behavior

The model should run without error

Screenshots

If applicable, add screenshots to help explain your problem.

Context

Please, complete the following to better understand the system you are using to run MUSE.

• Operating system (eg. Windows 10):
• MUSE version (eg. 1.0.1): dev
• Installation method (eg. pipx, pip, development mode): dev
• Python version (you can get this running python --version): 3.12

[ahawkes]

Worth looking at this further, and if possible giving Sharwari a work around in the meantime. I'm not entirely clear on what each of the ..._path options are for or how they differ - is this well documented?

[tsmbland]

Sharwari's model works with the version of MUSE that she's using, just not the latest dev version (the problem is related to the changes we made when upgrading the pandas and xarray versions). The different _path options are for different ways of specifying demand/comsumption data. You use either demand_path or consumption_path depending on how the data is structured in the csv file. It's documented here https://muse-os.readthedocs.io/en/latest/inputs/toml.html#preset-sectors, but could certainly be clarified.

None of the example models we're using for testing use the demand_path format, which is why we hadn't spotted this bug until now.

[ahawkes]

Thanks - ok right. Thing is that Sharwari and others will need to use the latest version when we release it (soon hopefully) as we have fixed a lot of things. A

---

From: Tom Bland @.> Sent: Friday, June 28, 2024 4:49:53 PM To: EnergySystemsModellingLab/MUSE_OS @.> Cc: Hawkes, Adam D @.>; Comment @.> Subject: Re: [EnergySystemsModellingLab/MUSE_OS] No longer compatible with consumption data in the demand_path format [BUG] (Issue #364)

This email from @.*** originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders listhttps://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address.

Sharwari's model works with the version of MUSE that she's using, just not the latest dev version (the problem is related to the changes we made when upgrading the pandas and xarray versions). The different _path options are for different ways of specifying demand/comsumption data. You use either demand_path or consumption_path depending on how the data is structured in the csv file. It's documented here https://muse-os.readthedocs.io/en/latest/inputs/toml.html#preset-sectors, but could certainly be clarified.

None of the example models we're using for testing use the demand_path format, which is why we hadn't spotted this bug until now.

— Reply to this email directly, view it on GitHubhttps://github.com/EnergySystemsModellingLab/MUSE_OS/issues/364#issuecomment-2197215360, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC37JLOTXCLLHYJCKZWZ3YLZJWA2DAVCNFSM6AAAAABJWHWGGKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJXGIYTKMZWGA. You are receiving this because you commented.Message ID: @.***>

[tsmbland]

Ok, will prioritise next week