search

ugr-sail / sinergym

Gym environment for building simulation and control using reinforcement learning
https://ugr-sail.github.io/sinergym/
MIT License
127 stars 34 forks source link

[Question] Different Action Spaces? #395

Closed kad99kev closed 5 months ago

kad99kev commented 6 months ago

Question ❓

Are there any environments that do not have an action space of shape 2? In the documentation for previous versions here, it shows that there are multiple environments with a Box(4), Box(5) etc., but when I go into the configuration JSON files on GitHub, they have only two actions defined. Could you please assist with this?

Thank you!

kad99kev commented 6 months ago

Related to the same question, when I use the Eplus-datacenter-hot-continuous-stochastic-v1 enviornment, I get 100% comfort violations all the time. On further inspection, it seems like the actions are only being applied to the west zone and the east zone is ignored. When I save the zone indoor temperature from the observations, only the west zone is within the defined comfort range. The east zone constantly get values above the comfort zone (i.e. approx 28 degree Celsius). Is there a way to redefine the action space to ensure that both zones are controlled?

AlejandroCN7 commented 6 months ago

Hello @kad99kev!

We reduced the action spaces of the default Sinergym environments to make them simpler. In the environment configuration files, you can define those building components you want to control and the new action space (documentation here). Or you can directly use the constructor of the environment class.

In the case of the datacenter, as the building is designed, the temperature setpoints are applied to both zones simultaneously. However, as both zones are not the same, they do not have the same results. If you want to control each zone separately, you need to have explicit knowledge about the building model and change it yourself. Another option would be to modify the parameters of the reward function so that it only takes into account the west zone, but it would optimize only based on that zone of course.

I hope this is helpful! Regards

kad99kev commented 6 months ago

Hi @AlejandroCN7.

Thank you so much for your reply. I completely understand your answer.

However, there might be a bug in the latest version of the data center. Irrespective of the weather, configuration, or heating/cooling setpoint values, the east zone indoor temperature does not change at all. When I track the indoor temperatures, only the west zone seems to be influenced by the setpoint temperatures. Could you please confirm this on your end? I have noticed this behaviour on versions 2.5.2, 3.1.0 and 3.20.

AlejandroCN7 commented 6 months ago

Then, I will check the data you tell me and see if it is related to any of the Sinergym versions. I leave this issue open and as soon as I have more information I will comment here.

Thank you very much for the information and help @kad99kev!

kad99kev commented 6 months ago

Thank you @AlejandroCN7! Do let me know if you observe the same thing. I will be happy to share my observations once you get back to me!

AlejandroCN7 commented 5 months ago

Hi @kad99kev!

I've researched this topic and here's what I found. I tried downloading the sample Datacenter file provided by the EnergyPlus v23.1.0 installation. The only significant change between this file and the one we use in Sinergym is as follows:

Captura desde 2024-03-26 12-16-09

However, this change actually worsens the internal temperature of the east zone:

prueba

So, if that part was well-controlled before, I assume it's related to one of the EnergyPlus updates. The only changes made to the building in the updates were to upgrade its version and convert it to epJSON using EnergyPlus's automatic tools.

For testing, I think it's best to focus on optimizing the west zone until we have more information. For now, I will update the default reward functions for this building to focus only on this zone.

If you find anything that could help or improve this, please feel free to make a PR or comment on an issue.

Thank you very much for everything!

AlejandroCN7 commented 5 months ago

I've attempted to control each zone independently, but unfortunately, it seems to have no impact on the east zone. Should I make any progress, I will reopen this issue.

Thank you very much for everything and I apologize for any inconvenience.

kad99kev commented 5 months ago

Hi @AlejandroCN7,

Thank you for the updates! I apologise for the naive questions, but the east zone issue is occurring irrespective of the datacenter file, meaning it is a EnergyPlus v23.1.0 issue? Would updating EnergyPlus to v23.2.0 fix it? Is there a way to flag this problem to EnergyPlus?

Thank you so much for working on this!

AlejandroCN7 commented 5 months ago

Hi @kad99kev,

Maybe the latest EnergyPlus update will fix the problem, I can't tell you at this time. As the simulation engine is pending update anyway, I will retest after the update to see if there has been any improvement.

Thank you very much and sorry for the inconvenience. Regards.

kad99kev commented 3 months ago

Hello @AlejandroCN7! I was just wondering if I were to update the EnergyPlus version from 23.1.0 to 23.2.0 to see if there is still a Datacenter issue, would that affect the library's workings? Would there be any side effects that would not be very obvious?

I haven't tried it yet, so I thought I would ask you in case you have already. Thank you!