TMY3reader - not compatible with other formats

MartinRaetz commented 2 years ago

The TMY3 Reader of the IBPSA is used by many users as a universal weather file reader. Many use the TRY converter (http://ebc-weather.os-cloud.eonerc.rwth-aachen.de:8000/TRYconverter/) which correctly converts the format and units of the data but does not account for the different measurement times (full hour, average of hour, etc.).

The low-order model does compute the incidence angle by the hour of the year. Having the weather data and the incident angle calculation shifted can lead to severe overheating, e.g. the irradiation intensity of the afternoon is hitting the window at a 90°C angle because the model matches the afternoon irradiation with the incidence angle of the sunset.

The TMY3 Reader can only be applied to TMY3 weather data files (or correctly converted files)!

We want to understand, what is going wrong and find a less complicated way to use other weather data formats. It should also be possible to use real (no TRY, TMY, etc.) weather data, that does not necessarily cover a non-leap year from 01.01. to 31.12.

MartinRaetz commented 2 years ago

TMY3 reader: the TMY3 reader really only refers to the TMY3 data format (Users Manual for TMY3 Data Sets (nrel.gov)). The following data is shifted by half an hour: o HDirNor o HGlobHor o HDifHor All other values are not shifted. However, in TMY3 other calculations are performed, e.g. calculation of the sun angle. Conclusion: When using the TMY3 reader, it is necessary to pay close attention to the desired format, otherwise time shifts/incorrect calculations may occur.

The documentation header in the TMY3.mos files is somewhat misleading, as the documentation of columns 9-11 are used, but the measured value acquisition is not documented. Columns 12-15 have documentation "during minutes preceding the indicated time" but are not touched in Dymola.

The data set of the DWD (TRY2017) has the following properties: o Start time: 01.01. 1 o'clock CET (UTC+1) o Temperature measurement: 1/2h before given time o Wind direction/wind speed: 10-minute average interval; we could not interpret exactly, probably average of the last 10 minutes before given time o Radiation: B(direct) & D(diffuse): average radiation values for the past hour. A(atmosphere): calculated at full hour

Converter from DWD(TRY2017) data to mos file: o No time shift (1h corresponds to 3600s) o Except normal radiation values all other values are simply taken over (possibly only unit conversion) o Calculation of the radiation values: HDirNor: is calculated from B via the sun angle (sun angle is determined "correctly" including the 1/2 hour shift", we assume HDirNor is correctly determined and saved without any time-shift (it is later done in the TMY3reader)). HGlobHor: B+D HDifHor: D

MartinRaetz commented 2 years ago

After discussion we (@FWuellhorst, @KremerMartin, @DaJansenGit, @Dawac, @MartinRaetz) agreed on the following solution:

LOM and Teaser keep using the TMY3reader. All weather data has to be converted to the TMY3reader format and time management (correct shifts)
HOM will be transformed for working with the TMY3reader as well. (@FWuellhorst will create an issue for that)

MartinRaetz commented 2 years ago

A from TRY2017 to TMY3 converted weather file with notes taken while reviewing. Non-Tabbed means: available in TRY2017 and used in TMY3reader single-tabbed means: either not in TRY2017 but used in TMY3reader or other way around double-tabbed means: not used in TMY3reader

grafik

MartinRaetz commented 2 years ago

Data source view of TMY3reader to see the default values that should be set if other weather data does not provide such information.

( grafik )

MartinRaetz commented 2 years ago

Necessary steps in the "Converters":

TRY2017 (.dat) to TMY3 (.mos) Converter (https://git-ce.rwth-aachen.de/ebc/ebc-general/python/ebc-weather-tool/-/blob/master/TRYconverter/views.py)

Header of mos file from the converter should be more informative and converter specific
- note which measurements are unused (Dummies)
- note which measurements have been processed and shortly how
- note which measurements have been shifted and how
- Temperatures have to be shifted by half an hour (to meet TMY3reader logic)

Real weather data puller for Modelica output (https://git-ce.rwth-aachen.de/ebc/ebc-general/python/ebc-weather-tool/-/tree/master)

Header of mos file should contain similar information as mentioned above
state correct time zone (should be UTC+1?)
check for all pulled data and whether it represents the indicated time or the average of the previous or following hour and shift accordingly (required for all output formats)
for Modelica output make sure that the time-shifting fits the TMY3reader logic (all data measured at the indicated time, except HDirNor, HGlobNor and HDifNor should represent the average of the previous hour before the indicated time)
the Modelica output should always start at 0 = 01.01. at 0:00 at the indicated time zone -> there is no need to add dummy values, just make sure the time stamps are consistent, e.g. 02.01. 10:00 (at indicated time zone) = (24+10)*3600. I do not yet know how to handle leap years.
make sure the weather file is not one year long (difference between first time stamp (e.g. 0) and last time stamp cannot be 365*243600) -> check for length and add dummy time stamp to end of file A one year weather data would result in a repeating the file in an for real data unwanted way. ` parameter Modelica.SIunits.Time lenWea=weaDatEndTim - weaDatStaTim "Length of weather data"; parameter Boolean canRepeatWeatherFile=abs(mod(lenWea, 36524*3600)) < 1E-2 `
- possibly the best way is to create a "UTC TimeStamp=MeasurementTime" to "TMY" converter, which could be used for any weather data. That UTC to TMY converter would need to be combined with a dwd weather data pull producing a "UTC TimeStamp=MeasurementTime" dataframe, which itself can be more easily used in other applications.

For every converter:

rework the whole header, delete everything that´s not used or contains information
don't use 99999 as dummy values for the unused parameters, use e.g. -0 for better visibility
don't use dummy values to fill up nans! Fill up only the specifically unused parameters!
use standard values if a required measurement is not available. The standard value can be found in the tab "Data source" of the TMY3reader, see picture below. If the standard value is used, note it at the header of that specific measurement with "no measurement, filled with default" Info: you should be able to add or delete as many lines to the "header" as long as they are commented with a leading #.

MartinRaetz commented 2 years ago

I want to suggest to use the following .mos header. If applicable, entries should be filled dynamically by the respective converter. The file itself: TMY3_Converter_HeaderSuggestion.txt (open with NotePad++ for correct intendation)

#1: for TMY3reader
double tab1(8760,30)
#LOCATION,,,,,,49.52,8.55,1.0,
#Explanation of Location line:
#   Element 7: latitude
#   Element 8: longitude
#   Element 9: time zone in hours from UTC, e.g. time zone = 1 -> all measured data is UTC+1
#
#DATA PERIODS, data available from 01.01.XXXX 00:00:00 (second=0) to 01.01.XXXX 00:00:00 (second=3.15324E7)
#info: preceding period filled to meet TMY3´s requirement: Time 0 = 01.01. 00:00:00 at local time)
#
#USED CONVERTER: TRY2017 (converted at 27.01.2022 19:55:33)
#info: the AixLib/IBPSA TMY3reader needs all data like mentioned at: OutConverter
#
#C1 Time in seconds. Beginning of a year is 0s.
#C2 Dry bulb temperature in Celsius,                    InConverter: average preceding hour, OutConverter: at indicated time
#C3 Dew point temperature in Celsius,                   InConverter: f(C2,C4), OutConverter: at indicated time 
#C4 Relative humidity in percent,                       InConverter: at indicated time, OutConverter: at indicated time 
#C5 Atmospheric station pressure in Pa,                         InConverter: at indicated time, OutConverter: at indicated time, per default not used in TMY3reader 
#C6 Extraterrestrial horizontal radiation in Wh/m2, not used in TMY3reader
#C7 Extraterrestrial direct normal radiation in Wh/m2, not used in TMY3reader
#C8 Horizontal infrared radiation intensity in Wh/m2,           InConverter: at indicated time, OutConverter: at indicated time
#C9 Global horizontal radiation in Wh/m2,               InConverter: average preceding hour, OutConverter: average preceding hour
#C10 Direct normal radiation in Wh/m2,                  InConverter: average preceding hour f(solar angle, radiation horizontal), OutConverter: average preceding hour 
#C11 Diffuse horizontal radiation in Wh/m2,             InConverter: average preceding hour, OutConverter: average preceding hour  
#C12 Averaged global horizontal illuminance in lux, not used in TMY3reader
#C13 Direct normal illuminance in lux, not used in TMY3reader
#C14 Diffuse horizontal illuminance in lux, not used in TMY3reader
#C15 Zenith luminance in Cd/m2, not used in TMY3reader
#C16 Wind direction N=0, E=90, S=180, W=270,            InConverter: at indicated time, OutConverter: at indicated time 
#C17 Wind speed in m/s,                                 InConverter: at indicated time, OutConverter: at indicated time 
#C18 Total sky cover in 10th at indicated time,                 InConverter: at indicated time, OutConverter: at indicated time 
#C19 Opaque sky cover at indicated time,                InConverter: at indicated time, OutConverter: at indicated time 
#C20 Visibility in km, not used in TMY3reader
#C21 Ceiling height in m,                               InConverter: at indicated time, OutConverter: at indicated time 
#C22 Present weather observation, not used in TMY3reader
#C23 Present weather codes, not used in TMY3reader
#C24 Precipitable water in mm, not used in TMY3reader
#C25 Aerosol optical depth, not used in TMY3reader
#C26 Snow depth in cm, not used in TMY3reader
#C27 Days since last snowfall, not used in TMY3reader
#C28 Albedo, not used in TMY3reader
#C29 Liquid precipitation depth in mm at indicated time, not used in TMY3reader
#C30 Liquid precipitation quantity, not used in TMY3reader

MartinRaetz commented 2 years ago

@Dawac @DaJansenGit @FWuellhorst: Could you review the suggested .mos file header and let me know if you agree with it.

FWuellhorst commented 2 years ago

Nice work, thanks a lot! I fully agree, only a small note: "InConverter" and "OutConverter" is a bit fuzzy to me. What to you think about: "Raw" and "In Modelica" or something to indicate what converter means? I assume OutConverter is the input for Modelica?

MartinRaetz commented 2 years ago

Thank you, I swopped things around a little and think they are better now. Though, i felt like "raw" could be missunderstood as people my interpret the values in that file as raw values. Since the converter in stated in the header anyway, i felt like ConverterInput should be clear? Let me know, if I did not get your point!

#1: for TMY3reader
double tab1(8760,30)
#LOCATION,,,,,,49.52,8.55,1.0,
#Explanation of Location line:
#   Element 7: latitude
#   Element 8: longitude
#   Element 9: time zone in hours from UTC, e.g. time zone = 1 -> all measured data is UTC+1
#
#DATA PERIODS, data available from 01.01.XXXX 00:00:00 (second=0) to 01.01.XXXX 00:00:00 (second=3.15324E7)
#info: preceding period filled to meet TMY3´s requirement: Time 0 = 01.01. 00:00:00 at local time)
#
#USED CONVERTER: TRY2017 to TMY3 (converted at 27.01.2022 19:55:33)
#info: the AixLib/IBPSA TMY3reader needs all data in the unit and time as mentioned below
#
#C1 Time in seconds. Beginning of a year is 0s.
#C2 Dry bulb temperature in Celsius at indicated time, ConverterInput: average preceding hour
#C3 Dew point temperature in Celsius at indicated time, ConverterInput: f(C2,C4) 
#C4 Relative humidity in percent at indicated time, ConverterInput: at indicated time 
#C5 Atmospheric station pressure in Pa at indicated time (not used per default), ConverterInput: at indicated time
#C6 Extraterrestrial horizontal radiation in Wh/m2, not used in TMY3reader
#C7 Extraterrestrial direct normal radiation in Wh/m2, not used in TMY3reader
#C8 Horizontal infrared radiation intensity in Wh/m2 at indicated time, ConverterInput: at indicated time 
#C9 Global horizontal radiation in Wh/m2 as average of preceding hour, ConverterInput: average of preceding hour
#C10 Direct normal radiation in Wh/m2  as average of preceding hour, ConverterInput: average of preceding hour f(solar angle, radiation horizontal) 
#C11 Diffuse horizontal radiation in Wh/m2 as average of preceding hour, ConverterInput: average of preceding hour  
#C12 Averaged global horizontal illuminance in lux, not used in TMY3reader
#C13 Direct normal illuminance in lux, not used in TMY3reader
#C14 Diffuse horizontal illuminance in lux, not used in TMY3reader
#C15 Zenith luminance in Cd/m2, not used in TMY3reader
#C16 Wind direction N=0, E=90, S=180, W=270 at indicated time, ConverterInput: at indicated time
#C17 Wind speed in m/s at indicated time, ConverterInput: at indicated time  
#C18 Total sky cover in 10th at indicated time at indicated time, ConverterInput: at indicated time 
#C19 Opaque sky cover at indicated time at indicated time, ConverterInput: at indicated time 
#C20 Visibility in km, not used in TMY3reader
#C21 Ceiling height in m at indicated time, ConverterInput: at indicated time 
#C22 Present weather observation, not used in TMY3reader
#C23 Present weather codes, not used in TMY3reader
#C24 Precipitable water in mm, not used in TMY3reader
#C25 Aerosol optical depth, not used in TMY3reader
#C26 Snow depth in cm, not used in TMY3reader
#C27 Days since last snowfall, not used in TMY3reader
#C28 Albedo, not used in TMY3reader
#C29 Liquid precipitation depth in mm at indicated time, not used in TMY3reader
#C30 Liquid precipitation quantity, not used in TMY3reader

MartinRaetz commented 2 years ago

Additionally, I want to mention that some parameters, despite being processed by the TMY3reader are not used in the LOM. They might possibly be used in some other models. Should we state that information somewhere? If yes where?

Used in LOM: TDryBul RelHum TBlaSky (calculated via: HInfHor, TDryBul, TDewPoi and OpaSkyCov) HGloHor HDifHor HDirNor

Not used LOM: total sky cover ceiling height wind direction wind speed atmospheric pressure

FWuellhorst commented 2 years ago

If we use the same reader for the HOM now, we will still need at least wind speed. You should state that in the LOM. The weather reader reads the weather for different models, stating something in the file for one specific model would not be good in my opinion.

MartinRaetz commented 2 years ago

I agree, that's what I thought. Though, I don't know where to document it. I could add it to the info of the multizone, though I think that will be overseen easily and retrieving the information directly through checking the weaBus usages will not take much longer than reading the info. And there would be the danger of the info being wrong if something is changed in the multizone.

I suggest not documenting it in Modelica but adding a docstring in the converter.

FWuellhorst commented 2 years ago

Please also check for this ongoing issue: https://github.com/ibpsa/modelica-ibpsa/issues/1375 It seems at least one of our issues will be solved.

tobinfiedler commented 2 years ago

@MartinRaetz What is the current status?

MartinRaetz commented 2 years ago

Thanks for checking by @tobinfiedler!

The changes are currently under review in the following PR: https://git-ce.rwth-aachen.de/ebc/ebc-general/python/ebc-weather-tool/-/merge_requests/18

After merging to master, we are only waiting for this Issue to be solved: https://github.com/ibpsa/modelica-ibpsa/issues/1375 As this Issue is inactive for quite some time, I have asked about the status :)

RWTH-EBC / AixLib

TMY3reader - not compatible with other formats #1222

Necessary steps in the "Converters":