Read and check the optical schemas and examples #42

Closed simon-wacker closed 1 year ago

In GitLab by @christoph-maurer on Jul 28, 2020, 16:02

@bruno.bueno @helen.rose.wilson Please do the following things

read the schemas and examples and check them for possible issues
create several realistic BSDF examples which cover together all keys defined by optical.json
if possible, create examples which should not be allowed and verify whether they are invalid against the schema
check if all optical data can be represented according to the schema or extensions of the schema are needed

Your work has the following targets

validate the schema by trying it for the first time
validate that other persons can learn to use the schema
create examples for convincing use cases for follow-up projects
create examples for documentation and dissemination so that others can learn more quickly
create examples which serve as a check regarding uncontrolled changes of the schema

Please let @christoph-maurer know whenever you need anything. @helen.rose.wilson plans to work on the topic in August and informs @bruno.bueno about her progress who works on the topic in September.

@HelenRoseWilson and BrunoBueno The example /optical/bsdfVerbose.json can be used to create more detailed and realistic examples. Regarding example bsdfIntermediate.json, it probably makes sense to implement first subschemas for Klems and Tensor-Tree, see #55.

Regarding example bsdfResource.json, we have been calling it bsdfFile.json in the past, but we want to be able to share streams as well as files. The metadata about a dataset will be written in GraphQL language and only the data will be in JSON and valid for example against opticalData.json. For resources, there will be no schema but full flexibility. But we need to define a format at IKDB, for example by a media type and a reference to a publication. The publication could be a PDF with a few sentences describing the format.

Instead of bsdfResource.json, we need an example in GraphQL language for the representation of data as resource. @simon-wacker will prepare a general example. When this is ready (it may take a while), then you could for example try to save examples for the whole workflow between raw data, different forms of derived data and the representation in intermediate and verbose form. That would probably help us a lot for dissemination.

Bruno Bueno commented:

@christoph-maurer
@HelenRoseWilson

I've reviewed the schemas: optical.json, opticalData.json and the examples: bsdfFile.json, bsdfIntermediate.json and bsdfVerbose.json. I have some comments/requests:

Where can I define whether opticalData corresponds to Front/Back, e.g. transmission front.
The description of opticalData.json and optical.json is the same. When do we use one or the other?
We need an additional property: results/properties/BSDF. I believe the BSDF values in the output file of pg2 are consine corrected. @HelenRoseWilson can you please confirm? we need to check if the BSDF values in Klems format are also cosine corrected.
In bsdfVerbose.json, line 24, "wavelenghts" and "range" only applies to "incidence". What about "emergence"? is it assumed to have the same as "incidence"? In Klems for example, wavelenghts and range is one level above, so it applies to both "incidence" and "emergence".
In bsdfIntermediate.json, line 41, I don't understand how "direction/list" is being used. I would expect some mention to "incidence", "profile", "polar angle".
Can we define a new direction schema where we have a LowerTheta, UpperTheta and nPhis like in Klems format?

--

@HelenRoseWilson please focus on

[ ] read the optical.json and opticalData.json
[ ] read the optical examples
[ ] check if all optical data can be represented according to the schema or extensions of the schema are needed

I also find it important to check that things that are not supposed to be representable cannot be represented. This is less easier to grasp but one should always have an eye on that when reading the JSON Schema.

Dear @bbuenoun , thank you for reviewing the schemas optical.json, opticalData.json and the examples bsdfFile.json, bsdfIntermediate.json and bsdfVerbose.json. I try to answer your comments/requests.

Where can I define whether opticalData corresponds to Front/Back, e.g. transmission front.

"definitionOfSurfacesAndPrimeDirection" provides the option to define which surface is named prime, for example with a reference to a publication or standard. "surface" can be used to define whether the incidence is on the prime or nonPrime surface or whether the sample is symmetric. tree.json is a test which uses "surface" in Line 55, but not yet "definitionOfSurfacesAndPrimeDirection". In order to improve this, I have created #140 .

The description of opticalData.json and optical.json is the same. When do we use one or the other?

optical.json is used to define the metadata about a dataset and opticalData.json is used to define the data itself. In this way, I can create a cryptographic approval for example of a spectrum according to opticalData.json and the approval is still valid if the name of the spectrum changes because this is in the metadata according to optical.json. Usually, we use both. In an example, "optical" in Line 5 uses optical.json and "data" in Line 27 uses opticalData.json. The differentiation between calorimetric.json and calorimetricData.json, photovoltaic.json and photovoltaicData.json is the same.

We need an additional property: results/properties/BSDF. I believe the BSDF values in the output file of pg2 are consine corrected. @HelenRoseWilson can you please confirm? we need to check if the BSDF values in Klems format are also cosine corrected.

It's an important topic, therefore I opened #142 to discuss it.

I need to take a break and will answer 4.-6. as soon as possible.

In bsdfVerbose.json, line 24, "wavelenghts" and "range" only applies to "incidence". What about "emergence"? is it assumed to have the same as "incidence"? In Klems for example, wavelenghts and range is one level above, so it applies to both "incidence" and "emergence".

In bsdfVerbose.json "wavelengths" and "range" apply to "incidence" and also to "emergence", because no "shiftedWavelengths" is defined within "emergence".

In bsdfIntermediate.json, line 41, I don't understand how "direction/list" is being used. I would expect some mention to "incidence", "profile", "polar angle".

In the form of a resource, the directions of the incidence may be defined in an XML file like

        <IncidentDataStructure>Columns</IncidentDataStructure>
        <AngleBasis>
            <AngleBasisName>LBNL/Klems Full</AngleBasisName>
            <AngleBasisBlock>
            <Theta>0</Theta>
            <nPhis>1</nPhis>
            <ThetaBounds>
                <LowerTheta>0</LowerTheta>
                <UpperTheta>5</UpperTheta>
            </ThetaBounds>

In intermediate form, "direction": [0, 1, 0, 5, ...] just lists the same four values. The format for the lists is defined here.

Can we define a new direction schema where we have a LowerTheta, UpperTheta and nPhis like in Klems format?

Yes. You can create a new format, for example a modification of "LBNL/Klems Full" which gives you full flexibility how to define the directions of incidence as a list. If you use the existing format "LBNL/Klems Full", you can change the values of LowerTheta, UpperTheta and nPhis via the values of the list. But you cannot change the pattern with which the list is interpreted as LowerTheta, UpperTheta and nPhis.

Dear Christoph, I hope that writing an email is the correct way to respond to this notification. I understand the entries referring to theta but how do you specify the actual phi boundary values for a specific entry? If nPhis is the number of phi subdivisions in a Klems-type format, you also need to specify the phi positions of the "first" phi boundary values and also whether the phi values are incremented in a clockwise or an anticlockwise direction. Best regards, Helen Rose.

Am 03.11.2020 um 18:13 schrieb Christoph Maurer:

In bsdfIntermediate.json, line 41, I don't understand how "direction/list" is being used. I would expect some mention to "incidence", "profile", "polar angle".

In the form of a resource, the directions of the incidence may be defined in an XML file like

|Columns
LBNL/Klems Full 0 1 0 5 | In intermediate form, the |["direction"](https://github.com/ise621/building-envelope-data/blob/6af9034b0964133386100d484d7bd9fdcf8e5afb/examples/dbe/optical/bsdfIntermediate.json#L28): [0, 1, 0, 5, ...]| just lists the same four values. The format for the lists is defined at (https://github.com/ise621/building-envelope-data/blob/6af9034b0964133386100d484d7bd9fdcf8e5afb/examples/dbe/optical/bsdfIntermediate.json#L16 ). 6. Can we define a new direction schema where we have a LowerTheta, UpperTheta and nPhis like in Klems format? Yes. You can create a new format, for example a modification of "LBNL/Klems Full" which gives you full flexibility how to define the directions of incidence as a list. If you use the existing format "LBNL/Klems Full" , you can change the values of LowerTheta, UpperTheta and nPhis via the values of the list. But you cannot change the pattern with which the list is interpreted as LowerTheta, UpperTheta and nPhis. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or unsubscribe .

-- Dr Helen Rose Wilson Solar Building Envelopes Energy-Efficient Buildings Fraunhofer Institute for Solar Energy Systems Heidenhofstr. 2, 79110 Freiburg i. Br., Germany Phone (bus.): +49 761 4588-5149 Phone (priv.): +49 761 441247 helen.rose.wilson@ise.fraunhofer.de

Web: www.ise.fraunhofer.de https://www.ise.fraunhofer.de

Dear @HelenRoseWilson , yes, it seems to work fine with the email which becomes automatically a comment which the others can read, too. I think the procedure to calculate the actual phi boundary values based on nPhi needs to be defined exactly by the reference. This reference also needs to specify the phi positions of the "first" phi boundary values and whether the phi values are incremented in a clockwise or an anticlockwise direction.

If you want to deviate from that format and reference, you can either create a new format for intermediate representation or use verbose representation.

An alternative would be to not only have the current intermediate and verbose representations of the data but also special representations for Klems and other formats. Or to have multiple intermediate representations in the schema where some values are not lists but multi-dimensional matrices for dimensions 2, 3, ... In general, if there already is a standardized format for Klems for example, then it is advisable to just use that format (instead of turning it into JSON).

Dear @HelenRoseWilson , I have outsourced Bruno's work on product data examples for solar control to #148. The only thing left to do in this issue are for you to

[ ] read the optical.json and opticalData.json
[ ] read the optical examples
[ ] check if all optical data can be represented according to the schema or extensions of the schema are needed

Please let us know when there is progress and whenever we can assist.

An alternative would be to not only have the current intermediate and verbose representations of the data but also special representations for Klems and other formats. Or to have multiple intermediate representations in the schema where some values are not lists but multi-dimensional matrices for dimensions 2, 3, ... In general, if there already is a standardized format for Klems for example, then it is advisable to just use that format (instead of turning it into JSON).

LBNL/Klems is by far the most widely exchanged BSDF format. I believe one can reference a Klems xml file in bsdfResource.json, thus we don't need a json Klems schema. On the other hand, the way the numerical values are separated in the xml file may differ depending on the program which generates the xml file (Window, genBSDF, etc). I had sometimes difficulties parsing BSDF data from Klems xml files because of that. I'm not sure whether this fact favors a json Klems schema.

I think we should definetely use bsdfResource.json for tensor-trees.

And we should make sure that bsdfVerbose.json remains as flexible as possible. We might have to spend some time on that.

I don't understand the current bsdfIntermediate.json format which mixes theta, nPhi, lowerTheta, upperTheta and so on. We may consider eliminating this format.

Yes, I agree. It would be good to standardize how the numerical values are separated (if that has not been done yet and the implementations are just sloppy). Is there some format specification by LBNL available in the WWW where we could check that? Could we talks with LBNL and ask them to standardize the separator, or a list of separators that need to be supported, (if that is necessary)? If the separator or list of separators is already standardized, then we should tell the software companies behind the implementations, that the should adhere to the standard an support all standardized separators.

@simon-wacker The XML Klems format is given in section 6.4 in the WINDOW 6.1 / THERM 6.1 Research Version User Manual here. Please check if the specification is unique. Otherwise, I'll contact the LBNL colleagues

I just had a look at section 6.4. There is no definition of the schema but just an example. In the example for the content of the XML tag ScatteringData it says "long string with space delimited all the values of the BSDF – contact LBNL (WINDOWHelp@lbl.gov) for details about the data format if you want to use this feature". The example itself specifies http://windows.lbl.gov BSDF.xsd as the schema URL, which is not a valid URL, and the URL http://windows.lbl.gov/BSDF.xsd does not exist. Someone else asked for the URL in https://discourse.radiance-online.org/t/xml-schema-for-bsdf-data/4709 which used to be http://windows.lbl.gov/BSDF-v1.4.xsd but had not existed anymore back then and only got the reply that there is a copy on Google Drive https://drive.google.com/file/d/1nJpSsgn8-gWwyd8cEOxMAEeJd0AevmBk/view by Taoning Wang. It is unclear whether this copy is the current version. Do you know a valid URL of the schema? It sould be equal to the value of the attribute xsi:schemaLocation in your XML files.

I searched LBNL-ETA in github for xsd and didn't find it neither.

Our working group has read and checked the optical schemas and examples with the available resources. Additional comments and reviews are welcome anytime.

building-envelope-data / api

Read and check the optical schemas and examples #42