IGSDB serving ICON json formatted data

[StephenCzarnecki]

I have added some initial functionality to allow igsdb-icon.herokuapp.com to be able to return product data formatted based on the ICON json schemas. It is not everything but I think there is enough that it is worth reviewing. A few notes:

1. The mechanism for getting the ICON-formatted product data is a querystring parameter named json_format. If that is set to "icon" the returned json will be based on the ICON schemas. The querystring parameter cannot be "format" because that seems to already be reserved by the framework we are using.
2. The value in locator field in the graphql metadata field has been changed accordingly. So now it contains a url to igsdb-icon.herokuapp.com and also contains the querystring set to return the ICON format. I realize that the metatadata resourceTree can potentially contain links to both formats. But for the moment at least I wanted to keep it simple while evaluating the ICON json data. And if we want to have multiple resources for different formats we can add that in later.
3. I have not updated the hashValue fields yet. If this is important let me know and I can do it. I have not because the functionality for return ICON json data is in flux and so therefore is the calculated hash.
4. This really only works for glazing products and not shading layers so far. I expect it would work for shade materials as well but haven't tried. There are no restrictions in place based on product type but for now please only focus on glazing products.

And a question that came up when implementing this regarding the timestamp in the optical.json and the dataPoints in the opticalData.json. The dataPoints section seems to be where to put both measured values as well as calculated values. For example the measured values for the prime surface for wavelength .305 might look like

{
  "incidence":{
    "direction":{
      "polar":0
    }
  },
  "emergence":{
    "direction":{
      "polar":0
    }
  },"
  wavelengths":{
    "wavelength":0.305
  },
  "results":{
    "transmittance":0.0,
    "reflectance":0.0520000010728836
  }
}

And the direct-hemispherical calculated integrated results for the solar spectrum on the prime side might look like this

{
  "incidence":{
    "direction":{
      "polar":0
    }
  },
  "emergence":{
    "direction":"hemispherical"
  },
  "wavelengths": "solar",
  "results":{
    "transmittance":0.5483174,
    "reflectance":0.05789395
  }
}

And that seems like it will work out. Because when the measured data changes then the integrated results get recalculated. However there is a potential issue with the infrared values. Because the IGDB allows submitters to provide integrated infrared results instead of individual measured values in that range. And those values seem like they should also go into the dataPoints section. E.g. one might look like

{
  "incidence":{
    "direction":"hemispherical"
  },
  "emergence":{
    "direction":"hemispherical"
  },
  "wavelengths":"infrared",
  "results":{
    "transmittance":0.0,
    "absorptanceEmittance":0.839999973773956
  }
}

Where the potential issue might arise is that currently the IGSDB stores the ir transmittance and emissivity in a different location than either the measured values or other integrated results. So it is at least conceptually possible that the measured values and other integrated results have one timestamp and the infrared values have a different timestamp.

I can think of at least a couple potential ways of dealing with this but wanted to discuss it:

1. Just always use the spectral data timestamp and not worry about it.
2. Have multiple entries in the optical section of the json. One for the measured spectral data, one for infrared values and one for the integrated results. Maybe this is the best since it will allow for different integrated results based on the optical standard. So there would be one entry for measured spectral data, one entry for submitter-supplied ir values, and then one entry for each optical standard used to calculate the other integrated results.

[christoph-maurer]

I recommend to use

"polar": 8,

for nearnormal instead of 0°. Reflectance cannot be measured at 0° and transmittance is often measured at 8°, too.

I recommend to use json_format=bed or json_format=buildingenvelopedata.org instead of json_format=icon so that we don't need to explain icon in the years to come.

When you find the time, it could be useful to implement resources in addition to resourceTree.

[StephenCzarnecki]

@christoph-maurer I have changed the angle from 0 to 8. I have also changed to using json_format=buildingenvelopedata.org. I thought that was clearer and did not leave room for confusion like using bed might. These changes are now live on https://igsdb-icon.herokuapp.com/ e.g. https://igsdb-icon.herokuapp.com/api/v1/products/196/?json_format=buildingenvelopedata.org

[StephenCzarnecki]

I made a few fixes and added some more fields that just return TODO. I tried to only pick fields that relate to glazing products and have not included that only relates to shading layers. And I only picked fields that I thought the IGSDB might currently have data for. However there may be things I added that are not appropriate for glazings and things I missed that are.

While doing so I remembered another thing that may impact the issue of how to format the optical data. The IGSDB has integrated results for wavelength bands other than solar and visible. E.g. the are transmittance results for UV and SPF ranges. The opticalData.json schema does not have those ranges. It just has an other defined with the following comment:

...Otherwise, use the value "other" and make sure that its meaning is given in the description of the method used to generate this data set, for example, in the method's reference."

I have not added those results to the ICON json data. It seems that if that data were to be included (and I am not saying here if it should or not) then these other results would need to have their own entries in the list of optical data. Because each would need to add information about which wavelength ranges were used to calculate the result. E.g. the UV results would specify that they were created using the TUV method and the SPF results would specify they were created using the SPF method.

However when I look at the standards file that is used to calculate those values those methods have a note saying not in NFRC 300-2003. And they are not the only methods. Here are a list of methods that are used to calculate results that have the note not in NFRC 300-2003:

• Color Tristim X, Y, and Z
• Thermal IR
• TUV
• SPF
• TKR

So it seems that the methods that are defined in NFRC 300-2003 are

• Solar
• Photopic
• TDW

I am not sure exactly how this impacts things at this point but wanted to mention what I found.

[christoph-maurer]

@StephenCzarnecki I used https://igsdb-icon.herokuapp.com/api/v1/products/196/?json_format=buildingenvelopedata.org and made it valid against the schemas in commit ebc26437f993e0fd1dd41a3e184dc4c9e6054235 . I don't know how much this helps, because especially the enumerations don't allow any tags like "TODO" or "9999".

[StephenCzarnecki]

@christoph-maurer This was very helpful, thanks for doing it. I have corrected the buildingenvelopedata.org formatted json accordingly. I think using lots of 9s is clear enough and provides an easy way to search for fields that are not yet implemented in the IGSDB. Let me know if you find any more things to correct.

Note that there are still some fields in that response for 196 that are null and not "9999" such as description, non-prime transmittance, and permeabilityFactor. I did not make those "9999" because they are actually implement and are not present for the 196 record. I think figuring out what to do about nulls as a different issue from figuring out which additional fields need to be implemented for the demos. It's possible to have logic like "return prime transmittance if non-prime transmittance is missing" or "return 1 for permeabilityFactor if it is missing" but I think that is something that needs to be figured out still, possibly on a case-by-case basis.

[christoph-maurer]

An example for symmetric transmittance is

{
  "components": [
    {
      "id": "a99941df-5bb8-4080-89a4-4f3d230afeb6",
      "optical": [
        {
          "id": {
            "value": "b8435146-5270-46f5-b925-4b00de265823",
            "issuedBy": "ce588e6a-1d3c-4909-afdd-bcde4c33864e"
          },
          "data": [
            {
                "componentCharacteristics": {
                  "surface": "symmetric"
                },
                "dataPoints": [
                  {
                    "incidence": {
                      "direction": {
                        "polar": 8
                      },
                      "wavelengths": { "wavelength": 300 }
                    },
                    "emergence": {
                      "direction": "hemispherical"
                    },
                    "results": {
                      "transmittance": 0.002,
                      "reflectance": 0.047
                    }
                  }

[christoph-maurer]

If the IGSDB has no data to return for a key of the opticalData, than they should not return this key instead of returning it with the value null. Do you agree, @simon-wacker ?

[simon-wacker]

Well, I don't really know the context so it's impossible to answer. What do you mean by key? What exactly is that thing you refer to as opticalData? What request/query is being made? If you mean id by key and mean the opticalData query by opticalData, then the answer is as follows:

The query opticalData has [OpticalData!] as return type, which means that for the query query { opticalData(id: "....") {...} }, if the given id exists, the data is accessible, the query itself is well-formed, and so forth, then the corresponding data must be returned, and if the id does not exist (or the query itself contains errors) then those errors must be reported.

As a side note: The response of a GraphQL query is a JSON object with two possible properties data and errors, where on success data would be a JSON object itself with the property opticalData (and errors would not be given at all) and on failure data would not be given at all and errors would be an array with error information (at least that's the usual case). From the GraphQL website

If there were no errors returned, the "errors" field should not be present on the response. If no data is returned, according to the GraphQL spec, the "data" field should only be included if the error occurred during execution.

[simon-wacker]

And if the "query" being made is an HTTP(S) GET request to some resource URL, then if that resource exists and is accessible, the data should be returned with HTTP Status Code 200, and if not, then the data should not be returned but instead some error information with the corresponding HTTP Status Code.

[simon-wacker]

In general regarding whether something can be null or not depends on the schema specification: If the property is allowed to be null, then it can be null and if not then not. It's that simple actually. And if that means that something can be null that you don't want to be null, then you have to adapt the schema.

[christoph-maurer]

@StephenCzarnecki I talked to @simon-wacker . My comment

If the IGSDB has no data to return for a key of the opticalData, than they should not return this key instead of returning it with the value null.

is correct.

If you want to validate a data set against the JSON Schemas, you can clone the repository and copy the data set file into building-envelope-data/examples/dbe/optical. Then use make shell and make example to validate the examples. To validate a single data set, you can use a command like npx --no-install ajv --spec=draft2019 -c ajv-formats validate -s ./schemas/dbe.json -d ./examples/dbe/optical/integralAccordingToStandard.json -r ./schemas/layer.json -r ./schemas/method.json -r ./schemas/optical.json -r ./schemas/identifier.json -r ./schemas/dsb.json -r ./schemas/photovoltaicData.json -r ./schemas/building.json -r ./schemas/calorimetricData.json -r ./schemas/component.json -r ./schemas/gaps.json -r ./schemas/common.json -r ./schemas/geometry.json -r ./schemas/string.json -r ./schemas/stakeholder.json -r ./schemas/opticalData.json -r ./schemas/number.json -r ./schemas/hygrothermalData.json -r ./schemas/hygrothermal.json -r ./schemas/calorimetric.json -r ./schemas/unit.json -r ./schemas/photovoltaic.json -r ./schemas/material.json.

[christoph-maurer]

@StephenCzarnecki The unit of wavelength is nanometers. Can you therefore please multiply the current values with 1000?

[RDmitchell]

Is it possible to specify the units, and therefore we are not changing the form of our original data?

I think it would be better for us not to be doing "math" on the data that we are making available.

[christoph-maurer]

@RDmitchell We don't plan to introduce multiple units for one key in order to make it easier for software companies to use the data and for planners to search the data. We also think that nanometer is a reasonable unit for wavelengths. Would it be much effort to convert the wavelengths in nanometers?

[RDmitchell]

@christoph-maurer -- it doesn't seem great that we are transforming data that was submitted to us, which I believe is the data we are talking about, but Jacob Jonsson should be the person to weigh in on this. It doesn't seem like he is a member of this repo, but possibly should be.

[StephenCzarnecki]

I have updated https://igsdb-icon.herokuapp.com/ to include some of the changes discussed at the meeting yesterday.

I updated the json returned using the json_format=buildingenvelopedata.org querystring to only return data specified by the opticalData.json schema.

I also updated the wavelengths to be nanometers by multiplying by 1000 when creating the json.

Let me know if you have any issues with the changes or anything else should be changed.

[christoph-maurer]

It seems to work well that IGSDB serves data sets about glass panes according to JSON format of buildingenvelopedata.org which is defined in this repository.

Currently, we don't know when we can implement that IGSDB serves data sets about shading according this JSON format.