Inaccurate varName in logs when printing for null value attribute

[robitaillej]

When looking in the log.txt file for problems at loading time, it seems there is a small issue when printing the variable attributes that have a null value. Here is an example of what I got:

Warning: varName=units attribute=units has values=null
Warning: varName=sdn_uom_urn attribute=sdn_uom_urn has values=null
Warning: varName=sdn_uom_name attribute=sdn_uom_name has values=null
Warning: varName=sdn_uom_name attribute=sdn_uom_name has values=null
Warning: varName=units attribute=units has values=null

As you can see, the "varName" actually takes the same value as the "attribute". I have erddap version 2.23.

[ayushsingh01042003]

@ChrisJohnNOAA I think I found a relevant fix for this.

Attribute att = (Attribute)it.next(); 
String name = att.getName();
attributes.add(name, getAttributePA(name, att));

getAttributePA requires the arguments varName and attribute and here we pass the attribute name instead is probably the cause(or I may be wrong)

However I am unable reproduce the issue and cannot test the solution out, some help on how I could reproduce these Warning messages in my log.txt would be great.

For reference:

• I am using the default at the time of download datasets.xml file which is serving 70 datasets by default
• I am able to access the datasets normally via my local setup but the log file is filled with a bunch of http errors.

[ChrisJohnNOAA]

@ChrisJohnNOAA I think I found a relevant fix for this.

Attribute att = (Attribute)it.next(); 
String name = att.getName();
attributes.add(name, getAttributePA(name, att));

getAttributePA requires the arguments varName and attribute and here we pass the attribute name instead is probably the cause(or I may be wrong)

However I am unable reproduce the issue and cannot test the solution out, some help on how I could reproduce these Warning messages in my log.txt would be great.

For reference:

• I am using the default at the time of download datasets.xml file which is serving 70 datasets by default
• I am able to access the datasets normally via my local setup but the log file is filled with a bunch of http errors.

@ayushsingh01042003 I haven't run into this issue myself. @robitaillej any advice on reproducing the bad logs? Is there perhaps a specific dataset that caused the issue?

[BobSimons]

I'm speaking from memory, not having looked anything up, but I think that:

• A data source, e.g., a .nc file, may have an attribute with a null value.
• If addAttributes (in datasets.xml) has an attribute with a null value, it is telling ERDDAP to remove this attribute.
• So the message in the log is from when processing a data source.
• The message is likely to be shown by generateDatasetsXml, as it processes new data sources and prints a lot of diagnostics. It may also be shown by ERDDAP in various situations.
• I think the standard for the IOOS Glider DAC (https://gliders.ioos.us/) specifically says to use null attributes if there is an unknown value (as a placeholder). So you can probably find sample files with null attributes at the IOOS Glider DAC or by contacting John Kerfoot or someone else there.

[robitaillej]

Hello, The dataset that was causing the problem no longer does because I looped through the data in order to get rid of the attributes causing the warning. But attached is an example of a netcdf file that will cause the warning because, as Bob mentionned, there are attributes that have null values. And it does print the warning when loading the dataset, but also when requesting data from the dataset. So it quickly pollutes the logs if those attributes are left in the files.

CTD_1982001_10_10_DN.ODF.zip

[ayushsingh01042003]

@ChrisJohnNOAA

• I was wondering how can I get my local erddap to serve this .nc file. Uptill now I was just using default datasets.xml which used the < sourceUrl > and a url inside it to serve the datasets so if you could point me the relevant steps I need to do that would help.

• Also as mentioned the given dataset is causing the issue because of certain null attributes, what I want to understand is are the warnings caused due to the dataset file alone or some null tag xml that we are writing for the dataset

for example:

<dataset type="EDDGridFromDap"    datasetID="erdTAsshl1day">
    <sourceUrl>https://oceanwatch.pfeg.noaa.gov/thredds/dodsC/satellite/TA/sshl/1day</sourceUrl>
    <reloadEveryNMinutes>10080</reloadEveryNMinutes>
    <addAttributes> 
        <att name="cols">null</att>

Here, att name = 'cols' is null, so is it this piece of xml thats causing the warning or the dataset erdTAsshl1day alone causing the warning or maybe both.

(Sorry if it was not very clear, I can articulate some parts better if you need me to)

[rmendels]

@ayushsingh01042003 @ChrisJohnNOAA That THREDDS is no longer active, that is why it is failing. To link to that data use EDDGridFromErddap from https://coastwatch.pfeg.noaa.gov/erddap/griddap, datasetID erdTAsshl1day. However that will fail until about 6:00pm PDT because our building is totally shutdown for electrical work.

Also it is just good practice to check your URL before including in the datasets.xml file

[ChrisJohnNOAA]

For setting up a new dataset, it might be useful to take a look at GenerateDatasetsXml.

[rmendels]

Yes, always try to use GenerateDatasetsXml first. But for that particular dataset, that program will be unhappy until we come back up later today.

[BobSimons]

@ayushsingh01042003, This code:

    <addAttributes> 
        <att name="cols">null</att>

will not cause the error. The incorrect log message only occurs when an attribute in a source data file has a value of null.

[ayushsingh01042003]

Warning: varName= attribute=cruise_description has values=null
Warning: varName= attribute=acknowledgment has values=null
    ioos_category=Unknown for |prespr01|||prespr01|
Warning: varName=PRESPR01_QC attribute=units has values=null
Warning: varName=PRESPR01_QC attribute=sdn_uom_urn has values=null
Warning: varName=PRESPR01_QC attribute=sdn_uom_name has values=null
    ioos_category=Unknown for |dephpr01|||dephpr01|
Warning: varName=DEPHPR01_QC attribute=units has values=null
Warning: varName=DEPHPR01_QC attribute=sdn_uom_urn has values=null
Warning: varName=DEPHPR01_QC attribute=sdn_uom_name has values=null
Warning: varName=PSALST01 attribute=sdn_uom_name has values=null
    ioos_category=Unknown for |psalst01|||psalst01|
Warning: varName=PSALST01_QC attribute=units has values=null
Warning: varName=PSALST01_QC attribute=sdn_uom_urn has values=null
Warning: varName=PSALST01_QC attribute=sdn_uom_name has values=null
    ioos_category=Unknown for |temps901|||temps901|
Warning: varName=TEMPS901_QC attribute=units has values=null
Warning: varName=TEMPS901_QC attribute=sdn_uom_urn has values=null
Warning: varName=TEMPS901_QC attribute=sdn_uom_name has values=null
    ioos_category=Unknown for |sigteqst|||sigteqst|
Warning: varName=SIGTEQST_QC attribute=units has values=null
Warning: varName=SIGTEQST_QC attribute=sdn_uom_urn has values=null
Warning: varName=SIGTEQST_QC attribute=sdn_uom_name has values=null

It looks like I was successful at reproducing the logs and printing the correct variable names. If this solution seems acceptable do I raise a PR? (Just a 1 word change)

[ChrisJohnNOAA]

Yes, go ahead and make a pull request then I can take a look at it.