search

cioos-siooc / cioos-pacific-erddap

CIOOS Pacific ERDDAP main repository used to maintain, develop and integrate ERDDAP datasets.
https://data.cioospacific.ca/erddap
2 stars 0 forks source link

[Issue]: IOS_ADCP_Moorings bad source files #51

Open JessyBarrette opened 1 year ago

JessyBarrette commented 1 year ago

Contact Details

@jessy.barrette

Level

Low

Issue Description

ERDDAP is reporting a conflict for some ADCP source files where the _FillValue assigned to some variables is different than others. This seems to be affecting a significant number of files

_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.

Issue Solution

Perhaps the _FillValue was changed overtime by the conversion tool. We could rerun the conversion tool on all the historical data.

Relevant extra information (log output,etc.)

https://localhost:8443/erddap reports:

********************************************
List of Bad Files for datasetID=IOS_ADCP_Moorings

/datasets/IOS_ADCP_moorings/a1_20080430_20080918_0489m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/a1_20130707_20140524_0498m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/a1_20170715_20180614_0027m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/buw02_20151108_20160530_0041m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/chat2_20200728_20210609_0102m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/chat3_20210608_20220721_0042m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/chat3_20210608_20220721_0152m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/dixon1_20210609_20220720_0102m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/dixon1_20210609_20220720_0202m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/dixon1_20210609_20220720_0326m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/e01_20210602_20220715_0097m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/e03_20210602_20220714_0388m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/fortune1_20171007_20180714_0090m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/hak1_20210604_20220718_0132m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/hak1_20210703_20220430_0042m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/juan2_20210605_20220718_0044m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/juan2_20210605_20220720_0213m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/lemmen1_20181127_20190514_0030m.adcp.L1.nc
java.lang.RuntimeException: Your query produced too much data.  Try to request less data. [memory]  The request needs more memory (345 MB) than is currently available in this Java setup (241 MB). (DoubleArray)
 at com.cohort.util.Math2.ensureMemoryAvailable(Math2.java:516)

/datasets/IOS_ADCP_moorings/quat1_20211011_20220716_0041m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/quat1_20211011_20220716_0101m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/quat1_20211011_20220716_0178m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/quat2_20211011_20220716_0041m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/quat2_20211011_20220716_0123m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/rant1_20190619_20191009_0005m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/scott3_20210603_20220717_0045m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/scott3_20210603_20220718_0230m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/srn1_20210606_20220719_0043m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/srn1_20210606_20220719_0184m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

/datasets/IOS_ADCP_moorings/srn1_20210606_20220719_0187m.adcp.L1.nc
java.lang.RuntimeException: 
ERROR in Test.ensureEqual(double):
_FillValue sourceAttribute value observed!=expected for sourceName=LCEWAP01_QC.
Specifically: 1.0E15 != 0.0
 at com.cohort.util.Test.error(Test.java:43)

********************************************

  tFileNamePA.size()=378
  dirTable.nRows()=1
  fileTable.nRows()=349
    fileTableInMemory=false
    nUnchanged=0
    nRemoved=29 (nNoLastMod=0, nNoSize=0)
    nReadFile=378 (nDifferentModTime=0 nNew=378) readFileCumTime=12m 44s avg=2023ms