Add MODA daily OC 1-km files to ERDDAP server

[dotis]

For the NOAA Multi-stressor project, we want to add MODA 1-km daily OC files to the IMaRS server. There is not much room on dune:/mnt/sdb. There is room on /sdc, but not quite enough. The 1-day files for the GoM are around 800GB. @7yl4r please advise.

[7yl4r]

The erddap server is still showing the x,y length of the images instead of the lat,lon. We may need to mess with the dataset.xml some more.

[7yl4r]

Getting an OutOfMemory error when trying to run DasDds the conventional way. Found this by cating DasDds.sh and adding a 0 to the Xms and Xmx:

docker exec -it erddap bash -c "cd webapps/erddap/WEB-INF/ && java -cp classes:../../../lib/servlet-api.jar:lib/* -Xms10000M -Xmx10000M gov.noaa.pfel.erddap.DasDds "$@" -verbose"

That runs DasDds with way more RAM available to the JVM.

[7yl4r]

attempts at dataset.xml

result	dataset.xml config
lat & lon values wrong	latitude & longitude as generated by GenerateDataset.xml w/o lat,lon dataVars
lat & lon values wrong	latitude & longitude as generated by GenerateDataset.xml w/ lat,lon dataVars
DasDds error: large number of "Removing duplicates"	latitude & longitude modified to use lat,lon sourceVar w/ lat,lon dataVars

DasDds output for the modified case:

Which datasetID (default="")
? moda_oc_1d_gom

*** DasDds moda_oc_1d_gom
*** deleting cached dataset info for datasetID=moda_oc_1d_gom
File2.deleteIfOld(/erddapData/dataset/om/moda_oc_1d_gom/                 ) nDir=   0 nDeleted=   2 nRemain=   0

EDD.oneFromDatasetsXml(moda_oc_1d_gom)...
SimpleXmlReader encoding=UTF-8

*** constructing EDDGridFromFiles(xmlReader)...

*** constructing EDDGridFromFiles moda_oc_1d_gom
axis0 ***fileName format=yyyyDDD class=DOUBLE regex=A(\d{7})_GOM_OC_1D\.nc captureGroup=1
sourceAxisNames="***fileName,timeFormat=yyyyDDD,A(\\d{7})_GOM_OC_1D\\.nc,1", lat, lon
sourceDataNames=Rrs_667, chlor_a, STRAYLIGHT_MASK, ABI, Kd_490, lat, lon
sourceDataTypes=float, float, float, float, float, double, double
dir/file table doesn't exist: /erddapData/dataset/om/moda_oc_1d_gom/dirTable.nc
dir/file table doesn't exist: /erddapData/dataset/om/moda_oc_1d_gom/fileTable.nc
creating new dirTable and fileTable (dirTable=null?true fileTable=null?true badFileMap=null?false)
doQuickRestart=false
365 files found in /mnt/sdb/moda_oc_1d_gom/
regex=.*\.nc recursive=true pathRegex=.* time=26ms
old nBadFiles size=0
sortTime=2ms
old fileTable size=0   nFilesMissing=0
#0 file=/mnt/sdb/moda_oc_1d_gom/A2023001_GOM_OC_1D.nc
#0 inserted in cache
#1 file=/mnt/sdb/moda_oc_1d_gom/A2023002_GOM_OC_1D.nc
#1 inserted in cache
[...]
#364 file=/mnt/sdb/moda_oc_1d_gom/A2023365_GOM_OC_1D.nc
#364 inserted in cache
fileTable updated; time=60038ms
2nd sortTime=1ms

  tFileNamePA.size=365
  dirTable.nRows=1
  fileTable.nRows=365
    fileTableInMemory=false
    nUnchanged=0
    nRemoved=0 (nNoLastMod=0, nNoSize=0)
    nReadFile=365 (nDifferentModTime=0 nNew=365) readFileCumTime=54.137 s avg=148ms
  Table.saveAsFlatNc /erddapData/dataset/om/moda_oc_1d_gom/dirTable.nc1766359582 done. nColumns=1 nRows=1 TIME=19ms
  Table.saveAsFlatNc /erddapData/dataset/om/moda_oc_1d_gom/fileTable.nc1766359582 done. nColumns=9 nRows=365 TIME=20ms
save fileTable(first 5 rows)=
dirIndex,fileList,lastMod,size,nValues,min,max,csvValues,startIndex
0,A2023001_GOM_OC_1D.nc,1680186243000,110437092,1,1.6725312E9,1.6725312E9,1.6725312E9,0
0,A2023002_GOM_OC_1D.nc,1680186284000,110437232,1,1.6726176E9,1.6726176E9,1.6726176E9,1
0,A2023003_GOM_OC_1D.nc,1680186326000,110437092,1,1.672704E9,1.672704E9,1.672704E9,2
0,A2023004_GOM_OC_1D.nc,1680186359000,110436952,1,1.6727904E9,1.6727904E9,1.6727904E9,3
0,A2023005_GOM_OC_1D.nc,1680186388000,110436812,1,1.6728768E9,1.6728768E9,1.6728768E9,4
...

getting metadataFrom /mnt/sdb/moda_oc_1d_gom/A2023365_GOM_OC_1D.nc
  ftLastMod first=2023-03-30T14:24:03Z last=2024-01-06T06:03:16Z
  latitude: DoubleArray isn't sorted in ascending order: [2144]=31.0 > [2145]=30.990911744270562.
[...]
  Removing duplicates at [3067331] and [3067332] = 18.01288256263424
  Removing duplicates at [3067332] and [3067333] = 18.01288256263424
  Removing duplicates at [3067333] and [3067334] = 18.01288256263424
  Removing duplicates at [3067334] and [3067335] = 18.01288256263424
  Removing duplicates at [3067335] and [3067336] = 18.01288256263424
  Removing duplicates at [3067336] and [3067337] = 18.01288256263424
  Removing duplicates at [3067337] and [3067338] = 18.01288256263424
  Removing duplicates at [3067338] and [3067339] = 18.01288256263424
  Removing duplicates at [3067339] and [3067340] = 18.01288256263424
  Removing duplicates at [3067340] and [3067341] = 18.01288256263424
  Removing duplicates at [3067341] and [3067342] = 18.01288256263424
  Removing duplicates at [3067342] and [3067343] = 18.01288256263424
  Removing duplicates at [3067343] and [3067344] = 18.01288256263424
  Removing duplicates at [3067344] and [3067345] = 18.01288256263424
  Removing duplicates at [3067345] and [3067346] = 18.01288256263424
  Removing duplicates at [3067346] and [3067347] = 18.01288256263424
  Removing duplicates at [3067347] and [3067348] = 18.01288256263424
  Removing duplicates at [3067348] and [3067349] = 18.01288256263424

 at gov.noaa.pfel.erddap.variable.EDVGridAxis.<init>(EDVGridAxis.java:108)
 at gov.noaa.pfel.erddap.variable.EDVLatGridAxis.<init>(EDVLatGridAxis.java:42)
 at gov.noaa.pfel.erddap.dataset.EDDGrid.makeAxisVariable(EDDGrid.java:386)
 at gov.noaa.pfel.erddap.dataset.EDDGridFromFiles.<init>(EDDGridFromFiles.java:1221)
 at gov.noaa.pfel.erddap.dataset.EDDGridFromNcLow.<init>(EDDGridFromNcLow.java:96)
 at gov.noaa.pfel.erddap.dataset.EDDGridFromNcFiles.<init>(EDDGridFromNcFiles.java:103)
 at gov.noaa.pfel.erddap.dataset.EDDGridFromFiles.fromXml(EDDGridFromFiles.java:302)
 at gov.noaa.pfel.erddap.dataset.EDD.fromXml(EDD.java:445)
 ... 4 more

[7yl4r]

Dan is going to try flipping over the image so lat is in ascending order.

We have seen this issue before: https://groups.google.com/g/erddap/c/h4YITDjp26w/m/_4FkXEJABwAJ

[dotis]

There is a new set of test images in /mnt/sdb/moda_oc_1d_gom.

These are made just like the 7D mean files that are on ERDDAP. We may need a new "dataset.xml" file.

[dotis]

I have changed things a bit. The 1D files for the entire Gulf of Mexico were too large. So, I created subsetted files for Florida only.

They are on dune: /mnt/sdb/moda_oc_1d_fl

I will try to create a dataset.xml file and ingest them.

[dotis]

And it's working! Test images are up on the ERDDAP site. Woot.

[dotis]

moda_oc_1d_gom has been deleted.

[dotis]

Opening this again. I have moved the images to /mnt/sdc on dune and updated the dataset.xml. I see that the datasets.xml has updated with the change, but I don't see the images on the ERDDAP site. I still see the old ones that are on /mnt/sdb, which seems weird.