search

marinebon / sdg14-shiny

Shiny apps related to MBON SDG14
https://marinebon.github.io/sdg14-shiny
1 stars 0 forks source link

Pull in data from ERDDAP #10

Closed 7yl4r closed 5 years ago

7yl4r commented 6 years ago

@bbest 's plan for a daily cronjob sound like a good way to handle this to me. I don't see an easy way to implement real-time pulls from ERDDAP.

Probably a note about the age of the data should be included.

7yl4r commented 6 years ago

Real-timeness of data? cron job for:

7yl4r commented 6 years ago

New plan that better aligns with our plans to ally with Axiom :

spin up an ERDDAP server here at IMaRS

7yl4r commented 6 years ago

In the meantime Ben will modify the frontend to pull in (lower-res) ERDDAP data from existing NOAA sources. Details here.

7yl4r commented 6 years ago

ERDDAP server running at 192.168.1.100 (IMaRS internal access only for now). Config files used are in https://github.com/USF-IMARS/erddap-config.

Next up is to try loading Dan's chl data using EDDGridFromNcFiles.. or perhaps EDDGridFromNcFilesUnpacked... :thinking:

7yl4r commented 6 years ago

hmm... why isn't this working?

Which EDDType (default="EDDGridFromNcFilesUnpacked")
? 
Parent directory (default="/srv/imars-objects/gom/chlor_a_l3_pass")
? /srv/imars-objects/gom/chlor_a_l3_pass/
File name regex (e.g., ".*\.nc") (default=".*\.hdf")
? 
Full file name of one file (or leave empty to use first matching fileName) (default="chlor_a_l3_pass_A2018251202000.hdf")
? 
ReloadEveryNMinutes (e.g., 10080) (default="")
? 
working...

*** EDDGridFromNcFilesUnpacked.generateDatasetsXml
fileDir=/srv/imars-objects/gom/chlor_a_l3_pass/ fileNameRegex=.*\.hdf sampleFileName=chlor_a_l3_pass_A2018251202000.hdf
reloadEveryNMinutes=10080
externalAddGlobalAttributes=null
Let's see if netcdf-java can tell us the structure of the sample file:
file not found= chlor_a_l3_pass_A2018251202000.hdf
java.io.FileNotFoundException: chlor_a_l3_pass_A2018251202000.hdf (No such file or directory)
 at ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:350)
 at ucar.unidata.io.RandomAccessFile.acquire(RandomAccessFile.java:224)
 at ucar.nc2.NetcdfFile.getRaf(NetcdfFile.java:575)
 at ucar.nc2.NetcdfFile.open(NetcdfFile.java:421)
 at ucar.nc2.NetcdfFile.open(NetcdfFile.java:394)
 at ucar.nc2.NetcdfFile.open(NetcdfFile.java:381)
 at ucar.nc2.NetcdfFile.open(NetcdfFile.java:369)
 at gov.noaa.pfel.coastwatch.griddata.NcHelper.openFile(NcHelper.java:640)
 at gov.noaa.pfel.erddap.dataset.EDDGridFromNcLow.generateDatasetsXml(EDDGridFromNcLow.java:404)
 at gov.noaa.pfel.erddap.dataset.EDDGridFromNcFilesUnpacked.generateDatasetsXml(EDDGridFromNcFilesUnpacked.java:78)
 at gov.noaa.pfel.erddap.GenerateDatasetsXml.doIt(GenerateDatasetsXml.java:293)
 at gov.noaa.pfel.erddap.GenerateDatasetsXml.main(GenerateDatasetsXml.java:834)

when

imars-admin@imars-physalis:~$ ls -lh /srv/imars-objects/gom/chlor_a_l3_pass/chlor_a_l3_pass_A2018251202000.hdf
-rw-r--r-- 1 nobody 4294967294 194M Sep  9 01:12 /srv/imars-objects/gom/chlor_a_l3_pass/chlor_a_l3_pass_A2018251202000.hdf
7yl4r commented 6 years ago

Set up permissions for the user

imars-admin@imars-physalis:~$ namei -l /srv/imars-objects/gom/chlor_a_l3_pass/chlor_a_l3_pass_A2018251202000.hdf
f: /srv/imars-objects/gom/chlor_a_l3_pass/chlor_a_l3_pass_A2018251202000.hdf
drwxr-xr-x root   root         /
drwxr-xr-x root   root         srv
drwxr-xr-x root   root         imars-objects
drwxrwxr-x nobody imars-common gom
drwxrwxr-x nobody imars-common chlor_a_l3_pass
-rw-rw-r-- nobody imars-common chlor_a_l3_pass_A2018251202000.hdf

still getting same error.

7yl4r commented 6 years ago

Oh derp. Forgot about inside vs outside the docker container.

7yl4r commented 6 years ago

Getting closer... Now seeing this error in /erddapData/logs/log.txt:

datasets.xml error on line #384
While trying to load datasetID=chlor_a_l3_pass_db6d_37b4_723b (after 8193 ms)
java.lang.RuntimeException: datasets.xml error on or before line #384: Outer axis overlap between files.
max=1429.0 for /chlor_a_l3_pass/chlor_a_l3_pass_A2002185191000.hdf
is greater than
min=0.0 for /chlor_a_l3_pass/chlor_a_l3_pass_A2002186181500.hdf
 at gov.noaa.pfel.erddap.dataset.EDD.fromXml(EDD.java:429)
 at gov.noaa.pfel.erddap.LoadDatasets.run(LoadDatasets.java:336)
Caused by: java.lang.RuntimeException: Outer axis overlap between files.
max=1429.0 for /chlor_a_l3_pass/chlor_a_l3_pass_A2002185191000.hdf
is greater than
min=0.0 for /chlor_a_l3_pass/chlor_a_l3_pass_A2002186181500.hdf

 at gov.noaa.pfel.erddap.dataset.EDDGridFromFiles.updateValidateFileTable(EDDGridFromFiles.java:1158)
 at gov.noaa.pfel.erddap.dataset.EDDGridFromFiles.<init>(EDDGridFromFiles.java:1009)
 at gov.noaa.pfel.erddap.dataset.EDDGridFromNcLow.<init>(EDDGridFromNcLow.java:94)
 at gov.noaa.pfel.erddap.dataset.EDDGridFromNcFilesUnpacked.<init>(EDDGridFromNcFilesUnpacked.java:97)
 at gov.noaa.pfel.erddap.dataset.EDDGridFromFiles.fromXml(EDDGridFromFiles.java:272)
 at gov.noaa.pfel.erddap.dataset.EDD.fromXml(EDD.java:392)
 ... 1 more

I believe this is because the leftmost axis has not been explicitly set correctly during aggregation [ref]

7yl4r commented 6 years ago

Data now loaded in but lat/lon are not set up right. example graph

@dotis : I think we spoke on this before but can you summarize for me the difference between "longitude", "lon" and "latitude", "lat" in the modis L3 files? ERDDAP is picking up latitude and longitude as degrees in the range 0, 1429 and 0, 2089 respectively. That is obviously not right and you can see in the image below the tiny, confused land mask in the bottom left corner:

image

dotis commented 6 years ago

I think the "lat" and "lon" fields should be what we want here. However, I would say we might just generate one L3 for the whole Gulf each day by combining the pass files. I guess it all depends on who we are serving these to. Some people may want the individual passes, but I tend to just use the daily mosaic files. We can discuss.

On Thu, Sep 13, 2018 at 2:56 PM Tylar notifications@github.com wrote:

Data now loaded in but lat/lon are not set up right. example graph http://192.168.1.100:8080/erddap/griddap/chlor_a_l3_pass_db6d_37b4_723b.graph?chlor_a%5B(2018-09-01T20:10:00Z)%5D%5B(0.0):(1429.0)%5D%5B(0.0):(2089.0)%5D&.draw=surface&.vars=longitude%7Clatitude%7Cchlor_a&.colorBar=%7C%7C%7C%7C%7C&.land=over&.bgColor=0xffccccff

@dotis https://github.com/dotis : I think we spoke on this before but can you summarize for me the difference between "longitude", "lon" and "latitude", "lat" in the modis L3 files? ERDDAP is picking up latitude and longitude as degrees in the range 0, 1429 and 0, 2089 respectively. That is obviously not right and you can see in the image below the tiny land mask in the bottom left corner:

[image: image] https://user-images.githubusercontent.com/1051390/45509278-a82fb680-b786-11e8-94c4-6983cfadf4a6.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/marinebon/sdg14-shiny/issues/10#issuecomment-421114711, or mute the thread https://github.com/notifications/unsubscribe-auth/AZUtjd9a_5FP3iyb4aVpBK3KlDGHOH8Zks5uaqpVgaJpZM4RtTKz .

--

Dan Otis, PhDScientific ResearcherUniversity of South FloridaCollege of Marine ScienceInstitute for Marine Remote Sensing (IMaRS)140 7th Ave SouthSt. Petersburg, FL 33701

7yl4r commented 5 years ago

backend is all set