search

USF-IMARS / erddap-config

Content dir for docker-erddap incl setup.xml & dataset.xml
0 stars 1 forks source link

Load S3 images #8

Open sebastiandig opened 5 years ago

sebastiandig commented 5 years ago

@7yl4r We will probably need to adjust the S3 coordinates. Using the command: [sebastian@seashell S3_map]$ ncdump -h S3A_OL_1_EFR____20181003.SEN3.map I found 

netcdf S3A_OL_1_EFR____20181003.SEN3 {
dimensions:
        y = 2226 ;
        x = 2226 ;
double lat(y, x) ;
                lat:units = "degrees_north" ;
                lat:long_name = "latitude coordinate" ;
                lat:standard_name = "latitude" ;
double lon(y, x) ;
                lon:units = "degrees_east" ;
                lon:long_name = "longitude coordinate" ;
                lon:standard_name = "longitude" ;

It will probably produce something like the duplicates error from the chlor_a_I3_pass #1

7yl4r commented 5 years ago

Bummer. I don't see anything related in the .par or .xml files here.

Is there a trick to converting the matrix to vector, @dotis ?

dotis commented 5 years ago

Hmm. I tried creating the file and adding lat and lon manually, but got the same result. Basically, we need to take the lon matrix and take one row of it and make a vector, as all rows in the matrix are the same. This is true because the projection is equidistant. Likewise, we would take one column of the lat matrix and make a vector. It's possible I can do that in the gpt graph. The syntax used by gpt is C. Do you know the C syntax to grab one column or row of a matrix? In MATLAB, it would be lat(:,1) for the first column and lon(1,:) for the first row. I will keep trying this. Let me know if you have ideas.

On Wed, Jan 16, 2019 at 3:49 PM Tylar notifications@github.com wrote:

Bummer. I don't see anything related in the .par or .xml files here https://github.com/USF-IMARS/imars_dags/tree/master/dags/processing/s3_chloro_a .

Is there a trick to converting the matrix to vector, @dotis https://github.com/dotis ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/erddap-config/issues/8#issuecomment-454935308, or mute the thread https://github.com/notifications/unsubscribe-auth/AZUtjT0Cxx6fy_H-Sp_lGiXE3XEqiX7vks5vD5BngaJpZM4aD4VI .

--

Dan Otis, PhDScientific ResearcherUniversity of South FloridaCollege of Marine ScienceInstitute for Marine Remote Sensing (IMaRS)140 7th Ave SouthSt. Petersburg, FL 33701

dotis commented 5 years ago

Or, since lat and lon will be the same for all files in an roi, we could just manually load in the lat/lon for the roi if that's possible with ERDDAP.

On Wed, Jan 16, 2019 at 5:05 PM Daniel Otis dotis@mail.usf.edu wrote:

Hmm. I tried creating the file and adding lat and lon manually, but got the same result. Basically, we need to take the lon matrix and take one row of it and make a vector, as all rows in the matrix are the same. This is true because the projection is equidistant. Likewise, we would take one column of the lat matrix and make a vector. It's possible I can do that in the gpt graph. The syntax used by gpt is C. Do you know the C syntax to grab one column or row of a matrix? In MATLAB, it would be lat(:,1) for the first column and lon(1,:) for the first row. I will keep trying this. Let me know if you have ideas.

On Wed, Jan 16, 2019 at 3:49 PM Tylar notifications@github.com wrote:

Bummer. I don't see anything related in the .par or .xml files here https://github.com/USF-IMARS/imars_dags/tree/master/dags/processing/s3_chloro_a .

Is there a trick to converting the matrix to vector, @dotis https://github.com/dotis ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/erddap-config/issues/8#issuecomment-454935308, or mute the thread https://github.com/notifications/unsubscribe-auth/AZUtjT0Cxx6fy_H-Sp_lGiXE3XEqiX7vks5vD5BngaJpZM4aD4VI .

--

Dan Otis, PhDScientific ResearcherUniversity of South FloridaCollege of Marine ScienceInstitute for Marine Remote Sensing (IMaRS)140 7th Ave SouthSt. Petersburg, FL 33701

--

Dan Otis, PhDScientific ResearcherUniversity of South FloridaCollege of Marine ScienceInstitute for Marine Remote Sensing (IMaRS)140 7th Ave SouthSt. Petersburg, FL 33701

7yl4r commented 5 years ago

An untested python script to add vector lat/lon:

# based on https://stackoverflow.com/a/29268995/1483986
import netCDF4
import numpy as np

ncfile = netCDF4.('/path/to/your/ncfile.nc', 'w')
lat = ncfile.variables['lat'][:]
lon = ncfile.variables['lon'][:]
ncfile.createDimension('lat', lat[0, :])
ncfile.createDimension('lon', lon[:, 0])
ncfile.close()
dotis commented 5 years ago

Looks like that will work. Maybe we could tack this into airflow.

On Wed, Jan 16, 2019 at 5:30 PM Tylar notifications@github.com wrote:

An untested python script to add vector lat/lon:

based on https://stackoverflow.com/a/29268995/1483986import netCDF4import numpy as np

ncfile = netCDF4.('/path/to/your/ncfile.nc', 'w') lat = ncfile.variables['lat'][:] lon = ncfile.variables['lon'][:] ncfile.createDimension('lat', lat[0, :]) ncfile.createDimension('lon', lon[:, 0]) ncfile.close()

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/erddap-config/issues/8#issuecomment-454969200, or mute the thread https://github.com/notifications/unsubscribe-auth/AZUtjQlmRh2b6ntZm9VzXQUtYEUqHP7Qks5vD6gXgaJpZM4aD4VI .

--

Dan Otis, PhDScientific ResearcherUniversity of South FloridaCollege of Marine ScienceInstitute for Marine Remote Sensing (IMaRS)140 7th Ave SouthSt. Petersburg, FL 33701

7yl4r commented 5 years ago

Sebastian's thread from the ERDDAP forum has a lot of good information on this too.

In particular the suggestion to use THREDDS and this excerpt from Bob Simons:

the data file has latitude and longitude as 2D variables. But that is just the weird way the files were created, because the values (as you indicated) just reflect an underlying Equidistant Cylindrical projection. So a better long term solution is: Use NCML (https://coastwatch.pfeg.noaa.gov/erddap/download/setupDatasetsXml.html#NcML) to modify the files on-the-fly or NCO (https://coastwatch.pfeg.noaa.gov/erddap/download/setupDatasetsXml.html#NCO) to actually modify the files, so that the structure of the files appears to ERDDAP as

netcdf A2016001_2016007_7D_FGB_IOP {
dimensions:
        latitude = 770 ;
        longitude = 1100 ;
variables:
        double latitude(latitude);
        double longitude(longitude);
        double adg443_mean(latitude, longitude) ; 
        double adg443_anom(latitude, longitude) ;

I do something similar for a couple of datasets via ncml. I created a file with latitude and longitude dimensions and 1D variables. The header looks like:

dimensions:
        latitude = 770 ;
        longitude = 1100 ;
variables:
        double latitude(latitude);
        double longitude(longitude);

And then the .ncml says to use those new latitude and longitude dimensions and values for the data variables (in your case, adg443_mean and adg443_anom). (If you need help setting that up, let me know.)

So for me this boils down to a few open questions:

  1. Is there enough added value in hosting via THREDDS to justify resolving this that way?
    • if yes; problem solved.
  2. Is there any harm in converting the 2D lat/lon files to 1D lat/lon? Are there important programs that expect 2D lat/lon?
    • if no; we convert w/ NCO, python, matlab or whatever
    • if yes; we should use NCML

I don't know anything about (2), but for (1):

I am on the fence. IMaRS has an old THREDDS server running and linked on the website: http://imars.marine.usf.edu:8080/thredds/

As far as I know everything in there is broken and I haven't heard any complaints; I am not sure if anyone has ever used it and I don't know where it is hosted.