Ferret able to read data from ERDDAP

[karlmsmith]

Reported by steven.c.hankin on 1 Apr 2014 20:32 UTC Between the enhanced support for DSG files that has recently been added into Ferret; and the ability of ERDDAP to deliver DSG files, we are remarkably close to full ERDDAP support. How about this:

o new qualifier -- SET DATA/ERDDAP http://URL o create a temporary file with name derived from the URL o deal with special cases: already opened (just SET DATA); same-named file already exists (clobber it) o open that file o when CANCEL DATA is issued, delete the file

Migrated-From: http://dunkel.pmel.noaa.gov/trac/ferret/ticket/2163

[karlmsmith]

Comment by steven.c.hankin on 1 Apr 2014 21:39 UTC In case it may be relevant, here is easy c code that can stuff the contents of a URL into a file. Uses libcurl. From http://stackoverflow.com/questions/1636333/download-file-using-libcurl-in-c-c

#define CURL_STATICLIB
#include <stdio.h>
#include <curl/curl.h>
#include <curl/types.h>
#include <curl/easy.h>
#include <string>

size_t write_data(void *ptr, size_t size, size_t nmemb, FILE *stream) {
    size_t written;
    written = fwrite(ptr, size, nmemb, stream);
    return written;
}

int main(void) {
    CURL *curl;
    FILE *fp;
    CURLcode res;
    char *url = "http://localhost/aaa.txt";
    char outfilename[FILENAME_MAX] = "C:\\bbb.txt";
    curl = curl_easy_init();
    if (curl) {
        fp = fopen(outfilename,"wb");
        curl_easy_setopt(curl, CURLOPT_URL, url);
        curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
        curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
        res = curl_easy_perform(curl);
        curl_easy_cleanup(curl);
        fclose(fp);
    }
    return 0;
}

[karlmsmith]

Comment by @AnsleyManke on 2 Apr 2014 17:29 UTC We can already open erddap url's directly in Ferret. There's an email thread, 6/12/2013,

state of netCDF "translation"

The discussion was never documented in trac or in the Ferret docs.

We can open a url with a list of variables and constraints, for example:

use "http://coastwatch.pfeg.noaa.gov/erddap/tabledap/erdCinpKfmT?temperature,longitude,latitude,time&longitude>=-120.&longitude<=-118.4&latitude>=33&latitude<=34&time>=2007-01-01&time<=2007-02-01"

yes? sh dat
     currently SET data sets:
    1> http://coastwatch.pfeg.noaa.gov/erddap/tabledap/erdCinpKfmT?temperature,longitude,latitude,time&longitude>=-120.&longitude<=-118.4&latitude>=33&latitude<=34&time>=2007-01-01&time<=2007-02-01  (default)
 name     title                             I         J         K         L         M         N
 S.TEMPERATURE
          Sea Water Temperature            1:11160   ...       ...       ...       ...       ...
 S.LONGITUDE
          Longitude                        1:11160   ...       ...       ...       ...       ...
 S.LATITUDE
          Latitude                         1:11160   ...       ...       ...       ...       ...
 S.TIME   Time                             1:11160   ...       ...       ...       ...       ...

But the result is not organized as a DSG file; the Station, Longitude and Latitude variables are all just on the n-observations axis. We could think of a Ferret enhancement to define "id" variables from a subset of input variables.

In that email thread is also the suggestion to allow for variable names such as S.TEMPERATURE, as follows: Regard a dot as a legal character in a Ferret dataset variable name. Doing so would mean checking for a variable name match in the current dataset before processing the dot as an attribute indicator. If no match, then continue the attribute interpretation. Essentially Ferret would be "smart" in recognizing when a dot had one meaning or another.

The suggestion in the current ticket to download a dsg file from the erddap may be the best way to handle these datasets as DSG files.

[karlmsmith]

Comment by @AnsleyManke on 12 May 2014 20:41 UTC (remove comment -- made a note in the wrong ticket)

[karlmsmith]

Modified by @AnsleyManke on 12 May 2014 20:45 UTC

[karlmsmith]

Comment by @AnsleyManke on 13 May 2014 17:52 UTC The code in comment 1 works like a charm. I've got it hard-wired into a test version of Ferret, but have not yet put in Ferret qualifiers etc at this point.

I've found that to download all the variables in a dataset from erddap, we can specify just the constraints in the url -- there's no need to list the variables.

So this url as generated in the web interface:

http://dunkel.pmel.noaa.gov:8660/erddap/tabledap/dsg_files_badval_7f9a_0653_3fc1.ncCF?traj1,QC_flag,cruise_expocode,cruise_id,cruise_name,vessel_name,PIs,DOI,obs1,fCO2_recomputed,pressure,pressure_ncep_slp,pressure_equilibrium,salinity,salinity_woa5,temperature,temperature_equilibrium,longitude,latitude,depth,ETOPO2_depth,time,day_of_year,WOCE_flag,fCO2_source,region_id,data_id,tmonth,lon360&time>=2007-12-01T00:00:00Z&time<=2008-01-01T00:00:00Z

can be written as:

http://dunkel.pmel.noaa.gov:8660/erddap/tabledap/dsg_files_badval_7f9a_0653_3fc1.ncCF?&time>=2007-12-01T00:00:00Z&time<=2008-01-01T00:00:00Z

Or, adding some XY constraints:

http://dunkel.pmel.noaa.gov:8660/erddap/tabledap/dsg_files_badval_7f9a_0653_3fc1.ncCF?&longitude>=0&longitude<=100&latitude>=-90&latitude<=0&time>=2007-12-01T00:00:00Z&time<=2008-01-01T00:00:00Z