rmendels / rerddapXtracto

xtractomactic using rerddap
Other
14 stars 4 forks source link

Error when using rxtracto to access ssh #6

Closed abfleishman closed 7 years ago

abfleishman commented 7 years ago

I am getting an error when using the rxtracto(). We have successfully been using it to access the hycom_GLBa008_tyx dataset (we have extracted data for >23,000 points), however, halfway through we began to receive the following error:

Error in object[-omit, , drop = FALSE] : incorrect number of dimensions

Running through the function line by line I have traced it back to lines 208-215 of the rxtracto() source code. It is returning an NA when running line 209: paramdata <- ncdf4::ncvar_get(ncFile, parameter)

When looking at the output of line 208, it is saying that the date range and variable is coming from experiment: 91.2, however, when I look online it looks like it should be coming from experiment: 91.1 according to the date 2015-12-11 18:58:34 in example below.
Any chance you have encountered this problem before? Thoughts?

library(rerddapXtracto)
library(rerddap)

# sample of data we are using
dataSub<-structure(
  list(
    Latitude = c(
      18.411898,
      18.412279,
      18.413588,
      18.4138915454545,
      18.4141950909091,
      18.4144986363636
    ),
    Longitude = c(
      -104.210121,
      -104.21064,
      -104.211845,
      -104.212580272727,
      -104.213315545455,
      -104.214050818182
    ),
    date = structure(
      c(
        1449860314,
        1449860254,
        1449860194,
        1449860134,
        1449860074,
        1449860014
      ),
      class = c("POSIXct",
                "POSIXt"),
      tzone = "UTC"
    ),
    lon_hycom = c(
      2270.8734875,
      2270.867,
      2270.8519375,
      2270.84274659091,
      2270.83355568182,
      2270.82436477273
    ),
    lat_hycom = c(
      1898.32844792124,
      1898.33590025503,
      1898.36150420497,
      1898.36744153318,
      1898.37337886139,
      1898.37931618959
    )
  ),
  .Names = c("Latitude",
             "Longitude", "date", "lon_hycom", "lat_hycom"),
  row.names = c(NA,
                -6L),
  class = "data.frame"
)

# input for rxtracto
xpos <- dataSub$lon_hycom
ypos <- dataSub$lat_hycom
tpos <- dataSub$date

sshInfo <- rerddap::info('hycom_GLBa008_tyx')

ssh <- rxtracto(dataInfo = sshInfo, parameter = 'ssh', xcoord = xpos, ycoord = ypos, tcoord = tpos,
                xlen = 0.1, ylen = 0.1,xName="X",yName = "Y", tName = "time",verbose = TRUE)

Error in object[-omit, , drop = FALSE] : incorrect number of dimensions

rmendels commented 7 years ago

Thanks for the feedback.

The reason I said you will be living dangerously is that there are so many possible variations when we go to any GRID file on any ERDDAP, that there is no way in the world I can possibly test everything. I have slowly been testing it on a variety of datasets, but there is a reason I warn people.

One possibility is the hycom went away for awhile from the ERDDAP, which means that all of a sudden you are reading non-existent data. The other is you may not be getting what you think because the function finds the "nearest" actual grid point, and there may be ties in the nearest.

In the example below, which is the value where it bombs? I will try to get into the code and reproduce the problem if you can narrow down the values that failed.

-Roy

PS - But getting 23,000 points ain't half bad. As I was saying to Lynn, what gets me is it is the same code just repeating as you feed in values.

On Jan 24, 2017, at 9:21 AM, Abram Fleishman notifications@github.com wrote:

I am getting an error when using the rxtracto(). We have successfully been using it to access the hycom_GLBa008_tyx dataset (we have extracted data for >23,000 points), however, halfway through we began to receive the following error:

Error in object[-omit, , drop = FALSE] : incorrect number of dimensions

Running through the function line by line I have traced it back to lines 208-215 of the rxtracto() source code. It is returning an NA when running line 209: paramdata <- ncdf4::ncvar_get(ncFile, parameter)

When looking at the output of line 208, it is saying that the date range and variable is coming from experiment: 91.2, however, when I look online it looks like it should be coming from experiment: 91.1 according to the date 2015-12-11 18:58:34 in example below. Any chance you have encountered this problem before? Thoughts?

library(rerddapXtracto) library(rerddap)

sample of data we are using

dataSub<-structure( list( Latitude = c( 18.411898, 18.412279, 18.413588, 18.4138915454545, 18.4141950909091, 18.4144986363636 ), Longitude = c( -104.210121, -104.21064, -104.211845, -104.212580272727, -104.213315545455, -104.214050818182 ), date = structure( c( 1449860314, 1449860254, 1449860194, 1449860134, 1449860074, 1449860014 ), class = c("POSIXct", "POSIXt"), tzone = "UTC" ), lon_hycom = c( 2270.8734875, 2270.867, 2270.8519375, 2270.84274659091, 2270.83355568182, 2270.82436477273 ), lat_hycom = c( 1898.32844792124, 1898.33590025503, 1898.36150420497, 1898.36744153318, 1898.37337886139, 1898.37931618959 ) ), .Names = c("Latitude", "Longitude", "date", "lon_hycom", "lat_hycom"), row.names = c(NA, -6L), class = "data.frame" )

input for rxtracto

xpos <- dataSub$lon_hycom ypos <- dataSub$lat_hycom tpos <- dataSub$date

sshInfo <- rerddap::info('hycom_GLBa008_tyx')

ssh <- rxtracto(dataInfo = sshInfo, parameter = 'ssh', xcoord = xpos, ycoord = ypos, tcoord = tpos, xlen = 0.1, ylen = 0.1,xName="X",yName = "Y", tName = "time",verbose = TRUE)

Error in object[-omit, , drop = FALSE] : incorrect number of dimensions

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.


"The contents of this message do not reflect any position of the U.S. Government or NOAA."


Roy Mendelssohn Supervisory Operations Research Analyst NOAA/NMFS Environmental Research Division Southwest Fisheries Science Center Note new street address 110 McAllister Way Santa Cruz, CA 95060 Phone: (831)-420-3666 Fax: (831) 420-3980 e-mail: Roy.Mendelssohn@noaa.gov www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill." "From those who have been given much, much will be expected" "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

abfleishman commented 7 years ago

Gotcha! Living dangerously is exciting and frustrating, but has been known to pay off!

It breaks on all/any of the values in the example. We are running it on tracking data from a bunch of seabirds. We are looping through each individual trip to extract the data in smaller batches. When we got to the birds from Isla Clarion, MX (the 6 rows of data that I have provided) it broke. Next group of birds from West Mexico also didn't work.

rmendels commented 7 years ago

Thanks. I do not know if I can get to this today. But I will put it in the debugger and see what is going on. Looking over the code quickly today, I don't know that I look at what error codes, if any, rerddap throws if the request is unsuccessful. I likely need to be more careful there.

Anyway, thanks for the example, hopefully I can look at it tomorrow and let you know what is happening.

-Roy

On Jan 24, 2017, at 11:17 AM, Abram Fleishman notifications@github.com wrote:

Gotcha! Living dangerously is exciting and frustrating, but has been known to pay off!

It breaks on all/any of the values in the example. We are running it on tracking data from a bunch of seabirds. We are looping through each individual trip to extract the data in smaller batches. When we got to the birds from Isla Clarion, MX (the 6 rows of data that I have provided) it broke. Next group of birds from West Mexico also didn't work.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.


"The contents of this message do not reflect any position of the U.S. Government or NOAA."


Roy Mendelssohn Supervisory Operations Research Analyst NOAA/NMFS Environmental Research Division Southwest Fisheries Science Center Note new street address 110 McAllister Way Santa Cruz, CA 95060 Phone: (831)-420-3666 Fax: (831) 420-3980 e-mail: Roy.Mendelssohn@noaa.gov www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill." "From those who have been given much, much will be expected" "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

abfleishman commented 7 years ago

Great! Thank you! And thanks for your great tools!

rmendels commented 7 years ago

Can you send me a quick sample of something that did work? That will help me compare the calls being generated.

-Roy

On Jan 24, 2017, at 11:36 AM, Abram Fleishman notifications@github.com wrote:

Great! Thank you! And thanks for your great tools!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.


"The contents of this message do not reflect any position of the U.S. Government or NOAA."


Roy Mendelssohn Supervisory Operations Research Analyst NOAA/NMFS Environmental Research Division Southwest Fisheries Science Center Note new street address 110 McAllister Way Santa Cruz, CA 95060 Phone: (831)-420-3666 Fax: (831) 420-3980 e-mail: Roy.Mendelssohn@noaa.gov www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill." "From those who have been given much, much will be expected" "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

abfleishman commented 7 years ago

Here is a snippit from a bird that worked.

# Code for bird that worked:

bird <-
    structure(
        list(
            Latitude = c(5.14531, 5.146339, 5.148022, 5.149625,
                         5.150056, 5.15209),
            Longitude = c(
                -162.2787,
                -162.2885,
                -162.2978,-162.2721,
                -162.30225,
                -162.3067
            ),
            date = structure(
                c(
                    1282776358,
                    1282776418,
                    1282776478,
                    1282776298,
                    1282776538,
                    1282776598
                ),
                class = c("POSIXct",
                          "POSIXt"),
                tzone = "UTC"
            ),
            lon_hycom = c(
                1545.01625,
                1544.89375,
                1544.7775,
                1545.09875,
                1544.721875,
                1544.66625
            ),
            lat_hycom = c(
                1638.83489935354,
                1638.85502652274,
                1638.88794588696,
                1638.91930045668,
                1638.92773078702,
                1638.96751568709
            )
        ),
        .Names = c("Latitude", "Longitude", "date",
                   "lon_hycom", "lat_hycom"),
        class = c("data.table", "data.frame"),
        row.names = c(NA,-6L),
        .internal.selfref = < pointer:0x0000000002680788 >
    )

xpos <- bird$lon_hycom
ypos <- bird$lat_hycom
tpos <- bird$date

sshbird <-
    rxtracto(
        dataInfo = sshInfo,
        parameter = 'ssh',
        xcoord = xpos,
        ycoord = ypos,
        tcoord = tpos,
        xlen = 0.1,
        ylen = 0.1,
        xName = "X",
        yName = "Y",
        tName = "time",
        verbose = TRUE
    )
rmendels commented 7 years ago

Thanks.

-Roy

On Jan 24, 2017, at 12:29 PM, Abram Fleishman notifications@github.com wrote:

Here is a snippit from a bird that worked.

Code for bird that worked:

bird <- structure( list( Latitude = c(5.14531, 5.146339, 5.148022, 5.149625, 5.150056, 5.15209), Longitude = c( -162.2787, -162.2885, -162.2978,-162.2721, -162.30225, -162.3067 ), date = structure( c( 1282776358, 1282776418, 1282776478, 1282776298, 1282776538, 1282776598 ), class = c("POSIXct", "POSIXt"), tzone = "UTC" ), lon_hycom = c( 1545.01625, 1544.89375, 1544.7775, 1545.09875, 1544.721875, 1544.66625 ), lat_hycom = c( 1638.83489935354, 1638.85502652274, 1638.88794588696, 1638.91930045668, 1638.92773078702, 1638.96751568709 ) ), .Names = c("Latitude", "Longitude", "date", "lon_hycom", "lat_hycom"), class = c("data.table", "data.frame"), row.names = c(NA,-6L), .internal.selfref = < pointer:0x0000000002680788 > )

xpos <- bird$lon_hycom ypos <- bird$lat_hycom tpos <- bird$date

sshbird <- rxtracto( dataInfo = sshInfo, parameter = 'ssh', xcoord = xpos, ycoord = ypos, tcoord = tpos, xlen = 0.1, ylen = 0.1, xName = "X", yName = "Y", tName = "time", verbose = TRUE )

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.


"The contents of this message do not reflect any position of the U.S. Government or NOAA."


Roy Mendelssohn Supervisory Operations Research Analyst NOAA/NMFS Environmental Research Division Southwest Fisheries Science Center Note new street address 110 McAllister Way Santa Cruz, CA 95060 Phone: (831)-420-3666 Fax: (831) 420-3980 e-mail: Roy.Mendelssohn@noaa.gov www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill." "From those who have been given much, much will be expected" "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

abfleishman commented 7 years ago

I think I have tracked down our issue! Our transformation of our data lon/lat is not working correctly... The points are ending up in the wrong grid cells. I hope you have not wasted too much time on this :(

Currently we are trying to transform our lon/lat positions to the Hycom grid system, but we are having trouble. If you happen to know how to do this in R we would love your input!

Alternatively, there are some Hycom datasets that have been transformed to lat/lon but they are not available through ERDDAP (eg 'GLBu0.08' http://tds.hycom.org/thredds/GLBu0.08/expt_91.2.html?dataset=GLBu0.08-expt_91.2-ssh). Is there a tool that would allow us to access these data sets? I apologize if you have wasted your time on this. Thank you for being so responsive!

rmendels commented 7 years ago

I hadn't started to look at this. It was on my to do for tomorrow morning. I will take a look at what Hycom is doing, and let you know.

As for the other dataset, are you certain it is not available, see:

http://upwell.pfeg.noaa.gov/erddap/search/index.html?page=1&itemsPerPage=1000&searchFor=GLBu0.08

If that is what you are looking for, that should work with the rerddapXtracto. If not, you seem to know a lot about R. It really isn't that hard to write a program to use the THREDDS server. You can do so using ncdf4, and you just have to take the time to work in array space rather than coordinate space. I can give you an example I believe.

-Roy

On Jan 24, 2017, at 8:44 PM, Abram Fleishman notifications@github.com wrote:

I think I have tracked down our issue! Our transformation of our data lon/lat is not working correctly... The points are ending up in the wrong grid cells. I hope you have not wasted too much time on this :(

Currently we are trying to transform our lon/lat positions to the Hycom grid system, but we are having trouble. If you happen to know how to do this in R we would love your input!

Alternatively, there are some Hycom datasets that have been transformed to lat/lon but they are not available through ERDDAP (eg 'GLBu0.08' http://tds.hycom.org/thredds/GLBu0.08/expt_91.2.html?dataset=GLBu0.08-expt_91.2-ssh). Is there a tool that would allow us to access these data sets? I apologize if you have wasted your time on this. Thank you for being so responsive!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.


"The contents of this message do not reflect any position of the U.S. Government or NOAA."


Roy Mendelssohn Supervisory Operations Research Analyst NOAA/NMFS Environmental Research Division Southwest Fisheries Science Center Note new street address 110 McAllister Way Santa Cruz, CA 95060 Phone: (831)-420-3666 Fax: (831) 420-3980 e-mail: Roy.Mendelssohn@noaa.gov www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill." "From those who have been given much, much will be expected" "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

abfleishman commented 7 years ago

Ah ha! Somehow when we first searched we didn't find that list of datasets! We are running rerddapXtracto::rxtracto() now with a subset of data and it seems like it is running without a problem! Thanks for directing us to the right list of data sources.

rmendels commented 7 years ago

You are welcome. This is good testing for the code. As I said, it is really difficult to completely test, because you have all this model output that are on non-standard grids, and that the code works with one doesn't mean it works with another.

Let me know if you find any problems.

-Roy

On Jan 24, 2017, at 9:49 PM, Abram Fleishman notifications@github.com wrote:

Ah ha! Somehow when we first searched we didn't find that list of datasets! We are running rerddapXtracto::rxtracto() now with a subset of data and it seems like it is running without a problem! Thanks for directing us to the right list of data sources.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.


"The contents of this message do not reflect any position of the U.S. Government or NOAA."


Roy Mendelssohn Supervisory Operations Research Analyst NOAA/NMFS Environmental Research Division Southwest Fisheries Science Center Note new street address 110 McAllister Way Santa Cruz, CA 95060 Phone: (831)-420-3666 Fax: (831) 420-3980 e-mail: Roy.Mendelssohn@noaa.gov www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill." "From those who have been given much, much will be expected" "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

rmendels commented 7 years ago

BTW - I just did a quick search. HYCOM uses a Mercator curvilinear grid, which I assume is not quite the same as a straight Mercator grid. I have done a search for that, and it seems to exist in the Hycom code, which I am not about to wade through, and also in the Python CDAT package, which I am also not going to wade through,

If it is something you really need, I can refer you to my colleague Rich Signell who should know these sorts of things.

-Roy

PS - I just found this. It is the only thing I could find

https://hycom.org/tools/300-hycomlonlat2ij

On Jan 24, 2017, at 9:49 PM, Abram Fleishman notifications@github.com wrote:

Ah ha! Somehow when we first searched we didn't find that list of datasets! We are running rerddapXtracto::rxtracto() now with a subset of data and it seems like it is running without a problem! Thanks for directing us to the right list of data sources.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.


"The contents of this message do not reflect any position of the U.S. Government or NOAA."


Roy Mendelssohn Supervisory Operations Research Analyst NOAA/NMFS Environmental Research Division Southwest Fisheries Science Center Note new street address 110 McAllister Way Santa Cruz, CA 95060 Phone: (831)-420-3666 Fax: (831) 420-3980 e-mail: Roy.Mendelssohn@noaa.gov www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill." "From those who have been given much, much will be expected" "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

abfleishman commented 7 years ago

Thanks! We had done a bunch of searching and waded through the code for a few different tools and scripts (python, matlab, and the tool that you found) but We could not get any of them to work. The dataset that you pointed us to last night seems to be working (and quite fast!!!). So it looks like we might be able to avoid having to transform our data! Again thank you for you thoughts and assistance and the packages that you have authored!

rmendels commented 7 years ago

BTW - can I close this issue?

-roy

On Jan 25, 2017, at 9:11 AM, Abram Fleishman notifications@github.com wrote:

Thanks! We had done a bunch of searching and waded through the code for a few different tools and scripts (python, matlab, and the tool that you found) but We could not get any of them to work. The dataset that you pointed us to last night seems to be working (and quite fast!!!). So it looks like we might be able to avoid having to transform our data! Again thank you for you thoughts and assistance and the packages that you have authored!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.


"The contents of this message do not reflect any position of the U.S. Government or NOAA."


Roy Mendelssohn Supervisory Operations Research Analyst NOAA/NMFS Environmental Research Division Southwest Fisheries Science Center Note new street address 110 McAllister Way Santa Cruz, CA 95060 Phone: (831)-420-3666 Fax: (831) 420-3980 e-mail: Roy.Mendelssohn@noaa.gov www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill." "From those who have been given much, much will be expected" "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

abfleishman commented 7 years ago

Yup