search

ERDDAP / erddap

ERDDAP is a scientific data server that gives users a simple, consistent way to download subsets of gridded and tabular scientific datasets in common file formats and make graphs and maps. ERDDAP is a Free and Open Source (Apache and Apache-like) Java Servlet from NOAA NMFS SWFSC Environmental Research Division (ERD).
Creative Commons Zero v1.0 Universal
84 stars 59 forks source link

Values of some variables set to zero/near zero. #221

Open imcslatte opened 2 weeks ago

imcslatte commented 2 weeks ago

Describe the bug The variables of some variables in my ERDDAP dataset are set to zero/ near zero for requests of more than 1e7 rows.

To Reproduce

I ran into some unexpected behavior for a dataset we have on our ERDDAP server.

https://tds.marine.rutgers.edu/erddap/tabledap/CORA_Profiles_ECCOFSgrid.html

If I set the parameters to extract data from 2019/06/01 to 2019/10/01:

https://tds.marine.rutgers.edu/erddap/tabledap/CORA_Profiles_ECCOFSgrid.nc?time%2Clatitude%2Clongitude%2Cdepth%2Cprovenance%2CI%2CJ&time%3E=2019-06-01T00%3A00%3A00Z&time%3C=2019-10-01T23%3A54%3A00Z

The values in the variable provenance (range 800:809) after row 1e7 in the file are set to zero or near zero. The same goes for I and J. The other variables are fine.

If I switch to the time period 2019/09/15 to 2019/10/01:

https://tds.marine.rutgers.edu/erddap/tabledap/CORA_Profiles_ECCOFSgrid.nc?time%2Clatitude%2Clongitude%2Cdepth%2Cprovenance%2CI%2CJ&time%3E=2019-09-15T00%3A00%3A00Z&time%3C=2019-10-01T23%3A54%3A00Z

The values that were set to zero previously. are now within the expected range.

In addition, I checked the source files, and there are no values outside the expected range. The values also look fine if I pull up a time series of the entire dataset as an ERDDAP graph.

Expected behavior I expect all the values in the data set to be returned for a given query.

Server https://tds.marine.rutgers.edu/erddap erddap version 2.24

Desktop (please complete the following information):

Additional context

Datasets XML attached dataset_xml.txt Example data files.

CORA_SUBSET_ECCOFS_PR_20190120.zip

ChrisJohnNOAA commented 1 day ago

@imcslatte I'm trying to reproduce the problem locally, but unable to so far. Would you be able to share some additional files from the dataset?

ChrisJohnNOAA commented 1 day ago

Do you see issues with all export formats, or specifically with nc?

imcslatte commented 1 day ago

Hi Chris,

I saw issues with NetCDF and mat files, but I didn't try any others.

There's a year of data at:

https://marine.rutgers.edu/~hunter/stuff/testdata.tar.gz

Let me know if you have any problems accessing. Where you able to reproduce the problem from our server?

Thanks for looking into it.

Eli

From: Chris John @.> Sent: Thursday, November 14, 2024 2:00 PM To: ERDDAP/erddap @.> Cc: Elias Hunter @.>; Mention @.> Subject: Re: [ERDDAP/erddap] Values of some variables set to zero/near zero. (Issue #221)

Do you see issues with all export formats, or specifically with nc?

— Reply to this email directly, view it on GitHubhttps://github.com/ERDDAP/erddap/issues/221#issuecomment-2477192377, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABIXPAAKZWZCUO72INY3H632ATXMHAVCNFSM6AAAAABQYBGOVGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZXGE4TEMZXG4. You are receiving this because you were mentioned.Message ID: @.***>

ChrisJohnNOAA commented 1 day ago

Thanks for the files. I'll give that a try. I tried downloading a csv from your sever which as far as I could tell had all of the proper data. I did not try an nc file. If I can't reproduce locally with the new files I'll try your sever again with other file formats.