CRITFC / Helpdesk

Parent repository for tribal CDMS documentation and issues
1 stars 0 forks source link

Unable to query large quantities of data #63

Open tylerstright opened 2 years ago

tylerstright commented 2 years ago

There appears to be a limit on the amount of data CDMS will return. At this point I'm not sure if it's a data size limit or a record limit.

While testing our new Water Temperature dataset, I attempted to query out all data for a project/dataset. To do this I enter the Query tool, add no parameters and launch the query. In the first project, there are roughly 470,000 records and the data returned as expected. In the second project, there are ~1.4M records and no data is returned - the busy spinner keeps spinning, no error is returned, and the only way to register the error is through the F12/DevTools console.

I encountered the same error by querying all the Water Temperature data using an NPT-specific function that we use to access data using R instead of CDMS. This R function builds a URL that returns data. When I enter that URL into the browser, it returns the same error as through the query tool.

image

My investigation into this seems to point to .NET, ASP.NET, and/or ISS. As I suggested before, it seems like there is some limit placed on the size (data or record count) of exports/queries that CDMS will allow.

Helpful link: https://stackoverflow.com/questions/22289200/asp-net-overflow-or-underflow-in-the-arithmetic-operation-when-returning-large-f