search

CUAHSI / HydroDesktop

HydroDesktop is a free and open source GIS enabled desktop application that helps you search for, download, visualize, and analyze hydrologic and climate data registered with the CUAHSI Hydrologic Information System. HydroDesktop is part of the legacy CUAHSI toolkit and is not under active development. Click the "releases" link below to download the latest installer for Microsoft Windows.
73 stars 20 forks source link

Add single thread option to search download settings #117

Open xhqiao89 opened 6 years ago

xhqiao89 commented 6 years ago

twhitvine[CodePlex]
In Search tab, Download Settings button, you can tell HD to limit each each GetValues download request to approximately n number of values. The intent behind this option is to allow data retrieval from servers with limits on request volumes.

In testing, some servers still fail. It appears to be because HD download is still multi-threaded even when number of values per request is set.

I suggesting adding a checkbox to the Download Options dialog to allow the user to choose from multi-threaded vs. single threaded download. Multi-threaded should be the default, as it is now.

xhqiao89 commented 6 years ago

Sonic87[CodePlex]
I tried to replicate the issue, but when I went to harvest the metadata the error Object reference not set to an instance of an object occurred.

xhqiao89 commented 6 years ago

Sonic87[CodePlex]
** Closed by Sonic87 11/29/2013 11:43AM

xhqiao89 commented 6 years ago

twhitvine[CodePlex]
It's related, but not the same. They both have to do with server limitations.

Some services restrict the number of values you can get per request so that the servers are not overwhelmed. USGS used to do this although I don't know if they still do. This is related to issue 8649.

Other servers can only handle one or a few simultaneous requests for data. HydroDesktop's multithreaded download counts as several simultaneous requests. So, to make sure you don't overwhelm the server, we added the option described in issue 8733.

In my testing, the single thread option is working and I think this issue should be closed. Issue 8649 should remain open until it is fixed.

xhqiao89 commented 6 years ago

smithry[CodePlex]
Is this the same issue as https://hydrodesktop.codeplex.com/workitem/8649? I attempted to test this, but could not harvest the metadata. Could be that my connection is too slow.

xhqiao89 commented 6 years ago

twhitvine[CodePlex]
I think that HydroDesktop may have been working correctly after all. I think the problem is that the service I was testing with reported the wrong number of total values in GetSiteInfo, and so HydroDesktop couldn't accurately compute the number of values per request. Ill mark this as resolved, because in my testing with other web services, it seems to work fine.

xhqiao89 commented 6 years ago

twhitvine[CodePlex]
Something still isn't quite right. Maybe the code needs to sleep for a 0.5 seconds per request?

Here are steps to test:

Search tab, Select Data Sources, Local Data Source, Add Data Source. http://hydro1.sci.gsfc.nasa.gov/daac-bin/his/1.0/merra_mst1nxmld.cgi?WSDL Harvest metadata. There are 40,000 sites and it’s a slow service, so I suggest you cancel after grabbing a few sites. Search a. Draw rectangle around the southern tip of Chile, or wherever your local cache has sites.

b. Keyword: Transpiration c. 1/1/2011 - 1/3/2012 d. Make sure you’re using a local data source. Download a time series.

Based upon how long each request should take with this slow service, I think the download manager performs only one download request. Though it says 100 values per request, it actually got 4319 values in this one request. This is 1/2011 through 6/2011, so it didn’t reach 1/2012.

I tested in HydroExcel and found that the service does indeed have values to 1/3/2012. If I break the large request up into small requests, for about 1500 values per request, I was able to get all the values.

xhqiao89 commented 6 years ago

twhitvine[CodePlex]
Cool. I look forward to the next release so I can test this.

xhqiao89 commented 6 years ago

ericch[CodePlex]
New checkbox added to Download Settings form. Allows user to toggle between single thread and multi-thread downloading. By default, the downloading is set to multi-thread. Performed multiple tests and seems to be working. Additionally, I was able to download all the values from the example site that Tim posted although it performed the download slowly. For now, I'll consider this issue resolved.

xhqiao89 commented 6 years ago

twhitvine[CodePlex]
http://hydro1.sci.gsfc.nasa.gov/daac-bin/his/1.0/merra_mat1nxlnd.cgi?WSDL

xhqiao89 commented 6 years ago

ericch[CodePlex]
Hi Tim,

Could you give some example sites that aren't allowing download and the number of values with which you are setting them at? Thank You.

xhqiao89 commented 6 years ago

twhitvine[CodePlex]
Another way of putting it is, give the option to download data in series rather than parallel.