search

delving / culture-hub

The Delving Search and Administrative Interface
Apache License 2.0
66 stars 5 forks source link

Downloading Dataset #591

Closed geralddejong closed 11 years ago

geralddejong commented 12 years ago

On the DiMCoN test server, it is no longer possible to do anything with the largest dataset, maritiem-digitaal. As errors were encountered, the dataset has been through the upload cycle a number of times, but it appears that something has gone wrong with the cleaning mechanism since there now appear to be 2,447,973 records, while the original dataset contains only some 800,000 records.

Download is now impossible:

2012-06-30 10:39:26,132 - [INFO] - controllers.SipCreatorEndPoint$: maritiem-digitaal: Prepared 670000 of 2447973 records for download

geralddejong commented 12 years ago

The SIP-Creator request canceled, client stopped, but the server toils further:

2012-06-30 10:46:53,851 - [INFO] - controllers.SipCreatorEndPoint$: maritiem-digitaal: Prepared 2300000 of 2447973 records for download

manuelbernhardt commented 12 years ago

Ok, so this is a problem for large collections. The download of a set is an asynchronous operation, in order not to hog the connection pool, which means that canceling the request on the client side won't have much effect, though perhaps there's a way to check whether the connection is still alife, I'll look into that.

manuelbernhardt commented 11 years ago

There really isn't a good way of fixing this, but given that this happens much faster now I'll close the issue.