search

gmantele / vollt

Java libraries implementing the IVOA protocol: ADQL, UWS and TAP
http://cdsportal.u-strasbg.fr/taptuto/
29 stars 28 forks source link

Retrieving results > 1GB? #109

Closed vforchi closed 5 years ago

vforchi commented 5 years ago

Hi, We are experiencing issues when trying to retrieve results larger than 1GB. Before I try to understand if it's due to our DB driver(s), could you please confirm that this works on other DB servers. Cheers

gmantele commented 5 years ago

I will try this afternoon or tomorrow. But in the meantime, could you describe what kind of issue you have?

vforchi commented 5 years ago

Sorry, I forgot to describe the problem: it hangs after having downloaded 1GB

gmantele commented 5 years ago

I just tried on my local machine with a result of 2.2GB: no problem for me. I tried with a slightly different version of the library (without the few last commits...especially the one about Content-Length) and I also tried through Tomcat directly (port 8080) or through Apache (80) ; still no problem.

I do no have any special configuration in Tomcat (tested with 7 and 9) or Apache-2. For both, I have the default configuration.

So, maybe an issue with your hard drives, but most likely with your Web Server (Apache? Nginx?) or your firewall. Sorry that I can not help more :( Anyway, whenever you find out, I would be curious to know the origin of this problem.

vforchi commented 5 years ago

I tried to go directly to Tomcat, but I still get the issue. Also, this has nothing to do with the DB, because the file is already on disk, so I am a bit puzzled. I will try to upgrade to the latest version of taplib.

gmantele commented 5 years ago

I will try to upgrade to the latest version of taplib.

Not sure it will resolve this issue, but this is an easy/quick test to do, at least.

vforchi commented 5 years ago

The latest version did not fix the issue. I tried to debug it, and I saw an OutOfMemory heap space error. I monitored it with jvisualvm and I saw a lot of byte[] being allocated, until the JVM runs out of memory: can you check if you have a similar memory usage? I am using the library with tomcat in a Spring Boot application, but I don't think this has something to do with the problem.

vforchi commented 5 years ago

The problem was in a filter we added to the request, so not related to this project.

gmantele commented 5 years ago

Ok. Thanks for the update :-)