IGUANA cannot handle queries larger than ~2GB

dice-group / IGUANA

IGUANA is a benchmark execution framework for RDF triple stores and quad stores

http://iguana-benchmark.eu

GNU Affero General Public License v3.0

24 stars 15 forks source link

IGUANA cannot handle queries larger than ~2GB #176

Closed liss-h closed 2 days ago

liss-h commented 1 year ago

Java arrays and probably some other things are for some reason limited to roughtly Integer.MAX_VALUE (= 2^32 / 2) bytes which is just over 2GB. Which is problematic when the query file is larger than that (which can happen in DELETE DATA or INSERT DATA queries) as IGUANA will just crash with an out of memory error.

I don't know if that is fixable

bigerl commented 1 year ago

Larger queries should always be streamed from disk and not reside at RAM. Configurable threshold.
Supporting larger queries requires Array with long size type (or streams?)
two modes:
- chunked HTTP without content-length
- regular HTTP request

bigerl commented 1 month ago

This is almost solved. The only remaining issue is:

[ ] Right now, we can only hash queries up to 2GB because of the hashing scheme used. We should use a streaming XXH3 instead