ChristianGruen
commented
1 year ago Thanks for the observation.
In Java, both strings and arrays, which are used to cache the client request, are limited to 2^31 entries. It could potentially be possible to use a list of arrays for caching larger requests, but that would be a substantial change to the client architecture, and it would easily trigger OutOfMemory exceptions somewhere later.
One way out could be to add functions for streaming input, as e.g. realized in BaseXClient.java and in other client implementations.
MarcinBasiak
Description of the Problem
The issue is 100% reproducible. I try to create the new database using the BaseX C API. The file size is greater than 2GB. If the file is less than 2GB the issue does not occur. https://github.com/BaseXdb/basex/tree/9/basex-api/src/main/c
xyz@cluster:~$ basexserver [warning] /usr/bin/basexserver: Unable to locate /usr/share/java/tagsoup.jar in /usr/share/java [warning] /usr/bin/basexserver: Unable to locate /usr/share/java/xml-resolver.jar in /usr/share/java [warning] /usr/bin/basexserver: Unable to locate /usr/share/java/jing.jar in /usr/share/java BaseX 9.0.1 [Server] Server was started (port: 1984). Exception in thread "Thread-4" java.lang.ArrayIndexOutOfBoundsException: Maximum array size reached. at org.basex.util.Array.newSize(Array.java:238) at org.basex.util.list.ElementList.newSize(ElementList.java:27) at org.basex.util.list.ByteList.add(ByteList.java:42) at org.basex.io.in.BufferInput.readBytes(BufferInput.java:137) at org.basex.server.ClientListener.run(ClientListener.java:105)
basex/basex-core/src/main/java/org/basex/util/Array.java
* Maximum array size (see {@code MAX_ARRAY_SIZE} variable in {@link ArrayList}). / public static final int MAX_SIZE = Integer.MAX_VALUE - 8;
/**
@return new capacity / public static int newCapacity(final int size, final double factor) { return (int) Math.min(MAX_SIZE, factor checkCapacity(size + 1)); }
/**
@return argument as integer, or {@code 0} if the argument is negative */ public static int checkCapacity(final long size) { if(size > MAX_SIZE) throw new ArrayIndexOutOfBoundsException("Maximum array size reached."); return Math.max(0, (int) size); }
Integer.MAX_VALUE = 2147483647 -> 2147483647 Byte is equal to 2.0 Gigabyte.
Client Code: https://github.com/MarcinBasiak/BaseX/blob/e080b687e9b944f41c526dc32acead732f451c84/example.cpp
The issue has not have occurred if I used: https://docs.basex.org/wiki/Commands#CREATE_DB CREATE DB [name] ([input]) where input is the filename.
Question: Is it expected behavior? Does it mean that the BaseX will not support larger buffers than 2GB? How can I solve this issue? Should I modify the java code?
Expected Behavior
The C API should provide the same behavior as the command line interface. I don't want to save the received XML files to the disk because it's not optimal.
Steps to Reproduce the Behavior
Do you have an idea how to solve the issue?
public final class Array { /* Maximum array size (see {@code MAX_ARRAY_SIZE} variable in {@link ArrayList}). / public static final long MAX_SIZE = 3826003525L;
/**
@return new capacity / public static long newCapacity(final long size, final double factor) { return (long) Math.min(MAX_SIZE, factor checkCapacity(size + 1)); }
/**
@return argument as integer, or {@code 0} if the argument is negative */ public static long checkCapacity(final long size) { if(size > MAX_SIZE) throw new ArrayIndexOutOfBoundsException("Maximum array size reached."); return Math.max(0, size); }
The MAX_SIZE type can be changed to long for example. Additionally, this param should be possible to manage by config file?
What is your configuration?
Distributor ID: Ubuntu Description: Ubuntu 20.04.5 LTS Release: 20.04 Codename: focal
Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 32 On-line CPU(s) list: 0-31 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 32 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz Stepping: 4 CPU MHz: 2095.076 BogoMIPS: 4190.15 Hypervisor vendor: KVM Virtualization type: full L1d cache: 1 MiB L1i cache: 1 MiB L2 cache: 128 MiB L3 cache: 512 MiB NUMA node0 CPU(s): 0-31