Open line-o opened 2 weeks ago
I doubt it's an issue with Ubuntu. Is the JDK Version and vendor exactly the same on both versions of Ubuntu?
We have this error since a couple of month which leads to running out of space
Yes, JDKs are the same.
@line-o Are these ubuntu instances running 'on bare metal' of in a virtualized (docker, VMware, ...) environment ?
During the Monday-eve session you provided quite some details/convincing details, and that actually with a nice code cleanup (combined with try-resources) addresses the issue nicely.
The java implementation of util:string-to-binary might be the issue here as it is opening a stream (UnsynchronizedByteArrayInputStream) without using the form usually used throughout the exist-db codebase (try( ... ){ })
I think this is unlikely as there is no file operation in the Java code snippet you presented. The code new UnsynchronizedByteArrayInputStream(str.getBytes(encoding))
simply reads from a byte array in-memory. In the worst case scenario - that memory will be released by the JVM GC when the UnsynchronizedByteArrayInputStream
is no longer referenced. Also see the note below about ownership and cleanup.
The implementation of binary value types (i.e. xs:base64Binary
and xs:hexBinary
) in eXist-db is very difficult as eXist-db does not have a Query Planner. The main concerns are three-fold fold and tightly inter-related:
BinaryValue
is recorded in the XQueryContext of the executing XQuery and should be released when the query finishes executing. I think there are a few early-release edge-cases implemented too for optimisation purposes, but these have proven notoriously difficult to get right as at execution time we often lack enough information to determine if a variable is going to be used further in the query.I would suggest placing a breakpoint in the TemporaryFileManager so that you can see the call to create the temporary file in your simple query above, you can then use the stack trace to work backward and see what part of your query is creating it and why, you should then also check why the XQueryContext cleanup method is not called (it very much should be) to release the resources when the query finishes.
Also one final note, any changes to Temporary Files need to be thoroughly tested on Windows. The reason we have the TemporaryFileManager predominantly was to work around a load of previously reported and fixed bugs in the way that Windows handles Temporary Files.
Describe the bug
On a server running exist-db 6.2.0 on Ubuntu 22.04 the monitoring reported a constant growing number of open file handles. On further inspection it became apparent that these were all temporary files created by exist-db. These files would only be purged from the filesystem when exist-db was restarted. The same query executed on the same version of eXist-db running on Ubuntu 20.04 did not show this behaviour.
The smallest reproducible XQuery triggering this behaviour we found is:
The above will create a temporary file in eXist-db's temporary file folder. The folder can be found by searching for a subfolder whose name starts with
exist-db-temp-file-manager-
followed by a number of digits. The parent folder can be queried withutil:system-property("java.io.tmpdir")
The java implementation of
util:string-to-binary
might be the issue here as it is opening a stream (UnsynchronizedByteArrayInputStream
) without using the form usually used throughout the exist-db codebase (try( ... ){ }
).Expected behavior
All temporary files associated with a query to be removed after the query has finished running.
To Reproduce
Leaking
Safe
and
Context
Additional context
conf.xml
? several