siepkes
commented
11 months ago We (@GeertZondervan and I) did some additional troubleshooting and thought it might be helpful to share our findings here.
Overall I'm guessing by the use of LockOptions.SKIP_LOCKED the intention is to create a simple queue implementation in PostgreSQL (something similar as described here)?
The first thing I noticed is that the queue implementation will always retrieve all jobs. When implementing a simple queue with PostgreSQL's SKIP LOCKED usually you have a worker retrieve (and lock) a single job (i.e. row). Using SKIP LOCKED in your SQL query ensures that a worker will always get the next available job which isn't being worked on. However the current implementation retrieves all jobs and puts them in a local queue in memory and then starts working on that.
The main reason why a high volume of jobs crashed the system (by running out of heap space) was because Hibernate was retrieving far too much fields (i.e. not just a job ID and job status) from the database and then tried to stuff everything into a resultset. This is the point where the application goes out of heap space.
We thought we could easily fix this by having Hibernate detach only the things it needed. While this did fix the issue with going out of heap space it brought to light another issue. The application now locks-up after boot. I think this is because the queue execution is manually triggered here on application startup:
This blocks the thread in Spring boots which fires events (I don't think this thread is suitable for doing such long running work). This also has another side-effect: The job scheduler doesn't know about it. Meaning in 60 seconds the @Scheduled(fixedRate = 60000) of BatchJobChunkExecutionQueue.kt will kick in even if the manual invocation is still running. Using something like @Scheduled(fixedRate = 60000, initialDelay = 0) might be more appropriate.
JanCizmar
cyyynthia
GeertZondervan
Tolgee runs out of heap space when importing a couple of thousand translation keys. I think this is the same issue as described in #1930.
I think this is caused by Tolgee loading it's entire job queue in to memory. This includes entries it is not currently processing (i.e. the ones which are queued). I haven't investigated further yet but for some reason these job queue entries are fairly large in memory, 1MB+. This means when importing a translation file with for example 3000 entries Tolgee will run out of heap space even when running with a 2GB heap. Once the job queue is saturated Tulgee will never recover from it without manual intervention because on every reboot it will load the job queue and go out of heap space.
During testing we did not have these issues with the Docker image. I have a hunch this is related to the machine translation (which is not enabled in the Docker image). But I haven't investigated this yet.
After making a heap dump and importing it in the Eclipse Memory analyzer one can see the heap consists almost solely out of PostgreSQL tuples:
This is the stacktrace associated with these objects:
This is what the job queue looks like:
This originates from the Tolgee job queue code:
https://github.com/tolgee/tolgee-platform/blob/fa9d12497a0cba57b70cc781ee68cfd6f70cc9f1/backend/data/src/main/kotlin/io/tolgee/batch/BatchJobChunkExecutionQueue.kt#L48-L68
A possible fix would be to have Tolgee retrieve batches (of for example 10 or 20 entries) instead of retrieving the entire queue.