search

digitalfabrik / entitlementcard

App for 'Digitale Berechtigungskarten', generally benefit card for volunteers or socially vulnerable groups in Germany. App for Android & iOS + Backend + Administration Web Portal – 100% Open Source.
MIT License
36 stars 3 forks source link

Memory leaks backend #1544

Closed f1sh1918 closed 1 month ago

f1sh1918 commented 1 month ago

Describe the bug Since the release 2024.5.1 we are facing memory leaks on production. https://github.com/digitalfabrik/entitlementcard/releases/tag/2024.5.1-all Rollout of the update on production: 2024-06-07 - 11am It may be related to the javalin update, but its not clear what causes this issue Currently the backend service will be restarted (scheduled) every 12h as a workaround.

To Reproduce Steps to reproduce the behavior:

  1. Go to grafana and check increasing memory usage
  2. Also check the server logs if you can find something

Expected behavior Memory usage is not constantly increasing

Screenshots

image

Additional context We already tried to fix it by a possible issue with jetty (used by javalin) but it seems like it didn't fix the issue https://github.com/digitalfabrik/entitlementcard/pull/1510

Check directly grafana here: https://grafana.tuerantuer.org/d/l3Gr9b5Vk/server-vitals-with-host-selection?orgId=1&refresh=5m&var-HOST=entitlementcard.tuerantuer.org&viewPanel=8&from=now-24h&to=now

maxammann commented 1 month ago

I could only find this jetty issue https://github.com/jetty/jetty.project/pull/11782

Is that the one you mean?

maxammann commented 1 month ago

I plan on using this profiler: https://www.yourkit.com/docs/java-profiler/latest/help/remote_profiling.jsp