[Bug]: Memory/CPU leaks until it gets killed the kernel

TheFrenchGhosty commented 5 months ago

What happened?

Hello,

I recently moved to your fork and everything works... except that it has now memory leaked twice, until it was crashed by the kernel.

It happened again today.

This doesn't seem related to https://github.com/kagemomiji/airsonic-advanced/issues/218 since the podcast task isn't ran when it crashes.

My Setup:

Airsonic is setup to restart daily so that it doesn't builds up too much cached RAM, however, it crashes randomly even with less than 24 hours of uptime.
The media are on a rclone mount, those rclone mount have rate limit but this never caused an issues (other than missing media) on Airsonic or the original Airsonic-Advanced
There's roughly 200 000 files

Docker: ghcr.io/kagemomiji/airsonic-advanced latest ef09d7464590 2 months ago

Version

11.1.x (Edge)

Version Detail

11.1.3-SNAPSHOT.20231108102506 – November 8, 2023 at 11:25:06 AM CET [Commit: 03d4cf154f1bf5caa6af4806858f93142b471816]

Operating System

Linux / Docker

Java Version

The one from the image

Database

Other

DB Detail

Integrated one

Configuration paramter

# Airsonic preferences.  NOTE: This file is automatically generated. Do not modify while application is running

EncryptionKeyPassword=<REDACTED>
EncryptionKeySalt=<REDACTED>
SettingsChanged=<REDACTED>
JWTKey=<REDACTED>
IndexCreationInterval=2
IndexCreationHour=3
FastCacheEnabled=true
OrganizeByFolderStructure=true
ExcludePattern=
IgnoreSymLinks=false
IndexString=A B C D E F G H I J K L M N O P Q R S T U V W X-Z(XYZ)
IgnoredArticles=The El La Los Las Le Les
Shortcuts=New Incoming Podcast
PlaylistFolder=/Playlists
MusicFileTypes=mp3 ogg oga aac m4a m4b flac wav wma aif aiff ape mpc shn mka opus
VideoFileTypes=flv avi mpg mpeg mp4 m4v mkv mov wmv ogv divx m2ts webm
CoverArtFileTypes2=cover.jpg cover.png cover.gif folder.jpg jpg jpeg gif png
CoverArtSource=FILETAG
CoverArtConcurrency=4
CoverArtQuality=90
SortAlbumsByYear=true
GettingStartedEnabled=false
WelcomeTitle=Welcome to Airsonic!
WelcomeSubtitle=
WelcomeMessage2=
LoginMessage=
Theme=black
LocaleLanguage=en
LocaleCountry=
LocaleVariant=
PodcastUpdateInterval=24
PodcastEpisodeRetentionCount=10
PodcastEpisodeDownloadCount=1
PodcastFolder=/Podcasts
spring.liquibase.parameters.defaultPodcastFolder=/Podcasts
UploadsFolder=%{['USER_MUSIC_FOLDERS'][0]}/Incoming
DownsamplingCommand4=ffmpeg -i %s -map 0:0 -b:a %bk -v 0 -f mp3 -
HlsCommand4=ffmpeg -ss %o -i %s -s %wx%h -async 1 -c:v libx264 -flags +cgop -b:v %vk -maxrate %bk -preset superfast -copyts -b:a %rk -bufsize 256k -map 0:0 -map 0:%i -ac 2 -ar 44100 -v 0 -threads 0 -force_key_frames expr:gte(t,n_forced*10) -start_number %j -hls_time %d -hls_list_size 0 -hls_segment_filename %n %p
JukeboxCommand2=ffmpeg -ss %o -i %s -map 0:0 -v 0 -ar 44100 -ac 2 -f s16be -
VideoImageCommand=ffmpeg -r 1 -ss %o -t 1 -i %s -s %wx%h -v 0 -f mjpeg -
SubtitlesExtractionCommand=ffmpeg -i %s -map 0:%i -f %f -
TranscodeEstimateTimePadding=2000
TranscodeEstimateBytePadding=0
FullScan=false
ClearFullScanSettingAfterScan=false
EnableCueIndexing=true
HideIndexedFiles=true

Environment variable of the container:

    environment:
      - 'JAVA_OPTS=-Xmx6G'
      - 'TZ=Europe/Paris'
      - 'PUID=0'
      - 'PGID=0'

(6GB of ram allocated because of the massive collection - note that it might be the problem since there's only ~5GB available after all the software use their ram, but it's weird because it didn't cause a problem on the original Airsonic-Advanced on this same setup)

Relevant log output

Kernel logs:

[5646440.889753] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=docker-72b09089623dc6d3a89779dafead8e738198d9605c16414da674b78a06da9cd3.scope,mems_allowed=0,global_oom,task_memcg=/system.slice/docker-70ccf5be4e13c591a75a179f096a6d233b9a8df5cc74c741956d9e4439388206.scope,task=java,pid=1155538,uid=0
[5646440.894790] Out of memory: Killed process 1155538 (java) total-vm:12463012kB, anon-rss:5366880kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:11528kB oom_score_adj:0

Airsonic logs doesn't contain anything relevant, the last entry is the cache logger, here you can see the last line before I restarted it:

2024-01-15 17:10:48.299  INFO --- o.a.p.s.CacheConfiguration$CacheLogger   : Cache Key: 4 | EventType: CREATED | Old value: null | New value: org.airsonic.player.domain.Playlist@606c0aef
2024-01-15 23:44:47.028  INFO --- org.airsonic.player.Application          : Starting Application using Java 17.0.5 on 70ccf5be4e13 with PID 1 (/app/WEB-INF/classes started by root in /var)

kagemomiji commented 5 months ago

@TheFrenchGhosty I have implemented measures to reduce memory usage in #337 . Could you please check if the issue has been resolved in the latest edge-release?

TheFrenchGhosty commented 5 months ago

I actually haven't any problem in like a week... even with the same image version as before:

ghcr.io/kagemomiji/airsonic-advanced              latest         ef09d7464590   2 months ago    824MB

I just updated, I guess we'll see if things keep being fine.

bartmans commented 5 months ago

Been running 11.1.3-SNAPSHOT.20240124131128 – 24 januari 2024 om 14:11:28 CET [Commit: c1f65da643bae1e61b45e3113dae253692f8cf22] for a couple of days.

This is the most stable and speedy version in a while. Response is quick, indexing and scanning work as intended. Cover art is added.

Nice work!

kagemomiji commented 5 months ago

I consider this bug resolved by #337. If there are any issues, please Reopen it and I will address them.

TheFrenchGhosty commented 5 months ago

It hasn't leaked since my last update... so yeah, I guess it's solved.

kagemomiji / airsonic-advanced