search

ptr727 / PlexCleaner

Utility to optimize media files for Direct Play in Plex, Emby, Jellyfin
MIT License
247 stars 12 forks source link

High Memory Usage During Media Processing #363

Closed billyors closed 2 weeks ago

billyors commented 1 month ago

Is this a new issue that can be reproduced using the latest version?

Which operating systems reproduce the issue?

Version information.

OS Version: Ubuntu 24.04 Docker Version: 26.1.1, build 4cf5afa Docker Image: latest PlexCleaner getversioninfo: 3.8.19.46621 (Release)

Steps to reproduce?

I'm a little out of my element looking at this code but I think there is a potential memory leak during processing. I think the issue is related to theIsMediaCurrent code.

Steps to reproduce the behavior:

  1. Run PlexCleaner on a directory with large media files.
  2. Have file that triggers Sidecar LastWriteTimeUtc out of sync with media file and Sidecar SHA256 out of sync with media file.
  3. Monitor memory usage during the processing.
  4. Observe a gradual increase in memory usage, leading to high memory consumption.

Expected: Memory usage should remain stable and not increase excessively during processing.

Commandline.

docker run \
    --rm \
    --name PlexCleaner \
    --user "1024:100" \
    --group-add 997 \
    --env TZ=America/Chicago \
    --cpus=8 \
    --memory 20g \
    --memory-reservation 18g \
    --memory-swap 24g \
    --volume /mnt/nas:/mnt/nas:rw \
    --volume /plex-media:/plex-media:rw \
    --volume /mnt/archivedisk:/mnt/archivedisk:rw \
    docker.io/ptr727/plexcleaner:latest \
    /bin/bash -c "/PlexCleaner/PlexCleaner defaultsettings --settingsfile /PlexCleaner/PlexCleaner.json && \
               sed -i 's|\"DeleteEmptyFolders\": true|\"DeleteEmptyFolders\": false|g' /PlexCleaner/PlexCleaner.json && \
               /PlexCleaner/PlexCleaner --logfile /PlexCleaner/PlexCleaner.log --logwarning process --settingsfile /PlexCleaner/PlexCleaner.json --mediafiles /mnt/archivedisk/TV --mediafiles /mnt/nas/TV --mediafiles /mnt/nas/Movies --mediafiles /plex-media"

Relevant log output.

12:31:02 [INF] <1> "Process" (72.37%) Before : "/mnt/nas/TV/Resident Alien/Season 03/Resident Alien (2021) - S03E06 - Bye Bye Birdie [WEBDL-1080p][EAC3 5.1][x264]-SuccessfulCrab.mkv"
12:31:02 [INF] <1> Reading media info from sidecar : "Resident Alien (2021) - S03E06 - Bye Bye Birdie [WEBDL-1080p][EAC3 5.1][x264]-SuccessfulCrab.PlexCleaner"
12:31:02 [WRN] <1> Sidecar LastWriteTimeUtc out of sync with media file : 03/21/2024 13:05:55 != 03/21/2024 13:09:58 : "Resident Alien (2021) - S03E06 - Bye Bye Birdie [WEBDL-1080p][EAC3 5.1][x264]-SuccessfulCrab.PlexCleaner"
12:31:02 [WRN] <1> Sidecar SHA256 out of sync with media file : "REDACTED" != "REDACTED" : "Resident Alien (2021) - S03E06 - Bye Bye Birdie [WEBDL-1080p][EAC3 5.1][x264]-SuccessfulCrab.PlexCleaner"
12:31:02 [WRN] <1> Sidecar out of sync with media file, clearing state : "Resident Alien (2021) - S03E06 - Bye Bye Birdie [WEBDL-1080p][EAC3 5.1][x264]-SuccessfulCrab.PlexCleaner"
12:31:02 [INF] <1> Sidecar read : State: FileModified : "Resident Alien (2021) - S03E06 - Bye Bye Birdie [WEBDL-1080p][EAC3 5.1][x264]-SuccessfulCrab.PlexCleaner"
12:31:02 [INF] <1> Reading media info from tools : "Resident Alien (2021) - S03E06 - Bye Bye Birdie [WEBDL-1080p][EAC3 5.1][x264]-SuccessfulCrab.mkv"
12:31:02 [INF] <1> Executing MediaInfo : "--Output=XML \"/mnt/nas/TV/Resident Alien/Season 03/Resident Alien (2021) - S03E06 - Bye Bye Birdie [WEBDL-1080p][EAC3 5.1][x264]-SuccessfulCrab.mkv\""

Settings file.

settings.json

Log file.

No response

Media file information.

13:26:36 [INF] <1> Loading settings from : "PlexCleaner/PlexCleaner.json"
13:26:36 [INF] <1> Application Version : "PlexCleaner : 3.8.19.46621 (Release)"
13:26:36 [INF] <1> Runtime Version : ".NET 8.0.4" : "ubuntu.24.04-x64"
13:26:36 [INF] <1> OS Version : "Ubuntu 24.04 LTS"
13:26:36 [INF] <1> Build Date : 05/12/2024 21:16:24
13:26:36 [INF] <1> Parallel Processing: False : Thread Count: 1, Processor Count: 8
13:26:36 [INF] <1> Executing FfMpeg : "-version"
13:26:36 [INF] <1> FfMpeg : Version: "6.1.1", Path: "ffmpeg"
13:26:36 [INF] <1> Executing FfProbe : "-version"
13:26:36 [INF] <1> FfProbe : Version: "6.1.1", Path: "ffprobe"
13:26:36 [INF] <1> Executing MkvMerge : "--version"
13:26:36 [INF] <1> MkvMerge : Version: "82.0", Path: "mkvmerge"
13:26:36 [INF] <1> Executing MkvPropEdit : "--version"
13:26:36 [INF] <1> MkvPropEdit : Version: "82.0", Path: "mkvpropedit"
13:26:36 [INF] <1> Executing MkvExtract : "--version"
13:26:36 [INF] <1> MkvExtract : Version: "82.0", Path: "mkvextract"
13:26:36 [INF] <1> Executing MediaInfo : "--version"
13:26:36 [INF] <1> MediaInfo : Version: "24.01", Path: "mediainfo"
13:26:36 [INF] <1> Executing HandBrake : "--version"
13:26:36 [INF] <1> HandBrake : Version: "1.7.2", Path: "HandBrakeCLI"
13:26:36 [INF] <1> Executing SevenZip : ""
13:26:36 [INF] <1> SevenZip : Version: "(x64)", Path: "7z"
13:26:36 [INF] <1> Creating file and folder list ...
13:26:36 [INF] <1> Discovered 1 files from 0 directories
13:26:36 [INF] <1> Starting "Get Media Information", processing 1 files ...
13:26:36 [INF] <1> "Get Media Information" (0.00%) Before : "/mnt/nas/TV/Resident Alien/Season 03/Resident Alien (2021) - S03E06 - Bye Bye Birdie [WEBDL-1080p][EAC3 5.1][x264]-SuccessfulCrab.mkv"
13:26:36 [INF] <1> Reading media info from sidecar : "Resident Alien (2021) - S03E06 - Bye Bye Birdie [WEBDL-1080p][EAC3 5.1][x264]-SuccessfulCrab.PlexCleaner"
13:26:37 [WRN] <1> Sidecar LastWriteTimeUtc out of sync with media file : 03/21/2024 13:05:55 != 03/21/2024 13:09:58 : "Resident Alien (2021) - S03E06 - Bye Bye Birdie [WEBDL-1080p][EAC3 5.1][x264]-SuccessfulCrab.PlexCleaner"
13:26:37 [WRN] <1> Sidecar SHA256 out of sync with media file : "REDACTED" != "REDACTED" : "Resident Alien (2021) - S03E06 - Bye Bye Birdie [WEBDL-1080p][EAC3 5.1][x264]-SuccessfulCrab.PlexCleaner"
13:26:37 [WRN] <1> Sidecar out of sync with media file, clearing state : "Resident Alien (2021) - S03E06 - Bye Bye Birdie [WEBDL-1080p][EAC3 5.1][x264]-SuccessfulCrab.PlexCleaner"
13:26:37 [INF] <1> Sidecar read : State: FileModified : "Resident Alien (2021) - S03E06 - Bye Bye Birdie [WEBDL-1080p][EAC3 5.1][x264]-SuccessfulCrab.PlexCleaner"
13:26:37 [INF] <1> Reading media info from tools : "Resident Alien (2021) - S03E06 - Bye Bye Birdie [WEBDL-1080p][EAC3 5.1][x264]-SuccessfulCrab.mkv"
13:26:37 [INF] <1> Executing MediaInfo : "--Output=XML \"/mnt/nas/TV/Resident Alien/Season 03/Resident Alien (2021) - S03E06 - Bye Bye Birdie [WEBDL-1080p][EAC3 5.1][x264]-SuccessfulCrab.mkv\""
ptr727 commented 3 weeks ago

Are you monitoring the PlexCleaner process memory usage, or docker memory usage, or your host system memory usage?

Did you capture any data for the process memory usage over time, or can correlate it with file count?

Was there anything in particular in IsMediaCurrent() or ComputeHash() that you suspect is leaking, consider that the only memory allocations are being made in ComputeHash(), and that happens for all modified and new files, and the garbage collector does cleanup, this is C# not C++?

If you were monitoring host system or docker subsystem memory usage, refer to the readme on tuning your docker log that can consume large amounts of memory.

ptr727 commented 2 weeks ago

Closing due to no response.