[Docker] Liberation failing - Too Many Files Open

[kspillane]

Describe the bug

Book liberation is failing. Program launches, sees books to download, but fails to download and process them. Running via docker on an Asustor NAS, which is a Linux host running a custom version of linux. Doesn't look like a host issue s as no other containers have issues.

To Reproduce

This runs via docker-compose as follows:

libation: image: rmcrackan/libation container_name: libation restart: always volumes:

• /volume1/Docker/Libation:/config
• /volume1/Media/Audiobooks:/data

I can supply my config files if you'd like.

My suspicion is this could be related to me having a rather large library (2600 items), or possibly a book that failed to process?

Expected behavior

Book should be processed as expected and moved into my Audiobooks folder where they get picked up by Audiobookshelf.

I'd appreciate any help, advice, or assistance.

Screenshots Running via Docker - see docker logs below

Platform

Running via Docker on Linux Host (Asustor ADM NAS Linux)

Log Files

2024-07-02 21:04:04: Starting 2024-07-02 21:04:04: Sleep time is set to 30m 2024-07-02 21:04:04: Linking config directory to the Libation config directory 2024-07-02 21:04:04: Scanning accounts Unhandled exception. System.TypeInitializationException: The type initializer for 'LibationFileManager.AudibleFileStorage' threw an exception. ---> System.IO.IOException: Too many open files : '/data/John McManusD-Day Revisited꞉ The Invasion of Normandy' at System.IO.Enumeration.FileSystemEnumerator1.Init() at System.IO.Enumeration.FileSystemEnumerable1..ctor(String directory, FindTransform transform, EnumerationOptions options, Boolean isNormalized) at System.IO.Enumeration.FileSystemEnumerableFactory.UserDirectories(String directory, String expression, EnumerationOptions options) at System.IO.Directory.InternalEnumeratePaths(String path, String searchPattern, SearchTarget searchTarget, EnumerationOptions options) at FileManager.FileUtility.SaferEnumerateFiles(LongPath path, String searchPattern, SearchOption searchOption) in /Source/FileManager/FileUtility.cs:line 250 at FileManager.FileUtility.SaferEnumerateFiles(LongPath path, String searchPattern, SearchOption searchOption) in /Source/FileManager/FileUtility.cs:line 253 at FileManager.BackgroundFileSystem.Init() in /Source/FileManager/BackgroundFileSystem.cs:line 61 at LibationFileManager.AudioFileStorage.newBookDirectoryFiles() in /Source/LibationFileManager/AudibleFileStorage.cs:line 124 at LibationFileManager.AudioFileStorage..ctor() in /Source/LibationFileManager/AudibleFileStorage.cs:line 109 at LibationFileManager.AudibleFileStorage..cctor() in /Source/LibationFileManager/AudibleFileStorage.cs:line 37 --- End of inner exception stack trace --- at LibationFileManager.AudibleFileStorage.get_BooksDirectory() in /Source/LibationFileManager/AudibleFileStorage.cs:line 43 at AppScaffolding.LibationScaffolding.logStartupState(Configuration config) in /Source/AppScaffolding/LibationScaffolding.cs:line 233 at AppScaffolding.LibationScaffolding.RunPostMigrationScaffolding(Variety variety, Configuration config) in /Source/AppScaffolding/LibationScaffolding.cs:line 103 at LibationCli.Program.Main(String[] args) in /Source/LibationCli/Program.cs:line 65 at LibationCli.Program.

(String[] args) ./libation/liberate.sh: line 66: 12 Aborted /libation/LibationCli scan 2024-07-02 21:04:05: Liberating books Unhandled exception. System.TypeInitializationException: The type initializer for 'LibationFileManager.AudibleFileStorage' threw an exception. ---> System.IO.IOException: Too many open files : '/data/John McManusD-Day Revisited꞉ The Invasion of Normandy' at System.IO.Enumeration.FileSystemEnumerator1.CreateDirectoryHandle(String path, Boolean ignoreNotFound) at System.IO.Enumeration.FileSystemEnumerator1.Init() at System.IO.Enumeration.FileSystemEnumerable`1..ctor(String directory, FindTransform transform, EnumerationOptions options, Boolean isNormalized) at System.IO.Enumeration.FileSystemEnumerableFactory.UserDirectories(String directory, String expression, EnumerationOptions options) at System.IO.Directory.InternalEnumeratePaths(String path, String searchPattern, SearchTarget searchTarget, EnumerationOptions options) at FileManager.FileUtility.SaferEnumerateFiles(LongPath path, String searchPattern, SearchOption searchOption) in /Source/FileManager/FileUtility.cs:line 250 at FileManager.FileUtility.SaferEnumerateFiles(LongPath path, String searchPattern, SearchOption searchOption) in /Source/FileManager/FileUtility.cs:line 253 at FileManager.BackgroundFileSystem.Init() in /Source/FileManager/BackgroundFileSystem.cs:line 61 at LibationFileManager.AudioFileStorage.newBookDirectoryFiles() in /Source/LibationFileManager/AudibleFileStorage.cs:line 124 at LibationFileManager.AudioFileStorage..ctor() in /Source/LibationFileManager/AudibleFileStorage.cs:line 109 at LibationFileManager.AudibleFileStorage..cctor() in /Source/LibationFileManager/AudibleFileStorage.cs:line 37 --- End of inner exception stack trace --- at LibationFileManager.AudibleFileStorage.get_BooksDirectory() in /Source/LibationFileManager/AudibleFileStorage.cs:line 43 at AppScaffolding.LibationScaffolding.logStartupState(Configuration config) in /Source/AppScaffolding/LibationScaffolding.cs:line 233 at AppScaffolding.LibationScaffolding.RunPostMigrationScaffolding(Variety variety, Configuration config) in /Source/AppScaffolding/LibationScaffolding.cs:line 103 at LibationCli.Program.Main(String[] args) in /Source/LibationCli/Program.cs:line 65 at LibationCli.Program.

(String[] args) ./libation/liberate.sh: line 66: 25 Aborted /libation/LibationCli liberate 2024-07-02 21:04:06: Sleeping for 30m

[rmcrackan]

Just to be clear: the docker build is a courtesy and is not officially supported so I'll not be addressing this one. I'll leave this ticket open though in hopes that some one from the community can help. I'd also like to edit the title to make it more visible to such folks.

[wtanksleyjr]

I might have seen this, maybe not with Libation (because I don't run it under Docker). Vague memories. Check if your host system has a high enough number of file descriptors by running ulimit -n; if it seems low, you could temporarily increase it and see if it helps by running ulimit -n 65536 (or something), that'll be effective in your current session. I think you make that permanent using /etc/security/limits.conf, but honestly I'm not sure anymore. You'll also want to check docker exec <container_id> lsof | wc -l to see that the number of open files. Also check /etc/docker/daemon.json, which will have Hard and Soft ulimits for "nofile" ulimits.

[kspillane]

So, I have been working on this for a couple of hours now and need to go to bed. Here's what I have discovered so far for anyone interested:

• I don't think this is a ulimit issue. By default ulimit is inherited from the docker host, it can also be manually set on launch (or via compose), I upped it to 8192, the prior limit was 1024, the number of open files was only 31 inside the container itself. So, I don't think this is the issue.
• The Dockerfile CMD calls /libation/liberate.sh on launch, it checks some things and then calls /libation/LibationCli scan and liberate - That's where the dotnet error happens.
• The error happens when the container is started via docker run or by bringing up the stack.
• BUT the script (and calls to LibationCLI) runs normally, with no errors when called via docker exec, or when ran via interactive console session inside the container.
• It was an issue with neither filenames (I fixed a type in my foldertemplate in settings.json that I thought might be causing it).
• Nor was it an issue with the particular book. I flagged that book as downloaded, and it just threw the error on the next book in the list.

If I ever get time this week, I will clone the repo and do some more troubleshooting, like having the script run printenv to see if there is anything weird between calling it via docker run and docker exec.

[wtanksleyjr]

Nice work.

Our script does a kind of unusual thing (IIRC) in that we copy outside files into the container, which will actually put them into a filesystem layer (Docker filesystems are immutable). It might be related to that... although I'd kind of expect to see it work the same for run vs exec, but maybe not.