Scanning lib folder fails on MacOS file system inside docker

hamada147 commented 4 days ago

This issue respects the following points:

[x] This is a bug, not a question or a configuration issue; Please visit our forum or chat rooms first to troubleshoot with volunteers, before creating a report. The links can be found here.
[x] This issue is not already reported on GitHub (I've searched it).
[x] I'm using an up to date version of Jellyfin Server stable, unstable or master; We generally do not support previous older versions. If possible, please update to the latest version before opening an issue.
[x] I agree to follow Jellyfin's Code of Conduct.
[x] This report addresses only a single issue; If you encounter multiple issues, kindly create separate reports for each one.

Description of the bug

On a new installation of JellyFin on a MacOS using docker, it fails to scan a folder with the MacOS file system because it is trying to read files that start with ._ and ._.DS_Store.

Reproduction steps

Have a fresh installation of JellyFin with docker from linuxserver.
Scan a lib folder with the macOS file system on the macOS directory.

What is the current bug behavior?

When reading a folder directory on a macOS file system, it tries to read and access all files even the ones that starts with ._ and ._.DS_Store.

What is the expected correct behavior?

When reading a folder directory on a macOS file system, it should ignore all files that start with ._ and ._.DS_Store.

Jellyfin Server version

10.10.0+

Specify commit id

No response

Specify unstable release number

No response

Specify version number

No response

Specify the build version

10.10.3

Environment

- OS: macOS
- Linux Kernel:
- Virtualization: Docker
- Clients:
- Browser: Google Chrome
- FFmpeg Version:
- Playback Method:
- Hardware Acceleration:
- GPU Model:
- Plugins:
- Reverse Proxy:
- Base URL:
- Networking:
- Storage:

Jellyfin logs

[2024-11-23 05:30:15.768 +02:00] [ERR] [22] Jellyfin.Api.Middleware.ExceptionMiddleware: Error processing request: "Could not find a part of the path '/movies'". URL "GET" "/Environment/DirectoryContents".
[2024-11-23 05:30:15.842 +02:00] [ERR] [22] Jellyfin.Api.Middleware.ExceptionMiddleware: Error processing request: "Could not find a part of the path '/movies'". URL "GET" "/Environment/DirectoryContents".
[2024-11-23 05:30:32.821 +02:00] [ERR] [3] Jellyfin.Api.Middleware.ExceptionMiddleware: Error processing request. URL "GET" "/Environment/DirectoryContents".
System.UnauthorizedAccessException: Access to the path '/data/movies/._.DS_Store' is denied.
 ---> System.IO.IOException: Operation not permitted
   --- End of inner exception stack trace ---
   at System.IO.FileSystemInfo.Create(String fullPath, String fileName, Boolean asDirectory, FileStatus& fileStatus)
   at System.IO.Enumeration.FileSystemEntry.ToFileSystemInfo()
   at System.IO.Enumeration.FileSystemEnumerator`1.MoveNext()
   at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.WhereEnumerableIterator`1.ToArray()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.OrderedEnumerable`1.GetEnumerator()+MoveNext()
   at System.Linq.Enumerable.SelectIPartitionIterator`2.MoveNext()
   at System.Text.Json.Serialization.Converters.IEnumerableDefaultConverter`2.OnWriteResume(Utf8JsonWriter writer, TCollection value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonCollectionConverter`2.OnTryWrite(Utf8JsonWriter writer, TCollection value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.SerializeAsync(Stream utf8Json, T rootValue, CancellationToken cancellationToken, Object rootValueBoxed)
   at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.SerializeAsync(Stream utf8Json, T rootValue, CancellationToken cancellationToken, Object rootValueBoxed)
   at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.SerializeAsync(Stream utf8Json, T rootValue, CancellationToken cancellationToken, Object rootValueBoxed)
   at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.SerializeAsync(Stream utf8Json, T rootValue, CancellationToken cancellationToken, Object rootValueBoxed)
   at Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonOutputFormatter.WriteResponseBodyAsync(OutputFormatterWriteContext context, Encoding selectedEncoding)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResultFilterAsync>g__Awaited|30_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Jellyfin.Api.Middleware.ServerStartupMessageMiddleware.Invoke(HttpContext httpContext, IServerApplicationHost serverApplicationHost, ILocalizationManager localizationManager)
   at Jellyfin.Api.Middleware.WebSocketHandlerMiddleware.Invoke(HttpContext httpContext, IWebSocketManager webSocketManager)
   at Jellyfin.Api.Middleware.IPBasedAccessValidationMiddleware.Invoke(HttpContext httpContext, INetworkManager networkManager)
   at Jellyfin.Api.Middleware.LanFilteringMiddleware.Invoke(HttpContext httpContext, INetworkManager networkManager, IServerConfigurationManager serverConfigurationManager)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Jellyfin.Api.Middleware.QueryStringDecodingMiddleware.Invoke(HttpContext httpContext)
   at Swashbuckle.AspNetCore.ReDoc.ReDocMiddleware.Invoke(HttpContext httpContext)
   at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Jellyfin.Api.Middleware.RobotsRedirectionMiddleware.Invoke(HttpContext httpContext)
   at Jellyfin.Api.Middleware.LegacyEmbyRouteRewriteMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.InvokeCore(HttpContext context)
   at Jellyfin.Api.Middleware.ResponseTimeMiddleware.Invoke(HttpContext context, IServerConfigurationManager serverConfigurationManager)
   at Jellyfin.Api.Middleware.ExceptionMiddleware.Invoke(HttpContext context)
[2024-11-23 05:38:31.457 +02:00] [ERR] [41] MediaBrowser.Controller.Entities.BaseItem: Error retrieving children
System.UnauthorizedAccessException: Access to the path '/data/tvshows/._Hellbound (2021)' is denied.
 ---> System.IO.IOException: Operation not permitted
   --- End of inner exception stack trace ---
   at System.IO.FileSystemInfo.Create(String fullPath, String fileName, Boolean asDirectory, FileStatus& fileStatus)
   at System.IO.Enumeration.FileSystemEntry.ToFileSystemInfo()
   at System.IO.Enumeration.FileSystemEnumerator`1.MoveNext()
   at System.Linq.Enumerable.SelectEnumerableIterator`2.ToArray()
   at MediaBrowser.Controller.Providers.DirectoryService.GetFileSystemEntries(String path)
   at MediaBrowser.Controller.Entities.BaseItem.GetFileSystemChildren(IDirectoryService directoryService)
   at MediaBrowser.Controller.Entities.Folder.GetNonCachedChildren(IDirectoryService directoryService)
   at MediaBrowser.Controller.Entities.Folder.ValidateChildrenInternal2(IProgress`1 progress, Boolean recursive, Boolean refreshChildMetadata, Boolean allowRemoveRoot, MetadataRefreshOptions refreshOptions, IDirectoryService directoryService, CancellationToken cancellationToken)

FFmpeg logs

Client / Browser logs

No response

Relevant screenshots or videos

No response

Additional information

No response

gnattu commented 4 days ago

All Unix hidden files should be ignored per this: https://github.com/jellyfin/jellyfin/blob/06c603428bceccdb793eb34cc0340e25552d6c1d/Emby.Server.Implementations/Library/IgnorePatterns.cs#L83-L88

I don’t know why it does not work in docker because this does not happen with the native install on macOS, the fact that any hidden files is being scanned is weird.

hamada147 commented 4 days ago

This is so weird, I just compiled and ran the source code from the master branch (commit SHA: 2b0f028b6f1c471db16996a36d0d6ee6f23bde57) and it does ignore all Unix hidden files. I don't know why in the docker instance it doesn't.

gnattu commented 3 days ago

I don't know why in the docker instance it doesn't.

After some investigation I found this is tricky to fix because it will change the file system enumeration quite a bit. The dotnet internal EnumerateFileSystemInfos requires read access to each and every file in the specified path as the IgnoreInaccessible property is not correctly handled inside the implementation: https://github.com/dotnet/runtime/blob/f1332ab0d82ee0e21ca387cbd1c8a87c5dfa4906/src/libraries/System.Private.CoreLib/src/System/IO/Enumeration/FileSystemEnumerableFactory.cs#L198

We have set the IgnoreInaccessible property to true in the EnumerationOptions. However, the implementation of .NET does not attempt to catch the exception raised by the ToFileSystemInfo call and simply throws even when IgnoreInaccessible is set to true. It should just silently ignore the file if an UnauthorizedAccessException was thrown.

jellyfin / jellyfin