AlexCSDev / PatreonDownloader

Powerful tool for downloading content posted by creators on patreon.com. Supports content hosted on patreon itself as well as external sites (additional plugins might be required).
MIT License
928 stars 95 forks source link

Seeing error that says "The process cannot access the file (...) because it is being used by another process" #163

Closed blohbeepblohgh closed 10 months ago

blohbeepblohgh commented 1 year ago

Using the most recent release 24.0.0.0 on W10 22H2.

Getting this error randomly because a bunch of different images have the same filename (e.g., "1.png"):

ERROR [PatreonDownloader.App.Program] Failed to download #the file URL#: Error while downloading #the file URL#: Unable to delete existing temporary file #my directory#\external_1.png.dwnldtmp. Inner Exception: System.IO.IOException: The process cannot access the file '#my directory#\external_1.png.dwnldtmp' because it is being used by another process.
   at System.IO.FileSystem.DeleteFile(String fullPath)
   at System.IO.File.Delete(String path)
   at UniversalDownloaderPlatform.DefaultImplementations.WebDownloader.DownloadFileInternal(String url, String path, String refererUrl, Int32 retry, Int32 retryTooManyRequests) in F:\Sources\BigProjects\PatreonDownloader\submodules\UniversalDownloaderPlatform\UniversalDownloaderPlatform.DefaultImplementations\WebDownloader.cs:line 287

These are the options I've used: --descriptions --embeds --json --campaign-images --log-save --use-sub-directories

If I run it 1 or 2 more times to the same directory eventually all files do get downloaded. When I tested downloading to a different directory, I'd see the same error but for different files.

AlexCSDev commented 1 year ago

Please replicate this error with --log-level Trace --log-save --json --use-sub-directories parameters and send log file as well as json files to alexcsdev@protonmail.com.

chairwalker commented 1 year ago

I get this when trying to do the above. This works with a campaign that is working normally, but not the one below (redacted).

2023-02-07 09:52:29.8515 FATAL Fatal error, application will be closed: System.IO.DirectoryNotFoundException: Could not find a part of the path 'E:\patreon\net6.0-win-x64-release1\download\[redacted]\page_1.json'.
   at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at System.IO.Strategies.FileStreamHelpers.ChooseStrategyCore(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at System.IO.Strategies.FileStreamHelpers.ChooseStrategy(FileStream fileStream, String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize)
   at System.IO.File.AsyncStreamWriter(String path, Encoding encoding, Boolean append)
   at System.IO.File.WriteAllTextAsync(String path, String contents, Encoding encoding, CancellationToken cancellationToken)
   at PatreonDownloader.Implementation.PatreonPageCrawler.Crawl(ICrawlTargetInfo crawlTargetInfo)
   at UniversalDownloaderPlatform.Engine.UniversalDownloader.Download(String url, IUniversalDownloaderPlatformSettings settings) in F:\Sources\BigProjects\PatreonDownloader\submodules\UniversalDownloaderPlatform\UniversalDownloaderPlatform.Engine\UniversalDownloader.cs:line 206
   at PatreonDownloader.App.Program.RunPatreonDownloader(CommandLineOptions commandLineOptions) in F:\Sources\BigProjects\PatreonDownloader\PatreonDownloader.App\Program.cs:line 128
   at PatreonDownloader.App.Program.Main(String[] args) in F:\Sources\BigProjects\PatreonDownloader\PatreonDownloader.App\Program.cs:line 68
AlexCSDev commented 1 year ago

This issue will be closed in a few days unless requested information will be provided by anyone who is having this issue.

Ovear commented 1 year ago

I encountered this problem with [Release 25] too.

It seems the program downloaded files inside posts with different order, which caused the program to download duplicated files.

There are multiple external_1_*_old_time files in directory, and the order of file is not same as website with distributed in random order.

Let me check what I can do.

PS: The filename of those files in the post are 1.png/jpg, and the server also won't provide its original name, so the program would take 1.png as the filename.

PS: This duplicated files's name are start with external_.

Update: I think this may be caused by PluginManager.cs of UniversalDownloaderPlatform. It uses HashSet to store address which doesn't guarantee the insertion order, and then it converted to List directly[1].

I will try to make it keep insertion order to see if fix this.

[1] https://github.com/AlexCSDev/UniversalDownloaderPlatform/blob/4f31a00746a6814ad8acb8cb72e35bc0db62b9d7/UniversalDownloaderPlatform.Engine/PluginManager.cs#L112

AlexCSDev commented 10 months ago

Should be fixed in Release 26/27, thanks to Ovear for the help!