search

dotnet / docfx

Static site generator for .NET API documentation.
https://dotnet.github.io/docfx/
MIT License
4.01k stars 853 forks source link

[Bug] System.IO.IOException: The process cannot access the file #8654

Open VaclavElias opened 1 year ago

VaclavElias commented 1 year ago

Describe the bug

I run docfx en/docfx.json and get this error below, but it is very rare and random. Maybe worth to log it here. Maybe it happens once per 20-30 runs in my case.

System.IO.IOException: The process cannot access the file 'D:\Projects\GitHub\stride-docs-next\en\_site\api\Valve.VR.PropertyWrite_t.html' because it is being used by another process.
   at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at Microsoft.DocAsCode.Common.ManifestFileWriter.Create(RelativePath file) in /_/src/Microsoft.DocAsCode.Common/FileAbstractLayer/ManifestFileWriter.cs:line 65
   at Microsoft.DocAsCode.Plugins.RootedFileAbstractLayer.Create(String file) in /_/src/Microsoft.DocAsCode.Plugins/RootedFileAbstractLayer.cs:line 31
   at Microsoft.DocAsCode.Build.Engine.HtmlPostProcessor.Process(Manifest manifest, String outputFolder) in /_/src/Microsoft.DocAsCode.Build.Engine/PostProcessors/HtmlPostProcessor.cs:line 44
   at Microsoft.DocAsCode.Build.Engine.PostProcessorsHandler.Handle(List`1 postProcessors, Manifest manifest, String outputFolder) in /_/src/Microsoft.DocAsCode.Build.Engine/PostProcessors/PostProcessorsHandler.cs:line 31
   at Microsoft.DocAsCode.Build.Engine.DocumentBuilder.Build(IList`1 parameters, String outputDirectory) in /_/src/Microsoft.DocAsCode.Build.Engine/DocumentBuilder.cs:line 173
   at Microsoft.DocAsCode.SubCommands.DocumentBuilderWrapper.BuildDocument(BuildJsonConfig config, BuildOptions options, TemplateManager templateManager, String baseDirectory, String outputDirectory, String pluginDirectory, String templateDirectory) in /_/src/[Microsoft.DocAsCode.App/Helpers/DocumentBuilderWrapper.cs:line](http://microsoft.docascode.app/Helpers/DocumentBuilderWrapper.cs:line) 47
   at Microsoft.DocAsCode.RunBuild.Exec(BuildJsonConfig config, BuildOptions options, String configDirectory, String outputDirectory) in /_/src/[Microsoft.DocAsCode.App/RunBuild.cs:line](http://microsoft.docascode.app/RunBuild.cs:line) 31

To Reproduce

Just a regular run of docfx build.

Expected behavior

Probably it shouldn't happen. The processes shouldn't be locking the file?

Context (please complete the following information):

Another example:

image
sakno commented 1 year ago

I have the same issue.

VaclavElias commented 1 year ago

Updated error 2.66.0

Error: System.IO.IOException: The process cannot access the file 'D:\Projects\GitHub\stride-docs-next\en\_site\manual\graphics\textures\index.html' because it is being used by another process.
   at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at System.IO.Strategies.FileStreamHelpers.ChooseStrategy(FileStream fileStream, String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize)
   at System.IO.File.Create(String path)
   at Microsoft.DocAsCode.Common.ManifestFileWriter.Create(RelativePath file) in /_/src/Microsoft.DocAsCode.Common/FileAbstractLayer/ManifestFileWriter.cs:line 52
   at Microsoft.DocAsCode.Common.FileAbstractLayer.Create(RelativePath file) in /_/src/Microsoft.DocAsCode.Common/FileAbstractLayer/FileAbstractLayer.cs:line 70
   at Microsoft.DocAsCode.Common.FileAbstractLayer.Create(String file) in /_/src/Microsoft.DocAsCode.Common/FileAbstractLayer/FileAbstractLayer.cs:line 136
   at Microsoft.DocAsCode.Plugins.RootedFileAbstractLayer.Create(String file) in /_/src/Microsoft.DocAsCode.Plugins/RootedFileAbstractLayer.cs:line 40
   at Microsoft.DocAsCode.Build.Engine.HtmlPostProcessor.Process(Manifest manifest, String outputFolder) in /_/src/Microsoft.DocAsCode.Build.Engine/PostProcessors/HtmlPostProcessor.cs:line 44
   at Microsoft.DocAsCode.Build.Engine.PostProcessorsHandler.Handle(List`1 postProcessors, Manifest manifest, String outputFolder) in /_/src/Microsoft.DocAsCode.Build.Engine/PostProcessors/PostProcessorsHandler.cs:line 31
   at Microsoft.DocAsCode.Build.Engine.DocumentBuilder.Build(IList`1 parameters, String outputDirectory) in /_/src/Microsoft.DocAsCode.Build.Engine/DocumentBuilder.cs:line 161
   at Microsoft.DocAsCode.DocumentBuilderWrapper.BuildDocument(BuildJsonConfig config, BuildOptions options, TemplateManager templateManager, String baseDirectory, String outputDirectory, String pluginDirectory, String templateDirectory) in
/_/src/Microsoft.DocAsCode.App/Helpers/DocumentBuilderWrapper.cs:line 47
   at Microsoft.DocAsCode.RunBuild.Exec(BuildJsonConfig config, BuildOptions options, String configDirectory, String outputDirectory) in /_/src/Microsoft.DocAsCode.App/RunBuild.cs:line 30
TIS-Tim commented 1 year ago

Same problem in 2.67.1 (2.67.1+9929fafef7443615f562bf47a49a3f783de3d375)

But not exactly rare for me, I would guess it happens once in every 4 tries.

VaclavElias commented 1 year ago

How many pages are you processing?

In my case it is rare with these:

Building 1620 file(s) in ResourceDocumentProcessor(ValidateResourceMetadata)...
Building 303 file(s) in ConceptualDocumentProcessor(BuildConceptualDocument=>CountWord=>ValidateConceptualDocumentMetadata)...
Building 4 file(s) in TocDocumentProcessor(BuildTocDocument)...
TIS-Tim commented 1 year ago

My project is way smaller I guess:

Building 2 file(s) in ResourceDocumentProcessor(ValidateResourceMetadata)...
Building 3 file(s) in ConceptualDocumentProcessor(BuildConceptualDocument=>CountWord=>ValidateConceptualDocumentMetadata)...
Building 3 file(s) in TocDocumentProcessor(BuildTocDocument)...
Building 72 file(s) in ManagedReferenceDocumentProcessor(BuildManagedReferenceDocument=>SplitClassPageToMemberLevel=>ValidateManagedReferenceDocumentMetadata=>ApplyOverwriteDocumentForMref=>FillReferenceInformation)...
VaclavElias commented 1 year ago

That is interesting. I hope our input will help to find and resolve the issue.

yufeih commented 1 year ago

@TIS-Tim Is it convenient to share your project for me to repro?

TIS-Tim commented 1 year ago

@yufeih

Once I tried to package the relevant files, I wanted to try to repro one more time, and it never happened.

Returning to my original project, with the solution opened in visual studio and continuing to work on that immediately triggered the issue again.

So my guess is it is probably not caused by the data, but by the workflow... like browsing the results, continue editing the sources during the build, having Visual Studio open on the project, something like that.

filzrev commented 1 year ago

I've tried docfx build command with reported docs content with related metadata.

But I can't reproduce IOException in my environment. (executed about 100 times) If there are specific procedure or conditions to reproduce problem. I would like to know.

Below is a description of the investigation

Execution logs

[01] Building 1620 file(s) in ResourceDocumentProcessor(ValidateResourceMetadata)...
[08] Building 5 file(s) in TocDocumentProcessor(BuildTocDocument)...
[07] Building 303 file(s) in ConceptualDocumentProcessor(BuildConceptualDocument=>CountWord=>ValidateConceptualDocumentMetadata)...
[12] Building 2707 file(s) in ManagedReferenceDocumentProcessor(BuildManagedReferenceDocument=>SplitClassPageToMemberLevel=>ValidateManagedReferenceDocumentMetadata=>ApplyOverwriteDocumentForMref=>FillReferenceInformation)...

Note: Leading number is thread id, It's appended to checking executed thread. for debug purpose.

Details

  1. IOException is occurred at HtmlPostProcessor (by reported logs)
    https://github.com/dotnet/docfx/blob/db8ab17c5770fb008fe13e600340caed20db5cab/src/Microsoft.DocAsCode.Build.Engine/PostProcessors/HtmlPostProcessor.cs#L85

  2. File::OpenRead operation is succeeded that is executed before HtmlPostProcessors handlers execution.
    https://github.com/dotnet/docfx/blob/db8ab17c5770fb008fe13e600340caed20db5cab/src/Microsoft.DocAsCode.Build.Engine/PostProcessors/HtmlPostProcessor.cs#L73

  3. HtmlPostProcessor's handlers(ValidateBookmark/RemoveDebugInfo) does't touch html file. So It's unlikely, this problem is caused by HtmlPostProcessor's processing. but other place or external process lock HTML files.

  4. PostProcessors are executed sequential order and HtmlPostProcessor is executed first.
    So It's unlikely this problem is caused by other PostProcessor's parallel operation.

  5. HTML file is created first at TemplateModelTransformer
    And if intentionally set exclusive lock to this file (by external process). IOException occurred here.
    https://github.com/dotnet/docfx/blob/db8ab17c5770fb008fe13e600340caed20db5cab/src/Microsoft.DocAsCode.Build.Engine/TemplateProcessors/TemplateModelTransformer.cs#L277

File lock related activity logs

I've get file access logs using Sysinternals's ProcessMonitor tool.
In my environment. Microsoft Defender(MsMpEng.exe) is working during docfx.exe execution.

image

Syncaidius commented 1 year ago

I'm also getting this error during a GitHub actions run of DocFX. Sometimes it works fine, sometimes it fails: https://github.com/Syncaidius/MoltenEngine/actions/runs/5689587121/job/15421370149

Exception:

[23-07-28 07:59:12.451]Error:System.IO.IOException: The process cannot access the file 'D:\a\MoltenEngine\MoltenEngine\docs\images/logo.svg' because it is being used by another process.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.File.InternalCopy(String sourceFileName, String destFileName, Boolean overwrite, Boolean checkHost)
   at Microsoft.DocAsCode.Common.RealFileWriter.Copy(PathMapping sourceFileName, RelativePath destFileName)
   at Microsoft.DocAsCode.Common.FileAbstractLayer.Copy(RelativePath sourceFileName, RelativePath destFileName)
   at Microsoft.DocAsCode.Common.ManifestFileHelper.<>c__DisplayClass8_0.<Dereference>b__4(OutputFileInfo ofi)
   at System.Threading.Tasks.Parallel.<>c__DisplayClass42_0`2.<PartitionerForEachWorker>b__1()
   at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
   at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object <p0>)
    514 Warning(s)
    1 Error(s)
Syncaidius commented 1 year ago

Interestingly, changing the docfx.json from:

"files": [
            "images/**"
          ]

to:

"files": [
            "images\\logo.svg"
          ]

Fixed it for me.

filzrev commented 11 months ago

I'm also encountered this problems with very rare rate. (I've set Windows Defender folder exclusion settings to docfx project folder)

And it seems also occurred at CI environment (Windows/ 2-CPU) https://github.com/dotnet/docfx/actions/runs/6404063982/job/17383853477

VaclavElias commented 11 months ago

A mysterious 👻🐛🤣. In the last 7 days, using DocFX daily, this issue occurred 3 times in a row on random files and then it was gone again.

Also, luckily this issue didn't occur through GitHub Action deployment to Azure App Service and in another app to GitHub Pages.

yufeih commented 11 months ago

Seems to be more obvious after perf improvements to #9314 . CI failed many times in a row.

filzrev commented 9 months ago

Is this problem still occurred on latest docfx? I've not encountered this problem about a month on local environment (and docfx CI build).

If it still reproduceable on latest docfx version. It might be better to add logics to logging what process cause file lock (For Windows). Similar to https://github.com/dotnet/msbuild/pull/9488.

gdimauro commented 2 weeks ago

exclude folders you are writing to from windows security/defender:

https://support.microsoft.com/en-us/windows/add-an-exclusion-to-windows-security-811816c0-4dfd-af4a-47e4-c301afe13b26#:~:text=Go%20to%20Start%20%3E%20Settings%20%3E%20Update,%2C%20file%20types%2C%20or%20process.