Docs: DocFx Build Failure for API Docs

NightOwl888 commented 6 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

Describe the bug

The API documentation is failing to build. The website is generated with no URLs being resolved.

Expected Behavior

The documentation will build and all links will be resolvable.

Steps To Reproduce

Clone the repository locally
Run the powershell command ./websites/apidocs/docs.ps1 -ServeDocs -LuceneNetVersion 4.8.0-beta00016 -BaseUrl http://localhost:8080

See the API Docs build instructions.

Exceptions (if any)

Building api metadata for D:\a\lucenenet\lucenenet\main-repo\websites\apidocs\docfx.codecs.json...
[24-02-01 10:39:03.573]Warning:[ExtractMetadata](D:/a/lucenenet/lucenenet/main-repo/src/Lucene.Net.Codecs/Lucene.Net.Codecs.csproj)Workspace failed with: [Failure] Msbuild failed when processing the file 'D:\a\lucenenet\lucenenet\main-repo\src\Lucene.Net.Codecs\Lucene.Net.Codecs.csproj' with message: Method not found: 'System.ReadOnlySpan`1<Char> Microsoft.IO.Path.GetFileName(System.ReadOnlySpan`1<Char>)'.
[24-02-01 10:39:04.042]Warning:[ExtractMetadata]Project 'D:\a\lucenenet\lucenenet\main-repo\src\Lucene.Net.Codecs\Lucene.Net.Codecs.csproj' does not contain any documents.
[24-02-01 10:39:04.417]Warning:[ExtractMetadata]No metadata is generated for Lucene.Net.Codecs.

Lucene.NET Version

Lucene.NET 4.8.0-beta00016

.NET Version

No response

Operating System

Windows 10 x64

Anything else?

I did a search and discovered the following issue that appears to be the same problem:

https://github.com/dotnet/docfx/issues/8097

The solution was to upgrade to 2.59.4. I attempted that next, and then got hit with the error that System.Memory had multiple versions referenced and could not load it similar to this comment.

In this case, we recently added a dependency on the System.Memory 4.5.5 NuGet package. Unfortunately, it seems that docfx has its own reference so it cannot resolve this, even if changing the assembly binding redirects.

Upgrade docfx to .NET Core?

Since it seems we have unresolvable conflicts, I attempted to upgrade to version 2.75.2 (which is now on .NET Core). It seems there have been some architecture changes, so I ran into the following issues:

[ ] LuceneDocsPlugins has a dependency on Microsoft.Composition, which is now deprecated and replaced by System.Composition (I tried 8.0.0).
[ ] LuceneDocsPlugins has a dependency on Microsoft.DocAsCode.Dfm, which is now deprecated. I wasn't able to work out what the replacement for that is.
[ ] LuceneDocsPlugins has a dependency on YamlDotNet 6.0.0 - but System.Composition has a transitive dependency on a newer version of it.
[ ] Even after updating the LuceneDocsPlugins so it will build, the plugins no longer seem to function with docfx 2.75.2.
[ ] The table of contents are empty on each of the pages.
[ ] The --cleanupCacheHistory, --force, and --forcePostProcess options no longer exist on docfx. I did a search in their repository, but it isn't clear why they were removed or what the recommended replacement is.
[ ] docfx-analysis-opennlp.json needs to be updated to use net6.0, since netstandard2.0 is not supported by IKVM.
[ ] Lucene.Net.Analysis.OpenNLP now fails to build metatdata when executed from docfx. This is most likely due to the fact that it uses <MavenReference> to build and add references on the fly, but docfx isn't running the <MavenReference> when building. See https://github.com/ikvmnet/ikvm-maven/issues/53.

A few things to note:

If we upgrade to use .NET Core, we can use docfx as a dotnet tool and install it from NuGet via command line. This will allow our docx to build xplat so anyone can contribute to them. It can be installed to a subdirectory using --toolpath.
If we cannot resolve the issue with <MavenReference>, docfx can build docs based off of either source code or the combination of DLL and XML (docs file).
Ideally, we would use the same version of docfx to build the website as we do for the api docs.

Shazwazza commented 6 months ago

oh my :(

NightOwl888 commented 5 months ago

@Shazwazza - Yeah, I think we will need to download the docfx source and run it in the debugger to determine how to fix the load issues with the plugins and also fix the MavenReference problem. For MavenReference I think we may need to switch to using the DLL/XML output instead of using the .csproj file, since it appears docfx doesn't actually run MSBuild.

filzrev commented 5 months ago

Lucene.Net.Analysis.OpenNLP now fails to build metatdata when executed from docfx.

On my environment. It can successfully execute docfx metadata docfx.analysis-opennlp.json command with following changes.

1. Remove MSBuild TargetFramework property from docfx.analysis-opennlp.json

It seems this settings override target framework(net6) that specified at Lucene.Net.Analysis.OpenNLP.csproj. https://github.com/apache/lucenenet/blob/43e0e894b9b40f5b28064a13ef98874a82c15330/websites/apidocs/docfx.analysis-opennlp.json#L17-L19

2. Build with .NET 8 SDK v8.0.200 I've confirmed following error is not occurred when build with .NET 8 preview SDK 8.0.200.

> ROSLYNCODETASKFACTORYCSHARPCOMPILER : error : Unknown command '/noconfig'.
C:\Users\sample\.nuget\packages\ikvm.image\8.7.5\buildTransitive\IKVM.Image.targets(95,9): error : The source file for this compilation can be found at: "C:\Users\sample\AppData\Local\Temp\MSBuildTempsample\tmpe1e76f87dcd94e66b4bcf0ec3cccbc5b.tmp"

Minimum reproducible .csproj for this issue

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net6</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="IKVM.Maven.Sdk" Version="1.6.8" />
    <PackageReference Include="IKVM.Image" Version="8.7.5" />
  </ItemGroup>
</Project>

paulirwin commented 4 months ago

When I try to run the docs.ps1 script on macOS arm64, I get the following error after the "Building api metadata for..." lines:

Building api metadata for /{path_to}/lucene.net/websites/apidocs/docfx.demo.json...
Building site output for /{path_to}/lucene.net/websites/apidocs/docfx.codecs.json...
Get-Content: /{path_to}/lucene.net/websites/apidocs/docs.ps1:222
Line |
 222 |          $xrefMap = Get-Content $xrefFile -Raw
     |                     ~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Cannot find path '/{path_to}/lucene.net/websites/apidocs/_site/api/codecs/xrefmap.yml' because it does
     | not exist.

apache / lucenenet