search

dotnet / docfx

Static site generator for .NET API documentation.
https://dotnet.github.io/docfx/
MIT License
3.94k stars 840 forks source link

feat: Add support for gzipped xrefmap that stored as local file #9966

Closed filzrev closed 1 month ago

filzrev commented 1 month ago

This PR add support for gzip compressed xrefmap file.

Background

xrefmap files can exceed 100 MB in size in some cases. For example. .NET API's xrefmap takes 132MB. (And 8.5MB when using gzip compression)

And when specifying xrefmap with URL schema. docfx download xrefmap file on every build.

So it's recommended to use pre downloaded local file for large xrefmap. But in this case, xrefmap can't be stored on GitHub repository. (By default it's limited to50MB and It requires git LFS for larger file)

And currently there is zipped xrefmap support that compressed by zip. But it support only yml format that is created by dotnet download command. And yml file processing run 4x slower than json.

This PR add support for gzipped xrefmap file to resolve described above issue.

codecov[bot] commented 1 month ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 78.91%. Comparing base (fe673ec) to head (72e686c). Report is 189 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #9966 +/- ## ========================================== + Coverage 74.31% 78.91% +4.59% ========================================== Files 536 540 +4 Lines 23189 23439 +250 Branches 4056 4060 +4 ========================================== + Hits 17234 18497 +1263 + Misses 4853 3804 -1049 - Partials 1102 1138 +36 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

yufeih commented 1 month ago

will a .xrefmap.json.gz be uploaded in a separate PR?

filzrev commented 1 month ago

will a .xrefmap.json.gz be uploaded in a separate PR?

This PR intended to support gzipped files that saved on local files. For network files. gzip support is not implemented. Because it should use HTTP Compression if available.

For https://github.com/dotnet/docfx/raw/main/.xrefmap.json file. It's stored on Git LFS. and don't support HTTP Compression. (https://github.com/dotnet/docfx/issues/9659#issuecomment-1914522207)

In this case. It need manually download xrefmap and compress as gzip.

Alternatively. It can use xrefmap provided by learn.microsoft.com (Though It might not officially supported)

https://learn.microsoft.com/en-us/dotnet/.xrefmap.json