fsprojects / FSharp.Formatting

F# tools for generating documentation (Markdown processor and F# code formatter)
https://fsprojects.github.io/FSharp.Formatting/
Other
462 stars 155 forks source link

Stack overflow on large amount of documents on windows #732

Closed kMutagene closed 2 years ago

kMutagene commented 2 years ago

The documentation of Plotly.NET has become quite large, and i recently started to run into problems when trying to watch/build the docs on windows. We are generating html, ipynb, and fsx output, so overall 210 (3x70) models are generated in one run.

On windows, after about 70% (using either fsdocs watch --eval or fsdocs build --eval) i get a stack overflow with a huge stack trace.

The offending function seems to be FSharp.Compiler.NameResolution.CollectAtMostOneResult , which loops over and over. I have not enough knowledge of the compiler to make anything out of this though.

Full console output:

build_log.txt

What i can say it that this has nothing to do with the actual file at which this happens. Deleting the file will just make this happen at the next one.

This does not happen on WSL ubuntu, nor on a full ubuntu machine, so my workaround for now is to use either of those to build docs.

I am afraid that the only way to reproduce this is to try and build the docs of plotly.net: https://github.com/plotly/Plotly.NET/tree/dev/docs

dsyme commented 2 years ago

Thanks for the report - strange problem.

Certainly this function in the F# compiler is not tail recursive:

let rec CollectAtMostOneResult f inputs = 
    match inputs with 
    | [] -> NoResultsOrUsefulErrors
    | [h] -> OneResult (f h)
    | h :: t ->
        match f h with
        | Result r -> Result [r]
        | Exception e -> AddResults (Exception e) (CollectAtMostOneResult f t)

However we've never seen a problem here before, there must be something about the environment being accumulated for type checking by all those scripts. We should really be resetting the processing environment for each new script in any case.

nhirschey commented 2 years ago

Are you still having issues @kMutagene? I was able to build plotly.net's documentation on windows today with dotnet fsdocs watch --eval. The watch process used 5GB of ram.

OS: Windows 11 (Windows_NT x64 10.0.22000) .Net version: 6.0.202

image
kMutagene commented 2 years ago

Interesting. I went ahead and checked this for your PR over at Plotly.NET, and it indeed works (which is great, thanks for pointing this out!). What i find strange is that i have not updated fsdocs recently AFAIK, so the fix has to come from somplace else. The only thing i can imagine is that i recently moved the build pipeline from using FAKE scripts to using the FAKE libraries in a project, and calling that one via dotnet run. Maybe this helps some users that come across this issue, but otherwise i cannot think of anything else that fixed this.

dsyme commented 2 years ago

Sounds like we can close this then? :)