dotnet / linker

388 stars 126 forks source link

Call CheckProcessed before enqueuing to MarkStep._methods #3139

Closed jtschuster closed 1 year ago

jtschuster commented 1 year ago

Fixes https://github.com/dotnet/linker/issues/3137

We unconditionally add a method to the MarkStep._methods queue and perform a filtering check in ProcessMethod later. This causes _methods to grow to be as large as 110 mb in some cases (see https://github.com/dotnet/linker/issues/3126#issuecomment-1329508820). If we instead perform the filtering check before enqueuing the method, MarkStep_methods reach a max size of ~14mb (in the same repro).