dotnet / linker

387 stars 134 forks source link

[release/7.0] Ports fixes for not crashing/haning on recursive arrays #3231

Closed vitek-karas closed 9 months ago

vitek-karas commented 9 months ago

This is a port of the trimmer/analyzer changes from https://github.com/dotnet/runtime/pull/82818.

The main changes is to disallow array values in arrays - if we find an array element which is itself an array we replace it with unknown value.

This stops all kinds of weird recursion behavior in the analysis and fixes cases where the tool would either crash with stack overflow or hang (exponential explosion of states).

Adds the related tests as well.

vitek-karas commented 9 months ago

This fixes the problem reported in https://github.com/dotnet/runtime/issues/93076.

vitek-karas commented 9 months ago

Replaced by a combined PR https://github.com/dotnet/linker/pull/3234.