PLINQ `SelectMany()` generates Native AOT warnings

DL444 commented 1 year ago

Description

Calling PLINQ .AsParallel().SelectMany() in code results in the following Native AOT warnings being generated when publishing for linux-x64, osx-arm64, and win-x64:

ILC: Method '[System.Linq.Parallel]System.Linq.Parallel.SelectManyQueryOperator`3<int32,int32,int32>.WrapPartitionedStreamNotIndexed<Pair`2<Pair`2<Pair`2<Pair`2<int32,int32>,int32>,int32>,int32>>(PartitionedStream`2<int32,Pair`2<Pair`2<Pair`2<Pair`2<int32,int32>,int32>,int32>,int32>>,IPartitionedStreamRecipient`1,QuerySettings)' will always throw because: Failed to load type 'System.Linq.Parallel.PartitionedStream`2<Int32, System.Linq.Parallel.Pair`2<System.Linq.Parallel.Pair`2<System.Linq.Parallel.Pair`2<System.Linq.Parallel.Pair`2<System.Linq.Parallel.Pair`2<Int32, Int32>, Int32>, Int32>, Int32>, Int32>>' from assembly 'System.Linq.Parallel, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'

Actual run-time behavior seems fine in my scenario, but there can be complications that I did not catch.

Reproduction Steps

Create a new project using console app template and configure for Native AOT deployment by adding <PublishAot>true</PublishAot> to .csproj.

Add the following reproduction code to Program.cs

int counter = 0;
Enumerable.Range(0, 5).AsParallel().SelectMany(x => Enumerable.Range(x * 10, 10).Select(x => 
{ 
int order = Interlocked.Increment(ref counter);
Console.WriteLine($"Begin {order}");
Thread.Sleep(Random.Shared.Next(1000)); // Simulate synchronous compute-intensive task.
Console.WriteLine($"End {order}");
return order; 
})).ToArray();

Publish for Native AOT by running command dotnet publish -c Release -r <platform-rid>

Expected behavior

Publish completes without warnings.

Actual behavior

Warning messages in the description section being generated. However, running the generated binary produces no exceptions, and the output reveals that the queries are indeed being executed in parallel.

Begin 1
Begin 2
Begin 3
Begin 4
Begin 5
End 2
Begin 6
End 3
Begin 7
End 1
Begin 8
End 8
Begin 9
End 9
Begin 10
End 10
Begin 11
End 7
Begin 12
...

Regression?

Likely not a regression. Warnings are generated for both .NET 7 and .NET 8 RC 1, with the former being the first release supporting Native AOT deployment.

Known Workarounds

Ignore the warning and test after publishing. Run-time behavior seems to be okay.
Remove or replace either .AsParallel() or .SelectMany() if possible. They publish fine when used alone.

Configuration

For `linux-x64`

.NET Version: 7.0.111 OS: Ubuntu 22.04.3 on WSL 2

For `osx-arm64`

.NET Version: 8.0.100-rc.1.23463.5 OS: macOS 12.6.8

For `win-x64`

.NET Version: 7.0.401 OS: Windows 11 23H2 22631.2338

Other information

Maybe using PLINQ in Native AOT is not currently supported after all. But I see no indication of so in documentations. If this is the case, then documentations should preferably be updated to reflect this.

ghost commented 1 year ago

Tagging subscribers to this area: @dotnet/area-system-linq-parallel See info in area-owners.md if you want to be subscribed.

Issue Details

### Description Calling PLINQ `.AsParallel().SelectMany()` in code results in the following Native AOT warnings being generated when publishing for `linux-x64`, `osx-arm64`, and `win-x64`: > ILC: Method '[System.Linq.Parallel]System.Linq.Parallel.SelectManyQueryOperator\`3.WrapPartitionedStreamNotIndexed,int32>,int32>,int32>>(PartitionedStream\`2,int32>,int32>,int32>>,IPartitionedStreamRecipient\`1,QuerySettings)' will always throw because: Failed to load type 'System.Linq.Parallel.PartitionedStream\`2, Int32>, Int32>, Int32>, Int32>>' from assembly 'System.Linq.Parallel, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' Actual run-time behavior seems fine in my scenario, but there can be complications that I did not catch. ### Reproduction Steps 1. Create a new project using console app template and configure for Native AOT deployment by adding `true` to .csproj. 2. Add the following reproduction code to Program.cs ```cs int counter = 0; Enumerable.Range(0, 5).AsParallel().SelectMany(x => Enumerable.Range(x * 10, 10).Select(x => { int order = Interlocked.Increment(ref counter); Console.WriteLine($"Begin {order}"); Thread.Sleep(Random.Shared.Next(1000)); // Simulate synchronous compute-intensive task. Console.WriteLine($"End {order}"); return order; })).ToArray(); ``` 3. Publish for Native AOT by running command `dotnet publish -c Release -r ` ### Expected behavior Publish completes without warnings. ### Actual behavior Warning messages in the description section being generated. However, running the generated binary produces no exceptions, and the output reveals that the queries are indeed being executed in parallel. ``` Begin 1 Begin 2 Begin 3 Begin 4 Begin 5 End 2 Begin 6 End 3 Begin 7 End 1 Begin 8 End 8 Begin 9 End 9 Begin 10 End 10 Begin 11 End 7 Begin 12 ... ``` ### Regression? Likely not a regression. Warnings are generated for both .NET 7 and .NET 8 RC 1, with the former being the first release supporting Native AOT deployment. ### Known Workarounds 1. Ignore the warning and test after publishing. Run-time behavior seems to be okay. 2. Remove or replace either `.AsParallel()` or `.SelectMany()` if possible. They publish fine when used alone. ### Configuration ### For `linux-x64` .NET Version: 7.0.111 OS: Ubuntu 22.04.3 on WSL 2 ### For `osx-arm64` .NET Version: 8.0.100-rc.1.23463.5 OS: macOS 12.6.8 ### For `win-x64` .NET Version: 7.0.401 OS: Windows 11 23H2 22631.2338 ### Other information Maybe using PLINQ in Native AOT is not currently supported after all. But I see no indication of so in documentations. If this is the case, then documentations should preferably be updated to reflect this.

Author:	DL444
Assignees:	-
Labels:	`area-System.Linq.Parallel`, `untriaged`
Milestone:	-

adamsitnik commented 1 year ago

I was able to reproduce it with .NET 8 RC2 bits:

PS D:\projects\repros\AotPLINQ> dotnet publish -c Release -r win-x64
MSBuild version 17.8.0-preview-23429-02+e9c5753b8 for .NET
  Determining projects to restore...
  Restored D:\projects\repros\AotPLINQ\AotPLINQ.csproj (in 229 ms).
C:\Program Files\dotnet\sdk\8.0.100-rc.2.23429.6\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInferenc
e.targets(311,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-polic
y [D:\projects\repros\AotPLINQ\AotPLINQ.csproj]
  AotPLINQ -> D:\projects\repros\AotPLINQ\bin\Release\net8.0\win-x64\AotPLINQ.dll
  Generating native code
  ILC: Method '[System.Linq.Parallel]System.Linq.Parallel.SelectManyQueryOperator`3<int32,int32,int32>.WrapPartitionedS
  treamNotIndexed<Pair`2<Pair`2<Pair`2<Pair`2<__Canon,int32>,int32>,int32>,int32>>(PartitionedStream`2<int32,Pair`2<Pai
  r`2<Pair`2<Pair`2<__Canon,int32>,int32>,int32>,int32>>,IPartitionedStreamRecipient`1<int32>,QuerySettings)' will alwa
  ys throw because: Failed to load type 'System.Linq.Parallel.IPartitionedStreamRecipient`1<Int32>' from assembly 'Syst
  em.Linq.Parallel, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
  ILC: Method '[System.Linq.Parallel]System.Linq.Parallel.SelectManyQueryOperator`3<int32,int32,int32>.WrapPartitionedS
  treamNotIndexed<Pair`2<Pair`2<Pair`2<Pair`2<int32,int32>,int32>,int32>,int32>>(PartitionedStream`2<int32,Pair`2<Pair`
  2<Pair`2<Pair`2<int32,int32>,int32>,int32>,int32>>,IPartitionedStreamRecipient`1<int32>,QuerySettings)' will always t
  hrow because: Failed to load type 'System.Linq.Parallel.PartitionedStream`2<Int32, System.Linq.Parallel.Pair`2<System
  .Linq.Parallel.Pair`2<System.Linq.Parallel.Pair`2<System.Linq.Parallel.Pair`2<System.Linq.Parallel.Pair`2<Int32, Int3
  2>, Int32>, Int32>, Int32>, Int32>>' from assembly 'System.Linq.Parallel, Version=8.0.0.0, Culture=neutral, PublicKey
  Token=b03f5f7f11d50a3a'
C:\Users\adsitnik\.nuget\packages\runtime.win-x64.microsoft.dotnet.ilcompiler\8.0.0-rc.2.23426.4\framework\System.Linq.
Parallel.dll : warning IL3053: Assembly 'System.Linq.Parallel' produced AOT analysis warnings. [D:\projects\repros\AotP
LINQ\AotPLINQ.csproj]
  AotPLINQ -> D:\projects\repros\AotPLINQ\bin\Release\net8.0\win-x64\publish\

@jeffhandley would a fix for that meet the .NET 8 RC2 bar? If so, do we have anyone with experience in this field who could provide a fix?

jeffhandley commented 1 year ago

This would not meet the bar for RC2 or GA. It's not a regression and we have not yet made an effort for PLINQ to be Native AOT compatible.

As far as I know, we've not gone through our docs to indicate which libraries have an have not been reviewed and updated for Native AOT compatibility, and we've instead relied on the warnings to provide the right guidance. As this document describes, "The call might be AOT-compatible now, but as you update your code, that might change, and you might forget to review all the suppressions."

DL444 commented 1 year ago

So, to sum it up, there would be no dedicated documentations on whether or not specific core library APIs are supported for Native AOT deployment. The warnings are to be considered the source of truth, and generation of such messages itself indicates that the relevant APIs are not AOT-ready.

Fortunately, this problem is relatively easy to work around in my code base, and I will go with that for now.

MichalStrehovsky commented 1 year ago

The infinite generic recursion issue is tracked in #92329 (I just moved the issue from runtimelab).

The actual warning is "warning IL3053: Assembly 'System.Linq.Parallel' produced AOT analysis warnings.". The ILC: Method '...' will always throw is just a diagnostic message that doesn't show up as a warning.

This is just a bug that we'll need to fix at some point. Similar generic recursions were successfully broken apart in the past (e.g. https://github.com/npgsql/npgsql/pull/4095).

dotnet / runtime