Infinite virtual generic method recursion in System.Linq.Parallel

[Bxaa]

Permanent loop with memory leak while compile :)

Dim Test_DataBase As New ConcurrentDictionary(Of String, Long)
Dim Test_Data As New ConcurrentBag(Of String)
Dim Test_Ilc As Object = Encoding.UTF8.GetBytes(Join(Test_DataBase.AsParallel.Select(Function(z) z.Key & "?" & z.Value).Concat(Test_Data.AsParallel.Select(Function(z) New FileInfo(z.Substring(z.IndexOf(">"c) + 1).Trim).FullName & "?" & New FileInfo(z.Substring(z.IndexOf(">"c) + 1).Trim).LastWriteTime.Ticks)).ToArray, "*"))

Test_Data.AsParallel.Select - This cause it :)

Just copy this strings to project :) This is just example...

[jkotas]

This is infinite virtual generic method instantion in System.Linq.Parallel. Here is the cycle:

BinaryQueryOperator<TLeftInput, TRightInput, TOutput>.WrapPartitionedStream<TLeftKey, TRightKey>
-> (GVM - derived method instantiations) ConcatQueryOperator<TSource>.WrapPartitionedStream<TLeftKey, TRightKey>
-> WrapHelper<TLeftKey, TRightKey>
-> WrapHelper2<TLeftKey, TRightKey>
-> IPartitionedStreamRecipient<TSource>.Receive<ConcatKey<TLeftKey, TRightKey>>
-> (GVM - ImplementingMethodInstantiation) BinaryQueryOperator.RightChildResultsRecipient<TLeftKey>.Receive<ConcatKey<TLeftKey, TRightKey> /* TRightKey */>
-> BinaryQueryOperator<TLeftInput, TRightInput, TOutput>.WrapPartitionedStream<TLeftKey, ConcatKey<TLeftKey, TRightKey> /* TRightKey */>

[kant2002]

This is minimal example of the problem

var x = new ImplementationOperator();
x.WrapPartitionedStream("");

abstract class BinaryQueryOperator
{
    public abstract void WrapPartitionedStream<TLeftKey>(TLeftKey key);
}

internal class ConcatKey<TLeftKey>
{
    internal readonly TLeftKey m_leftKey;

    internal ConcatKey(TLeftKey leftKey)
    {
        m_leftKey = leftKey;
    }
}

class ImplementationOperator : BinaryQueryOperator
{
    public override void WrapPartitionedStream<TLeftKey>(TLeftKey key)
    {
        var x = new ImplementationOperator();
        x.WrapPartitionedStream(new ConcatKey<TLeftKey>(key));
    }
}

Not sure how this should be solved.

[kant2002]

Isn't this is https://github.com/dotnet/runtimelab/issues/776 ?

[jkotas]

Not sure how this should be solved.

Redesign the algorithm to avoid the infinite generic recursion over valuetypes.

Isn't this is dotnet/runtimelab#776 ?

dotnet/runtimelab#776 is about detecting the infinite generic recursion in the compiler, and dealing with it transparently for recursion involving reference type instantiations, or failing the compilation with the error for value type instantiations.

[kant2002]

Redesign the algorithm to avoid the infinite generic recursion over valuetypes. A bit puzzled what algorithm should be re-designed? Somewhere in the System.Linq.Parallel or somewhere in the compiler?

If latter, then I have questions what's the difference. My guess is that difference in fact that currently infinite expansion happens in the Generic Virtual Method, where in dotnet/runtimelab#776 it happens during type instantiation. So for implementer this is two different issues. Please correct me if I'm wrong.

[jkotas]

Somewhere in the System.Linq.Parallel or somewhere in the compiler?

Somewhere in the System.LinqParallel.

[MichalStrehovsky]

The recursion is still there, but it doesn't crash the compiler when it's over reference types anymore (like in the example in the top post).

[ghost]

Tagging subscribers to this area: @dotnet/area-system-linq-parallel See info in area-owners.md if you want to be subscribed.

Issue Details

---

Permanent loop with memory leak while compile :) ``` Dim Test_DataBase As New ConcurrentDictionary(Of String, Long) Dim Test_Data As New ConcurrentBag(Of String) Dim Test_Ilc As Object = Encoding.UTF8.GetBytes(Join(Test_DataBase.AsParallel.Select(Function(z) z.Key & "?" & z.Value).Concat(Test_Data.AsParallel.Select(Function(z) New FileInfo(z.Substring(z.IndexOf(">"c) + 1).Trim).FullName & "?" & New FileInfo(z.Substring(z.IndexOf(">"c) + 1).Trim).LastWriteTime.Ticks)).ToArray, "*")) ``` `Test_Data.AsParallel.Select` - This cause it :) Just copy this strings to project :) This is just example...

Author:	Bxaa
Assignees:	-
Labels:	`area-System.Linq.Parallel`, `help wanted`, `untriaged`
Milestone:	-

[stealthin]

I'm getting those kind of errors as soon as I introduce AsParallel() in my code:

ILC: Method '[System.Linq.Parallel]System.Linq.Parallel.SelectManyQueryOperator`3<System.__Canon,System.__Canon,Syste
  m.__Canon>.WrapPartitionedStreamNotIndexed<Pair`2<Pair`2<Pair`2<Pair`2<__Canon,int32>,int32>,int32>,int32>>(Partition
  edStream`2<__Canon,Pair`2<Pair`2<Pair`2<Pair`2<__Canon,int32>,int32>,int32>,int32>>,IPartitionedStreamRecipient`1<__C
  anon>,QuerySettings)' will always throw because: Failed to load type 'System.Linq.Parallel.IPartitionedStreamRecipien
  t`1<System.__Canon>' from assembly 'System.Linq.Parallel, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11
  d50a3a'
  ILC: Method '[System.Linq.Parallel]System.Linq.Parallel.SelectManyQueryOperator`3<System.__Canon,System.__Canon,Syste
  m.__Canon>.WrapPartitionedStreamNotIndexed<Pair`2<Pair`2<Pair`2<Pair`2<int32,int32>,int32>,int32>,int32>>(Partitioned
  Stream`2<__Canon,Pair`2<Pair`2<Pair`2<Pair`2<int32,int32>,int32>,int32>,int32>>,IPartitionedStreamRecipient`1<__Canon
  >,QuerySettings)' will always throw because: Failed to load type 'System.Linq.Parallel.PartitionedStream`2<TOutput_Sy
  stem.__Canon, System.Linq.Parallel.Pair`2<System.Linq.Parallel.Pair`2<System.Linq.Parallel.Pair`2<System.Linq.Paralle
  l.Pair`2<System.Linq.Parallel.Pair`2<Int32, Int32>, Int32>, Int32>, Int32>, Int32>>' from assembly 'System.Linq.Paral
  lel, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'

I'm using Microsoft.DotNet.ILCompiler and runtime.win-x64.Microsoft.DotNet.ILCompiler (build version 9.0.0-alpha.1.23561.3).

As our LOB app relies heavily on parallelism, it kinda block migration to AOT. Do you have an ETA for being able to use PLinq with AOT?

[MichalStrehovsky]

I'm getting those kind of errors as soon as I introduce AsParallel() in my code:

These are just messages - they are not even warnings. Are you getting a build failure? If so, could you post the message?

This should at most result in generating IL3054 warning. The compilation will take longer than it should and the executable will be larger that it should, but it should work.

[stealthin]

will always throw because: Failed to load type 'System.Linq.Parallel.IPartitionedStreamRecipien t`1' from assembly 'System.Linq.Parallel, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11 d50a3a'

I was getting anxious about this message, are you telling this won't alter our production code while executing something like AsParallel().SelectMany(...) ?

[MichalStrehovsky]

will always throw because: Failed to load type 'System.Linq.Parallel.IPartitionedStreamRecipien t`1' from assembly 'System.Linq.Parallel, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11 d50a3a'

I was getting anxious about this message, are you telling this won't alter our production code while executing something like AsParallel().SelectMany(...) ?

The messages should eventually lead to IL3054 warning. The failure mode with that warning is described in the link in my previous comment. Our docs about warnings state: "If there are any AOT warnings, ensure there are no behavior changes by thoroughly testing your app after building as Native AOT."

There likely won't be an issue with it, but you should test. If there is an issue, it will manifest itself as a runtime exception (i.e. it will not be a data corruption or something terrible like that, just an exception).