StormHub
commented
6 years ago Can you try upgrade xunit and run again to find out? I vaguely remember saw something like that in the past
Closed yyjdelete closed 6 years ago
StormHub
commented
6 years ago Can you try upgrade xunit and run again to find out? I vaguely remember saw something like that in the past
yyjdelete
commented
6 years ago I can also reprouduct this with ConsoleApp and the same code. And I Found it's net472(preview) in my pc.
Seems when isClient == true, the client ask with tls12 and the server reply with tls10, which lead to the AuthenticationException.
An workaround is change
https://github.com/Azure/DotNetty/blob/1d3eda9a748125784efe2a74449bf7a18ab9c604/test/DotNetty.Handlers.Tests/SniHandlerTest.cs#L237
to something special the protocol
await Task.Run(() => driverStream.AuthenticateAsServerAsync(CertificateSelector(targetHost).Result.Certificate, false, protocol, false).WithTimeout(TimeSpan.FromSeconds(5)));
May be relate to THIS, and after I change the TargetFramework it works well.
***And seems it's only an test related issue, and not affact the main library*** ~~~, so leave it here to see if it will also happen with the stable version.~~~
---
Also found it timeout with `netcoreapp2.1`(preview), but no `AuthenticationException`.
An workaround is change
https://github.com/Azure/DotNetty/blob/1d3eda9a748125784efe2a74449bf7a18ab9c604/test/DotNetty.Handlers.Tests/SniHandlerTest.cs#L96
to
`await ReadOutboundAsync(async () => ch.ReadOutbound<IByteBuffer>(), output.Count - readResultBuffer.ReadableBytes, readResultBuffer, TimeSpan.FromSeconds(1));`
, which used by https://github.com/Azure/DotNetty/blob/1d3eda9a748125784efe2a74449bf7a18ab9c604/test/DotNetty.Handlers.Tests/SniHandlerTest.cs#L166-L172
that may ask more bytes to read than expected, and maybe the same thing for TlsHanderTest.Sorry I test with the wrong file, this also happen on net471/462.
Still not sure why this not happen on ci server, maybe it enable strong cryptography by default for safe?
StormHub
commented
6 years ago @yyjdelete Have you found anything yet? I think it might be in https://github.com/Azure/DotNetty/blob/dev/src/DotNetty.Handlers/Tls/TlsHandler.cs
caozhiyuan
commented
6 years ago netcoreapp2.1 has this issue , https://github.com/dotnet/corefx/issues/27675 you can run?
@yyjdelete
StormHub
commented
6 years ago @caozhiyuan I am not sure it’s related or not to the .net issue. For the threading problem, let me try to create some minimal repro.
yyjdelete
commented
6 years ago @caozhiyuan I'm not sure, but all unit tests pass with 2.1. What's the repro code and expected behavior with DotNetty?
And what's more, I use the daily preview builds 2.1.300-preview2-008293 instead of preview1 now.
Tested with Echo.Server(libuv=false), and see the same stack as you attached as .net core 2.0
caozhiyuan
commented
6 years ago 2.1 i run preview1 or daily build all dead lock @yyjdelete httpserver
caozhiyuan
commented
6 years ago @StormHub i just talk about 2.1, i don't konw this issue about.
yyjdelete
commented
6 years ago @caozhiyuan Ok, I can see it with VScode and dotnet run and attach, but not on VS 😕......
But Task does can be continue with TrySetResult before netcoreapp2.1, see https://github.com/Azure/DotNetty/issues/218#issuecomment-289268031
Seems TaskScheduler.TryExecuteTaskInline is not call in netcoreapp2.1
caozhiyuan
commented
6 years ago @yyjdelete #218 not this , it because 2.1 task continue , give TaskCreationOptions.RunContinuationsAsynchronously to TaskCompletionSource can work same as core 2.0(thread stack is same). 2.1 TrySetResult Continue in netty loop thread.
yyjdelete
commented
6 years ago using System;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
namespace ConsoleApp1
{
class Program
{
private static ExecutorTaskScheduler scheduler;
static void Main(string[] args)
{
//typeof(Task).GetField("s_asyncDebuggingEnabled", System.Reflection.BindingFlags.Static | System.Reflection.BindingFlags.NonPublic)?.SetValue(null, true);
scheduler = new ExecutorTaskScheduler();
MainAsync().Wait();
}
private static async Task MainAsync()
{
Console.WriteLine("MainAsync-Pre->" + Thread.CurrentThread.ManagedThreadId);
await X1();
Console.WriteLine("MainAsync-Post, and block->" + Thread.CurrentThread.ManagedThreadId);
//Console.WriteLine(Environment.StackTrace);
Console.ReadLine();
}
private static Task X1()
{
Console.WriteLine("X1->" + Thread.CurrentThread.ManagedThreadId);
var tmp = new TaskCompletionSource<int>();
Task.Factory.StartNew(() =>
{
Console.WriteLine("Task->" + Thread.CurrentThread.ManagedThreadId);
tmp.TrySetResult(0);
Console.WriteLine("Do something else.");
while (true)
{
Thread.Sleep(100);
}
}, CancellationToken.None, TaskCreationOptions.LongRunning, scheduler);
return tmp.Task;
}
public sealed class ExecutorTaskScheduler : TaskScheduler
{
protected override void QueueTask(Task task)
{
ThreadPool.QueueUserWorkItem((state) =>
{
this.TryExecuteTask(task);
});
}
protected override bool TryExecuteTaskInline(Task task, bool taskWasPreviouslyQueued)
{
return false;
}
protected override IEnumerable<Task> GetScheduledTasks() => null;
protected override bool TryDequeue(Task task) => false;
}
}
}https://github.com/Azure/DotNetty/blob/dev/src/DotNetty.Transport/Bootstrapping/AbstractBootstrap.cs#L232 this code , DoBind0Async promise trysetresult deadlock , you can copy netty code to core2.1 project
yyjdelete
commented
6 years ago @caozhiyuan I already make another repro code without DotNetty. You can run the code I attached with netcoreapp2.0 and 2.1, and see Do something else. is never show in 2.1 before press Enter.
caozhiyuan
commented
6 years ago because you change the code , dotnetty ExecutorTaskScheduler TryExecuteTask cause deaklock. but i can't reproduce in my minimal repro , in 2.0 TryExecuteTaskInline will not be called , 2.1 be called and cause deaklock, when add TaskCreationOptions.RunContinuationsAsynchronously to TaskCompletionSource , TryExecuteTaskInline will not be called
yyjdelete
commented
6 years ago @caozhiyuan But seems it's an internal behavior and may not run well in all runtimes. (for example, Mono < 4.0.0, version after 4.0 use the code from coreclr)
caozhiyuan
commented
6 years ago add TaskCreationOptions.RunContinuationsAsynchronously ?
yyjdelete
commented
6 years ago Not sure.
TaskCompletionSource and TaskCompletionSource<T>, all test in TlsHandleTest and End2End timeout...IdleStateHandlerTest.TestObserveWriterIdle failed for now the writeListener also run in async. See all TaskContinuationOptions.ExecuteSynchronously expect LinkOutcome.xunit use an custom SynchronizationContext so seem not be affacted by inline, but unknown why timeout.
caozhiyuan
commented
6 years ago 2.0 continuationObject is action , 2.1 is ITaskCompletionAction . add RunContinuationsAsynchronously and conditional build can solve this
private void RunContinuations(object continuationObject) // separated out of FinishContinuations to enable it to be inlined
{
Debug.Assert(continuationObject != null);
TplEtwProvider etw = TplEtwProvider.Log;
bool tplEtwProviderLoggingEnabled = etw.IsEnabled();
if (tplEtwProviderLoggingEnabled)
{
etw.RunningContinuation(Id, continuationObject);
}
if (AsyncCausalityTracer.LoggingOn)
AsyncCausalityTracer.TraceSynchronousWorkStart(CausalityTraceLevel.Required, this.Id, CausalitySynchronousWork.CompletionNotification);
// Skip synchronous execution of continuations if this task's thread was aborted
bool bCanInlineContinuations = !(((m_stateFlags & TASK_STATE_THREAD_WAS_ABORTED) != 0) ||
(RuntimeThread.CurrentThread.ThreadState == ThreadState.AbortRequested) ||
((m_stateFlags & (int)TaskCreationOptions.RunContinuationsAsynchronously) != 0));
// Handle the single-Action case
Action singleAction = continuationObject as Action;
if (singleAction != null)
{
AwaitTaskContinuation.RunOrScheduleAction(singleAction, bCanInlineContinuations, ref t_currentTask);
LogFinishCompletionNotification();
return;
}
// Handle the single-ITaskCompletionAction case
ITaskCompletionAction singleTaskCompletionAction = continuationObject as ITaskCompletionAction;
if (singleTaskCompletionAction != null)
{
if (bCanInlineContinuations || !singleTaskCompletionAction.InvokeMayRunArbitraryCode)
{
singleTaskCompletionAction.Invoke(this);
}
else
{
ThreadPool.UnsafeQueueCustomWorkItem(new CompletionActionInvoker(singleTaskCompletionAction, this), forceGlobal: false);
}
LogFinishCompletionNotification();
return;
}
// Handle the single-TaskContinuation case
TaskContinuation singleTaskContinuation = continuationObject as TaskContinuation;
if (singleTaskContinuation != null)
{
singleTaskContinuation.Run(this, bCanInlineContinuations);
LogFinishCompletionNotification();
return;
}
// Not a single; it must be a list.
List<object> continuations = (List<object>)continuationObject;
//
// Begin processing of continuation list
//
// Wait for any concurrent adds or removes to be retired
lock (continuations) { }
int continuationCount = continuations.Count;
// Fire the asynchronous continuations first ...
for (int i = 0; i < continuationCount; i++)
{
// Synchronous continuation tasks will have the ExecuteSynchronously option,
// and we're looking for asynchronous tasks...
var tc = continuations[i] as StandardTaskContinuation;
if (tc != null && (tc.m_options & TaskContinuationOptions.ExecuteSynchronously) == 0)
{
if (tplEtwProviderLoggingEnabled)
{
etw.RunningContinuationList(Id, i, tc);
}
continuations[i] = null; // so that we can skip this later
tc.Run(this, bCanInlineContinuations);
}
}
// ... and then fire the synchronous continuations (if there are any).
// This includes ITaskCompletionAction, AwaitTaskContinuations, and
// Action delegates, which are all by default implicitly synchronous.
for (int i = 0; i < continuationCount; i++)
{
object currentContinuation = continuations[i];
if (currentContinuation == null) continue;
continuations[i] = null; // to enable free'ing up memory earlier
if (tplEtwProviderLoggingEnabled)
{
etw.RunningContinuationList(Id, i, currentContinuation);
}
// If the continuation is an Action delegate, it came from an await continuation,
// and we should use AwaitTaskContinuation to run it.
Action ad = currentContinuation as Action;
if (ad != null)
{
AwaitTaskContinuation.RunOrScheduleAction(ad, bCanInlineContinuations, ref t_currentTask);
}
else
{
// If it's a TaskContinuation object of some kind, invoke it.
TaskContinuation tc = currentContinuation as TaskContinuation;
if (tc != null)
{
// We know that this is a synchronous continuation because the
// asynchronous ones have been weeded out
tc.Run(this, bCanInlineContinuations);
}
// Otherwise, it must be an ITaskCompletionAction, so invoke it.
else
{
Debug.Assert(currentContinuation is ITaskCompletionAction, "Expected continuation element to be Action, TaskContinuation, or ITaskContinuationAction");
var action = (ITaskCompletionAction)currentContinuation;
if (bCanInlineContinuations || !action.InvokeMayRunArbitraryCode)
{
action.Invoke(this);
}
else
{
ThreadPool.UnsafeQueueCustomWorkItem(new CompletionActionInvoker(action, this), forceGlobal: false);
}
}
}
}
LogFinishCompletionNotification();
}
I mean the test with net452, but my system is win10x64(17083 Insider Preview), so the runtime should be net471. And I doesn't see that with net452 on ci server.
Not sure happened, will try and attach some debug info latter.
BTW: Why the test for DotNetty.Buffers is 20 times slower in net452 than netcoreapp1.1?
Update:
TaskExtensions.WithTimeout(TimeSpan)and watch all FirstChanceException. 2.1 Seems it will never finish. 2.2 An FirstChanceException throw at AuthenticateAsServerAsync(Not throw to DotNetty's code) (Translate to English by Google) System.Security.Authentication.AuthenticationException: 'The call to SSPI failed, see internal exception. ' Internal exception: Win32Exception: The client and server can not communicate because of different algorithms May be related to https://github.com/dotnet/corefx/issues/26186 ?2.3 Another FirstChanceException throw at AuthenticateAsServerAsync(Not throw to DotNetty's code), almost the same as 2.2 (Translate to English by Google) System.Security.Authentication.AuthenticationException: 'The call to SSPI failed, see internal exception. ' Internal exception: Win32Exception: The client and server can not communicate because of different algorithms
2.4 The 3rd FirstChanceException, timeout, but won't stop the test.(Exception not set to MediationStream.BeginRead) Xunit.Sdk.TrueException: 'Did not reach expected state in time.'