How to diagnostic hanging Task

[patricksuo]

revisit dotnet/runtime#8860 . @mikem8361 fix code dump generation bug on Centos (#13887). But I still don't how to diagnostic Task hanging bug.

scenario: I use Task as lightweight thread to handle user request:

Task.Run(async() =>{
        while(true) {
                var data = await conn.ReadAsync();
                await someSemaSlim.WaitAsync();
                try {
                        // bz logic here.
                } 
                finally {
                        someSemaSlim.Release();
                }
        }
});

consider the follow buggy situation:

        SemaphoreSlim mSema1 = new SemaphoreSlim(1);
        SemaphoreSlim mSema2 = new SemaphoreSlim(0);
        SemaphoreSlim mSema3 = new SemaphoreSlim(0);

        Task.Run(async() =>{
                // logic a
                await mSema1.WaitAsync()
                // logic b
                await mSema2.WaitAsync(); // never finish
                // logic c
                await mSema3.WaitAsync();
                // logic d
        });

Is there a way to know 1) what Tasks are alive right now? 2) what status for each Task is? 3) for task blocking, what is it waiting for? and if yes 4) Is there already any tool/approach available on Linux/Mac ? 5) are they suitable for production?

[patricksuo]

using System;
using System.Threading;
using System.Threading.Tasks;

namespace ConsoleApp1
{
    class Program
    {
        static SemaphoreSlim mSema1 = new SemaphoreSlim(1);
        static SemaphoreSlim mSema2 = new SemaphoreSlim(0);
        static SemaphoreSlim mSema3 = new SemaphoreSlim(0);

        static void Main(string[] args)
        {
            Console.WriteLine("Hello World!");
            var t1 = Task.Run(BlockForTaskDelay);
            var t2 = Task.Run(BlockForSystemCall);
            var t3 = Task.Run(BlockForDeadlock);
            var t4 = Task.Run(RunningDummyWorkload);

            Task.WhenAll(t1, t2, t3, t4).Wait();
        }

        static async Task BlockForTaskDelay()
        {
            await Task.Delay(-1);
        }

        static async Task BlockForSystemCall()
        {
            await Task.Yield();
            Thread.Sleep(-1);
        }

        static async Task BlockForDeadlock()
        {
            await mSema1.WaitAsync();
            await mSema2.WaitAsync();
            await mSema3.WaitAsync();
        }

        static async Task RunningDummyWorkload()
        {
            await Task.Yield();
            while (true)
            {
                Thread.Yield();
            }
        }
    }
}

I can see something with VS. But not enough to know what BlockForDeadlock is waiting for.

[patricksuo]

Is it possible include Task info in minidump ? eg, Task id, status, blocking reason, blocking location

[patricksuo]

It seems no fix coming in near feature.

CC @stephentoub , could you give me some suggestions or point me to somewhere else? I can see you make significant contribution to Task library

[stephentoub]

It seems no fix coming in near feature.

Fix for what?

Is there a way to know

You can use sos to examine the heap. All of the information you're looking for is available there, in both live debugging and against a dump. When you find the Task object you're interested in, you can see its id, its status, etc., as they're just fields on the Task object. For async methods, you can find the state machine object and know exactly where it is by looking at its state field, which will tell you whether it's finished (-2), currently executing on some thread somewhere (-1), or at an await (the number of the await it's at, 0-indexed).

In your BlockForDeadlock example, you can use dumpheap -type BlockForDeadlock to find the associated state machine object on the heap. You can then look at that state machine object, and also do a gcroot on it... that gcroot will show you that it's being kept alive by the semaphore object it awaited.

[mattwarren]

Or if you want a slight more automated way, take a look at CLRMD, for example ClrMD Part 8 – Spelunking inside the .NET Thread Pool

[patricksuo]

You can use sos to examine the heap.

I missed the debugging-instructions . I will look into lldb and sos. Thank you.

Or if you want a slight more automated way, take a look at CLRMD, for example ClrMD Part 8 – Spelunking inside the .NET Thread Pool

I will look into this tool too. Thank you.