search

metacall / core

MetaCall: The ultimate polyglot programming experience.
https://metacall.io
Apache License 2.0
1.55k stars 160 forks source link

Livelock generated in test `metacall-node-python-await-extended-test` #519

Open viferga opened 1 month ago

viferga commented 1 month ago

🐛 Bug Report

This bug is being produced for no clear reason. For reproducing it, I have to run in parallel multiple tests with:

ctest --repeat until-fail:10000 -VV -R metacall-node-python-await-extended-test

This test happens when NodeJS tries to exit but there's async handles waiting to be executed, the behavior is strange because it is working as expected and suddently it enters into a livelock, blocking the process at 100% of CPU. It gets stuck at uv__async_io.

It is possible to stop and debug with GDB if you run the container with:

docker run --rm --privileged -it metacall/core:dev bash

The destroy hook for handling the handle counter seems to be what triggers this. But I am not sure if this happens due to abusing the async handles or because of the destroy mechanism itself.

Two options to improve this are: 1) Create a better way of handling async calls that does not create one async resource for each async call. We could implement this with a queue or similar and handle all the async calls with only one async resource. This may also improve the performance. 2) Debug in detail what is exactly being triggered from the destroy mechanism, and review the check or prepare hooks, then use only one that does not produce an interference with the async step in the libuv event loop.

Here are the logs of two runs of the livelock issue:

59: RESULT: 32 9607
59: RESULT: 32 9608
59: RESULT: 32 9609
59: RESULT: 32 9610
59: RESULT: 32 9611
59: RESULT: 32 9612
59: RESULT: 32 9613
59: RESULT: 32 9614
59: RESULT: 32 9615
59: RESULT: 32 9616
59: RESULT: 32 9617
59: RESULT: 32 9618
59: RESULT: 32 9619
59: RESULT: 32 9620
59: RESULT: 32 9621
59: RESULT: 32 9622
59: RESULT: 32 9623
59: RESULT: 32 9624
59: RESULT: 32 9625
59: RESULT: 32 9626
59: RESULT: 32 9627
59: RESULT: 32 9628
59: RESULT: 32 9629
59: RESULT: 32 9630
59: RESULT: 32 9631
59: RESULT: 32 9632
59: RESULT: 32 9633
59: RESULT: 32 9634
59: RESULT: 32 9635
59: RESULT: 32 9636
59: RESULT: 32 9637
59: RESULT: 32 9638
59: RESULT: 32 9639
59: RESULT: 32 9640
59: RESULT: 32 9641
59: RESULT: 32 9642
59: RESULT: 32 9643
59: RESULT: 32 9644
59: RESULT: 32 9645
59: RESULT: 32 9646
59: Number of active handles: 255
59: [active_handles] - [base_active_handles] - [extra_active_handles] + [active_reqs] + [closing]
59:        254        -           12          -            2           +    0 [> 0]    +    1
59: Number of user active handles: 241
59: [-AI] async    0x7f4bbec3cc70
59: [-A-] async    0x7f4bb005f4c0
59: [-A-] async    0x7f4bbec3b4e0
59: [-A-] check    0x7f4bb00cc0b0
59: [RA-] idle     0x7f4bb00cc128
59: [-A-] prepare  0x7f4bb00cc1a0
59: [-A-] check    0x7f4bb00cc218
59: [-A-] async    0x7f4bb00cc290
59: [RA-] async    0x7f4bb0124b68
59: [RA-] async    0x7f4bb00ee008
59: [RA-] async    0x7f4bb01e2308
59: [RA-] async    0x7f4bb01f2208
59: [RA-] async    0x7f4bb01f2438
59: [RA-] async    0x7f4bb01f2878
59: [RA-] async    0x7f4bb01e1898
59: [RA-] async    0x7f4bb01e1d28
59: [RA-] async    0x7f4bb01fff08
59: [RA-] async    0x7f4bb0200398
59: [RA-] async    0x7f4bb02008e8
59: [RA-] async    0x7f4bb0200bc8
59: [RA-] async    0x7f4bb0b17ff0
59: [RA-] async    0x7f4bb0a4f5f0
59: [RA-] async    0x7f4bb0a4f0b0
59: [RA-] async    0x7f4bb0a4eb20
59: [RA-] async    0x7f4bb0a4ec70
59: [RA-] async    0x7f4bb0a531d0
59: [RA-] async    0x7f4bb0a55440
59: [RA-] async    0x7f4bb0a54f20
59: [RA-] async    0x7f4bb0a54a20
59: [RA-] async    0x7f4bb0a59520
59: [RA-] async    0x7f4bb0a59020
59: [RA-] async    0x7f4bb0a58b20
59: [RA-] async    0x7f4bb0a58620
59: [RA-] async    0x7f4bb0a5cc20
59: [RA-] async    0x7f4bb0a01970
59: [RA-] async    0x7f4bb0a01a50
59: [RA-] async    0x7f4bb0a09410
59: [RA-] async    0x7f4bb0a940c0
59: [RA-] async    0x7f4bb0a94500
59: [RA-] async    0x7f4bb0a949e0
59: [RA-] async    0x7f4bb0a94ee0
59: [RA-] async    0x7f4bb0a94fc0
59: [RA-] async    0x7f4bb0a953e0
59: [RA-] async    0x7f4bb0a958e0
59: [RA-] async    0x7f4bb0a95de0
59: [RA-] async    0x7f4bb0a98ae0
59: [RA-] async    0x7f4bb0a98bc0
59: [RA-] async    0x7f4bb0a98fe0
59: [RA-] async    0x7f4bb0a994e0
59: [RA-] async    0x7f4bb0a999e0
59: [RA-] async    0x7f4bb0a99ee0
59: [RA-] async    0x7f4bb0a9a3e0
59: [RA-] async    0x7f4bb0a9a8e0
59: [RA-] async    0x7f4bb0a9d5e0
59: [RA-] async    0x7f4bb0a9dae0
59: [RA-] async    0x7f4bb0a9dfe0
59: [RA-] async    0x7f4bb0a9e4e0
59: [RA-] async    0x7f4bb0a9e9e0
59: [RA-] async    0x7f4bb0a9eee0
59: [RA-] async    0x7f4bb0a9f3e0
59: [RA-] async    0x7f4bb0a9f8e0
59: [RA-] async    0x7f4bb0aa25e0
59: [RA-] async    0x7f4bb0aa2ae0
59: [RA-] async    0x7f4bb0aa2fe0
59: [RA-] async    0x7f4bb0aa34e0
59: [RA-] async    0x7f4bb0b180d0
59: [RA-] async    0x7f4bb0b18190
59: [RA-] async    0x7f4bb0b18250
59: [RA-] async    0x7f4bb0b18310
59: [RA-] async    0x7f4bb0b18620
59: [RA-] async    0x7f4bb0b189a0
59: [RA-] async    0x7f4bb0b18e60
59: [RA-] async    0x7f4bb0b19360
59: [RA-] async    0x7f4bb0b19860
59: [RA-] async    0x7f4bb0b19d60
59: [RA-] async    0x7f4bb0b1a260
59: [RA-] async    0x7f4bb0b1a760
59: [RA-] async    0x7f4bb0b1ac60
59: [RA-] async    0x7f4bb0b1b160
59: [RA-] async    0x7f4bb0b1b660
59: [RA-] async    0x7f4bb0b1bb60
59: [RA-] async    0x7f4bb0b1c060
59: [RA-] async    0x7f4bb0b1c560
59: [RA-] async    0x7f4bb0b1ca60
59: [RA-] async    0x7f4bb0b1cf60
59: [RA-] async    0x7f4bb0b1d460
59: [RA-] async    0x7f4bb0b1d960
59: [RA-] async    0x7f4bb0b1de60
59: [RA-] async    0x7f4bb0b1e360
59: [RA-] async    0x7f4bb0b1e860
59: [RA-] async    0x7f4bb0b1ed60
59: [RA-] async    0x7f4bb0b1f260
59: [RA-] async    0x7f4bb0b217c0
59: [RA-] async    0x7f4bb0b21cc0
59: [RA-] async    0x7f4bb0b221c0
59: [RA-] async    0x7f4bb0b226c0
59: [RA-] async    0x7f4bb0b22bc0
59: [RA-] async    0x7f4bb0b230c0
59: [RA-] async    0x7f4bb0b235c0
59: [RA-] async    0x7f4bb0b23ac0
59: [RA-] async    0x7f4bb0b23fc0
59: [RA-] async    0x7f4bb0b244c0
59: [RA-] async    0x7f4bb0b249c0
59: [RA-] async    0x7f4bb0b24ec0
59: [RA-] async    0x7f4bb0b253c0
59: [RA-] async    0x7f4bb0b258c0
59: [RA-] async    0x7f4bb0b25dc0
59: [RA-] async    0x7f4bb0b262c0
59: [RA-] async    0x7f4bb0b267c0
59: [RA-] async    0x7f4bb0b26cc0
59: [RA-] async    0x7f4bb0b271c0
59: [RA-] async    0x7f4bb0aa3b30
59: [RA-] async    0x7f4bb0b276a0
59: [RA-] async    0x7f4bb0b27ba0
59: [RA-] async    0x7f4bb0b280a0
59: [RA-] async    0x7f4bb0b285a0
59: [RA-] async    0x7f4bb0b28aa0
59: [RA-] async    0x7f4bb0b28fa0
59: [RA-] async    0x7f4bb0b294a0
59: [RA-] async    0x7f4bb0b299a0
59: [RA-] async    0x7f4bb0b29ea0
59: [RA-] async    0x7f4bb0b2a3a0
59: [RA-] async    0x7f4bb0b2a8a0
59: [RA-] async    0x7f4bb0b2ada0
59: [RA-] async    0x7f4bb0b2b2a0
59: [RA-] async    0x7f4bb0b2b7a0
59: [RA-] async    0x7f4bb0b2bca0
59: [RA-] async    0x7f4bb0b2c1a0
59: [RA-] async    0x7f4bb0b2c6a0
59: [RA-] async    0x7f4bb0aa6ea0
59: [RA-] async    0x7f4bb0aa7360
59: [RA-] async    0x7f4bb0aa77d0
59: [RA-] async    0x7f4bb0aa7cd0
59: [RA-] async    0x7f4bb0aa81d0
59: [RA-] async    0x7f4bb0aa86d0
59: [RA-] async    0x7f4bb0aa8bd0
59: [RA-] async    0x7f4bb0aa90d0
59: [RA-] async    0x7f4bb0b2c780
59: [RA-] async    0x7f4bb0b2c840
59: [RA-] async    0x7f4bb0b2c900
59: [RA-] async    0x7f4bb0b2c9c0
59: [RA-] async    0x7f4bb0b2ccd0
59: [RA-] async    0x7f4bb0b2d090
59: [RA-] async    0x7f4bb0b2d410
59: [RA-] async    0x7f4bb0b2d790
59: [RA-] async    0x7f4bb09ed040
59: [RA-] async    0x7f4bb0b2db10
59: [RA-] async    0x7f4bb0b2dfd0
59: [RA-] async    0x7f4bb0b2e4d0
59: [RA-] async    0x7f4bb0b2e9d0
59: [RA-] async    0x7f4bb0b2eed0
59: [RA-] async    0x7f4bb0b2f3d0
59: [RA-] async    0x7f4bb0b2f8d0
59: [RA-] async    0x7f4bb0b2fdd0
59: [RA-] async    0x7f4bb0b302d0
59: [RA-] async    0x7f4bb0b307d0
59: [RA-] async    0x7f4bb0b30980
59: [RA-] async    0x7f4bb0b30b30
59: [RA-] async    0x7f4bb0aaae40
59: [RA-] async    0x7f4bb0aab190
59: [RA-] async    0x7f4bb0b30c10
59: [RA-] async    0x7f4bb0b30da0
59: [RA-] async    0x7f4bb0b31050
59: [RA-] async    0x7f4bb0b31300
59: [RA-] async    0x7f4bb0b31800
59: [RA-] async    0x7f4bb0b31d00
59: [RA-] async    0x7f4bb0b32200
59: [RA-] async    0x7f4bb0b32700
59: [RA-] async    0x7f4bb0b328b0
59: [RA-] async    0x7f4bb0b32a60
59: [RA-] async    0x7f4bb0b32c10
59: [RA-] async    0x7f4bb0b33090
59: [RA-] async    0x7f4bb0b33590
59: [RA-] async    0x7f4bb0b33a90
59: [RA-] async    0x7f4bb0b33f90
59: [RA-] async    0x7f4bb0b34490
59: [RA-] async    0x7f4bb0b34990
59: [RA-] async    0x7f4bb0b34e90
59: [RA-] async    0x7f4bb0b35390
59: [RA-] async    0x7f4bb0b35890
59: [RA-] async    0x7f4bb0b35d90
59: [RA-] async    0x7f4bb0b36290
59: [RA-] async    0x7f4bb0b36790
59: [RA-] async    0x7f4bb0b36c90
59: [RA-] async    0x7f4bb0b37190
59: [RA-] async    0x7f4bb0b37690
59: [RA-] async    0x7f4bb0b37b90
59: [RA-] async    0x7f4bb0b38090
59: [RA-] async    0x7f4bb0b38590
59: [RA-] async    0x7f4bb0b38a90
59: [RA-] async    0x7f4bb0b38f90
59: [RA-] async    0x7f4bb0b39490
59: [RA-] async    0x7f4bb0b39990
59: [RA-] async    0x7f4bb0b39e90
59: [RA-] async    0x7f4bb0b3a390
59: [RA-] async    0x7f4bb0b3a790
59: [RA-] async    0x7f4bb0b3abd0
59: [RA-] async    0x7f4bb0b3b0d0
59: [RA-] async    0x7f4bb0b3b5d0
59: [RA-] async    0x7f4bb0b3bad0
59: [RA-] async    0x7f4bb0b3bf10
59: [RA-] async    0x7f4bb0b3c410
59: [RA-] async    0x7f4bb0b3c8d0
59: [RA-] async    0x7f4bb0b3cdd0
59: [RA-] async    0x7f4bb0b3d2d0
59: [RA-] async    0x7f4bb0b3d7d0
59: [RA-] async    0x7f4bb0b3dcd0
59: [RA-] async    0x7f4bb0b3e1d0
59: [RA-] async    0x7f4bb0b3e6d0
59: [RA-] async    0x7f4bb0b3ebd0
59: [RA-] async    0x7f4bb0b3efd0
59: [RA-] async    0x7f4bb0b3f490
59: [RA-] async    0x7f4bb0b3f990
59: [RA-] async    0x7f4bb0b3fe90
59: [RA-] async    0x7f4bb0b40390
59: [RA-] async    0x7f4bb0b40890
59: [RA-] async    0x7f4bb0b40d90
59: [RA-] async    0x7f4bb0b41290
59: [RA-] async    0x7f4bb0b41790
59: [RA-] async    0x7f4bb0b41c90
59: [RA-] async    0x7f4bb0b441f0
59: [RA-] async    0x7f4bb0b446f0
59: [RA-] async    0x7f4bb0b44bf0
59: [RA-] async    0x7f4bb0b450f0
59: [RA-] async    0x7f4bb0b455f0
59: [RA-] async    0x7f4bb0b45af0
59: [RA-] async    0x7f4bb0b45ff0
59: [RA-] async    0x7f4bb0b464f0
59: [RA-] async    0x7f4bb0b469f0
59: [RA-] async    0x7f4bb0b46ef0
59: [RA-] async    0x7f4bb0b473f0
59: [RA-] async    0x7f4bb0b478f0
59: [RA-] async    0x7f4bb0b47df0
59: [RA-] async    0x7f4bb0b482f0
59: [RA-] async    0x7f4bb0b487f0
59: [RA-] async    0x7f4bb0b48cf0
59: [RA-] async    0x7f4bb0b491f0
59: [RA-] async    0x7f4bb0b496f0
59: [RA-] async    0x7f4bb0b49bf0
59: [RA-] async    0x7f4bb0b4a0f0
59: [RA-] async    0x7f4bb0b4a5f0
59: [RA-] async    0x7f4bb0b4aaf0
59: [RA-] async    0x7f4bb0b4aff0
59: [RA-] prepare  0x5565d509b780
59: [RA-] check    0x5565d509b7f8
59: [RA-] async    0x7f4bb0b4b4f0
59: [RA-] async    0x7f4bb0b4b970
59: [RA-] async    0x7f4bb0b4be70
59: [RA-] async    0x7f4bb0b4c370
59: [RA-] async    0x7f4bb0b4c870
59: [RA-] async    0x7f4bb0b4cd70
59: [RA-] async    0x7f4bb0b4d270
59: [RA-] async    0x7f4bb0b4d770
59: [RA-] async    0x7f4bb0b4dc70
59: [RA-] async    0x7f4bb0b4de20
59: [RA-] async    0x7f4bb0b4dfd0
59: [active_handles] - [base_active_handles] - [extra_active_handles] + [active_reqs] + [closing]
59:        254        -           12          -            2           +    0 [> 0]    +    1
59: RESULT: 32 9647
59: RESULT: 32 9648
59: RESULT: 32 9649
59: RESULT: 32 9650
59: RESULT: 32 9651
59: RESULT: 32 9652
59: RESULT: 32 9653
59: RESULT: 32 9654
59: RESULT: 32 9655
59: RESULT: 32 9656
59: RESULT: 32 9657
59: RESULT: 32 9658
59: RESULT: 32 9659
59: RESULT: 32 9660
59: RESULT: 32 9661
59: RESULT: 32 9662
59: RESULT: 32 9663
59: RESULT: 32 9664
59: RESULT: 32 9665
59: RESULT: 32 9666
59: RESULT: 32 9667
59: RESULT: 32 9668
59: RESULT: 32 9669
59: RESULT: 32 9670
59: RESULT: 32 9671
59: RESULT: 32 9672
59: RESULT: 32 9673
59: RESULT: 32 9674
59: RESULT: 32 9675
59: RESULT: 32 9676
59: RESULT: 32 9677
59: RESULT: 32 9678
59: RESULT: 32 9679
59: RESULT: 32 9680
59: RESULT: 32 9681
59: RESULT: 32 9682
59: RESULT: 32 9683
59: RESULT: 32 9684
59: RESULT: 32 9685
59: RESULT: 32 9686
59: RESULT: 32 9687
59: RESULT: 32 9688
59: RESULT: 32 9689
59: RESULT: 32 9690
59: RESULT: 32 9691
59: RESULT: 32 9692
59: RESULT: 32 9693
59: RESULT: 32 9694
59: RESULT: 32 9695
59: RESULT: 32 9696
59: RESULT: 32 9697
59: RESULT: 32 9698
59: RESULT: 32 9699
59: RESULT: 32 9700
59: RESULT: 32 9701
59: RESULT: 32 9702
59: RESULT: 32 9703
59: RESULT: 32 9704
59: RESULT: 32 9705
59: RESULT: 32 9706
59: RESULT: 32 9707
59: RESULT: 32 9708
59: RESULT: 32 9709
59: RESULT: 32 9710
59: RESULT: 32 9711
59: RESULT: 32 9712
59: RESULT: 32 9713
59: RESULT: 32 9714
59: RESULT: 32 9715
59: RESULT: 32 9716
59: RESULT: 32 9717
59: RESULT: 32 9718
59: RESULT: 32 9719
59: RESULT: 32 9720
59: RESULT: 32 9721
59: RESULT: 32 9722
59: RESULT: 32 9723
59: RESULT: 32 9724
59: RESULT: 32 9725
59: RESULT: 32 9726
59: RESULT: 32 9727
59: RESULT: 32 9728
59: RESULT: 32 9729
59: RESULT: 32 9730
59: RESULT: 32 9731
59: RESULT: 32 9732
59: RESULT: 32 9733
59: RESULT: 32 9734
59: RESULT: 32 9735
59: RESULT: 32 9736
59: RESULT: 32 9737
59: RESULT: 32 9738
59: RESULT: 32 9739
59: RESULT: 32 9740
59: RESULT: 32 9741
59: RESULT: 32 9742
59: RESULT: 32 9743
59: RESULT: 32 9744
59: RESULT: 32 9745
59: RESULT: 32 9746
59: RESULT: 32 9747
59: RESULT: 32 9748
59: RESULT: 32 9749
59: RESULT: 32 9750
59: RESULT: 32 9751
59: RESULT: 32 9752
59: RESULT: 32 9753
59: RESULT: 32 9754
59: RESULT: 32 9755
59: RESULT: 32 9756
59: RESULT: 32 9757
59: RESULT: 32 9758
59: RESULT: 32 9759
59: RESULT: 32 9760
59: RESULT: 32 9761
59: RESULT: 32 9762
59: RESULT: 32 9763
59: RESULT: 32 9764
59: RESULT: 32 9765
59: Number of active handles: 253
59: [active_handles] - [base_active_handles] - [extra_active_handles] + [active_reqs] + [closing]
59:        253        -           12          -            2           +    0 [> 0]    +    0
59: Number of user active handles: 239
59: [-AI] async    0x7f4bbec3cc70
59: [-A-] async    0x7f4bb005f4c0
59: [-A-] async    0x7f4bbec3b4e0
59: [-A-] check    0x7f4bb00cc0b0
59: [-A-] prepare  0x7f4bb00cc1a0
59: [-A-] check    0x7f4bb00cc218
59: [-A-] async    0x7f4bb00cc290
59: [RA-] async    0x7f4bb0124b68
59: [RA-] async    0x7f4bb00ee008
59: [RA-] async    0x7f4bb01e2308
59: [RA-] async    0x7f4bb01f2208
59: [RA-] async    0x7f4bb01f2438
59: [RA-] async    0x7f4bb01f2878
59: [RA-] async    0x7f4bb01e1898
59: [RA-] async    0x7f4bb01e1d28
59: [RA-] async    0x7f4bb01fff08
59: [RA-] async    0x7f4bb0200398
59: [RA-] async    0x7f4bb02008e8
59: [RA-] async    0x7f4bb0200bc8
59: [RA-] async    0x7f4bb0b17ff0
59: [RA-] async    0x7f4bb0a4f5f0
59: [RA-] async    0x7f4bb0a4f0b0
59: [RA-] async    0x7f4bb0a4eb20
59: [RA-] async    0x7f4bb0a4ec70
59: [RA-] async    0x7f4bb0a531d0
59: [RA-] async    0x7f4bb0a55440
59: [RA-] async    0x7f4bb0a54f20
59: [RA-] async    0x7f4bb0a54a20
59: [RA-] async    0x7f4bb0a59520
59: [RA-] async    0x7f4bb0a59020
59: [RA-] async    0x7f4bb0a58b20
59: [RA-] async    0x7f4bb0a58620
59: [RA-] async    0x7f4bb0a5cc20
59: [RA-] async    0x7f4bb0a01970
59: [RA-] async    0x7f4bb0a01a50
59: [RA-] async    0x7f4bb0a09410
59: [RA-] async    0x7f4bb0a940c0
59: [RA-] async    0x7f4bb0a94500
59: [RA-] async    0x7f4bb0a949e0
59: [RA-] async    0x7f4bb0a94ee0
59: [RA-] async    0x7f4bb0a94fc0
59: [RA-] async    0x7f4bb0a953e0
59: [RA-] async    0x7f4bb0a958e0
59: [RA-] async    0x7f4bb0a95de0
59: [RA-] async    0x7f4bb0a98ae0
59: [RA-] async    0x7f4bb0a98bc0
59: [RA-] async    0x7f4bb0a98fe0
59: [RA-] async    0x7f4bb0a994e0
59: [RA-] async    0x7f4bb0a999e0
59: [RA-] async    0x7f4bb0a99ee0
59: [RA-] async    0x7f4bb0a9a3e0
59: [RA-] async    0x7f4bb0a9a8e0
59: [RA-] async    0x7f4bb0a9d5e0
59: [RA-] async    0x7f4bb0a9dae0
59: [RA-] async    0x7f4bb0a9dfe0
59: [RA-] async    0x7f4bb0a9e4e0
59: [RA-] async    0x7f4bb0a9e9e0
59: [RA-] async    0x7f4bb0a9eee0
59: [RA-] async    0x7f4bb0a9f3e0
59: [RA-] async    0x7f4bb0a9f8e0
59: [RA-] async    0x7f4bb0aa25e0
59: [RA-] async    0x7f4bb0aa2ae0
59: [RA-] async    0x7f4bb0aa2fe0
59: [RA-] async    0x7f4bb0aa34e0
59: [RA-] async    0x7f4bb0b180d0
59: [RA-] async    0x7f4bb0b18190
59: [RA-] async    0x7f4bb0b18250
59: [RA-] async    0x7f4bb0b18310
59: [RA-] async    0x7f4bb0b18620
59: [RA-] async    0x7f4bb0b189a0
59: [RA-] async    0x7f4bb0b18e60
59: [RA-] async    0x7f4bb0b19360
59: [RA-] async    0x7f4bb0b19860
59: [RA-] async    0x7f4bb0b19d60
59: [RA-] async    0x7f4bb0b1a260
59: [RA-] async    0x7f4bb0b1a760
59: [RA-] async    0x7f4bb0b1ac60
59: [RA-] async    0x7f4bb0b1b160
59: [RA-] async    0x7f4bb0b1b660
59: [RA-] async    0x7f4bb0b1bb60
59: [RA-] async    0x7f4bb0b1c060
59: [RA-] async    0x7f4bb0b1c560
59: [RA-] async    0x7f4bb0b1ca60
59: [RA-] async    0x7f4bb0b1cf60
59: [RA-] async    0x7f4bb0b1d460
59: [RA-] async    0x7f4bb0b1d960
59: [RA-] async    0x7f4bb0b1de60
59: [RA-] async    0x7f4bb0b1e360
59: [RA-] async    0x7f4bb0b1e860
59: [RA-] async    0x7f4bb0b1ed60
59: [RA-] async    0x7f4bb0b1f260
59: [RA-] async    0x7f4bb0b217c0
59: [RA-] async    0x7f4bb0b21cc0
59: [RA-] async    0x7f4bb0b221c0
59: [RA-] async    0x7f4bb0b226c0
59: [RA-] async    0x7f4bb0b22bc0
59: [RA-] async    0x7f4bb0b230c0
59: [RA-] async    0x7f4bb0b235c0
59: [RA-] async    0x7f4bb0b23ac0
59: [RA-] async    0x7f4bb0b23fc0
59: [RA-] async    0x7f4bb0b244c0
59: [RA-] async    0x7f4bb0b249c0
59: [RA-] async    0x7f4bb0b24ec0
59: [RA-] async    0x7f4bb0b253c0
59: [RA-] async    0x7f4bb0b258c0
59: [RA-] async    0x7f4bb0b25dc0
59: [RA-] async    0x7f4bb0b262c0
59: [RA-] async    0x7f4bb0b267c0
59: [RA-] async    0x7f4bb0b26cc0
59: [RA-] async    0x7f4bb0b271c0
59: [RA-] async    0x7f4bb0aa3b30
59: [RA-] async    0x7f4bb0b276a0
59: [RA-] async    0x7f4bb0b27ba0
59: [RA-] async    0x7f4bb0b280a0
59: [RA-] async    0x7f4bb0b285a0
59: [RA-] async    0x7f4bb0b28aa0
59: [RA-] async    0x7f4bb0b28fa0
59: [RA-] async    0x7f4bb0b294a0
59: [RA-] async    0x7f4bb0b299a0
59: [RA-] async    0x7f4bb0b29ea0
59: [RA-] async    0x7f4bb0b2a3a0
59: [RA-] async    0x7f4bb0b2a8a0
59: [RA-] async    0x7f4bb0b2ada0
59: [RA-] async    0x7f4bb0b2b2a0
59: [RA-] async    0x7f4bb0b2b7a0
59: [RA-] async    0x7f4bb0b2bca0
59: [RA-] async    0x7f4bb0b2c1a0
59: [RA-] async    0x7f4bb0b2c6a0
59: [RA-] async    0x7f4bb0aa6ea0
59: [RA-] async    0x7f4bb0aa7360
59: [RA-] async    0x7f4bb0aa77d0
59: [RA-] async    0x7f4bb0aa7cd0
59: [RA-] async    0x7f4bb0aa81d0
59: [RA-] async    0x7f4bb0aa86d0
59: [RA-] async    0x7f4bb0aa8bd0
59: [RA-] async    0x7f4bb0aa90d0
59: [RA-] async    0x7f4bb0b2c780
59: [RA-] async    0x7f4bb0b2c840
59: [RA-] async    0x7f4bb0b2c900
59: [RA-] async    0x7f4bb0b2c9c0
59: [RA-] async    0x7f4bb0b2ccd0
59: [RA-] async    0x7f4bb0b2d090
59: [RA-] async    0x7f4bb0b2d410
59: [RA-] async    0x7f4bb0b2d790
59: [RA-] async    0x7f4bb09ed040
59: [RA-] async    0x7f4bb0b2db10
59: [RA-] async    0x7f4bb0b2dfd0
59: [RA-] async    0x7f4bb0b2e4d0
59: [RA-] async    0x7f4bb0b2e9d0
59: [RA-] async    0x7f4bb0b2eed0
59: [RA-] async    0x7f4bb0b2f3d0
59: [RA-] async    0x7f4bb0b2f8d0
59: [RA-] async    0x7f4bb0b2fdd0
59: [RA-] async    0x7f4bb0b302d0
59: [RA-] async    0x7f4bb0b307d0
59: [RA-] async    0x7f4bb0b30980
59: [RA-] async    0x7f4bb0b30b30
59: [RA-] async    0x7f4bb0aaae40
59: [RA-] async    0x7f4bb0aab190
59: [RA-] async    0x7f4bb0b30c10
59: [RA-] async    0x7f4bb0b30da0
59: [RA-] async    0x7f4bb0b31050
59: [RA-] async    0x7f4bb0b31300
59: [RA-] async    0x7f4bb0b31800
59: [RA-] async    0x7f4bb0b31d00
59: [RA-] async    0x7f4bb0b32200
59: [RA-] async    0x7f4bb0b32700
59: [RA-] async    0x7f4bb0b328b0
59: [RA-] async    0x7f4bb0b32a60
59: [RA-] async    0x7f4bb0b32c10
59: [RA-] async    0x7f4bb0b33090
59: [RA-] async    0x7f4bb0b33590
59: [RA-] async    0x7f4bb0b33a90
59: [RA-] async    0x7f4bb0b33f90
59: [RA-] async    0x7f4bb0b34490
59: [RA-] async    0x7f4bb0b34990
59: [RA-] async    0x7f4bb0b34e90
59: [RA-] async    0x7f4bb0b35390
59: [RA-] async    0x7f4bb0b35890
59: [RA-] async    0x7f4bb0b35d90
59: [RA-] async    0x7f4bb0b36290
59: [RA-] async    0x7f4bb0b36790
59: [RA-] async    0x7f4bb0b36c90
59: [RA-] async    0x7f4bb0b37190
59: [RA-] async    0x7f4bb0b37690
59: [RA-] async    0x7f4bb0b37b90
59: [RA-] async    0x7f4bb0b38090
59: [RA-] async    0x7f4bb0b38590
59: [RA-] async    0x7f4bb0b38a90
59: [RA-] async    0x7f4bb0b38f90
59: [RA-] async    0x7f4bb0b39490
59: [RA-] async    0x7f4bb0b39990
59: [RA-] async    0x7f4bb0b39e90
59: [RA-] async    0x7f4bb0b3a390
59: [RA-] async    0x7f4bb0b3a790
59: [RA-] async    0x7f4bb0b3abd0
59: [RA-] async    0x7f4bb0b3b0d0
59: [RA-] async    0x7f4bb0b3b5d0
59: [RA-] async    0x7f4bb0b3bad0
59: [RA-] async    0x7f4bb0b3bf10
59: [RA-] async    0x7f4bb0b3c410
59: [RA-] async    0x7f4bb0b3c8d0
59: [RA-] async    0x7f4bb0b3cdd0
59: [RA-] async    0x7f4bb0b3d2d0
59: [RA-] async    0x7f4bb0b3d7d0
59: [RA-] async    0x7f4bb0b3dcd0
59: [RA-] async    0x7f4bb0b3e1d0
59: [RA-] async    0x7f4bb0b3e6d0
59: [RA-] async    0x7f4bb0b3ebd0
59: [RA-] async    0x7f4bb0b3efd0
59: [RA-] async    0x7f4bb0b3f490
59: [RA-] async    0x7f4bb0b3f990
59: [RA-] async    0x7f4bb0b3fe90
59: [RA-] async    0x7f4bb0b40390
59: [RA-] async    0x7f4bb0b40890
59: [RA-] async    0x7f4bb0b40d90
59: [RA-] async    0x7f4bb0b41290
59: [RA-] async    0x7f4bb0b41790
59: [RA-] async    0x7f4bb0b41c90
59: [RA-] async    0x7f4bb0b441f0
59: [RA-] async    0x7f4bb0b446f0
59: [RA-] async    0x7f4bb0b44bf0
59: [RA-] async    0x7f4bb0b450f0
59: [RA-] async    0x7f4bb0b455f0
59: [RA-] async    0x7f4bb0b45af0
59: [RA-] async    0x7f4bb0b45ff0
59: [RA-] async    0x7f4bb0b464f0
59: [RA-] async    0x7f4bb0b469f0
59: [RA-] async    0x7f4bb0b46ef0
59: [RA-] async    0x7f4bb0b473f0
59: [RA-] async    0x7f4bb0b478f0
59: [RA-] async    0x7f4bb0b47df0
59: [RA-] async    0x7f4bb0b482f0
59: [RA-] async    0x7f4bb0b487f0
59: [RA-] async    0x7f4bb0b48cf0
59: [RA-] async    0x7f4bb0b491f0
59: [RA-] async    0x7f4bb0b496f0
59: [RA-] async    0x7f4bb0b49bf0
59: [RA-] async    0x7f4bb0b4a0f0
59: [RA-] async    0x7f4bb0b4a5f0
59: [RA-] async    0x7f4bb0b4aaf0
59: [RA-] async    0x7f4bb0b4aff0
59: [RA-] prepare  0x5565d509b780
59: [RA-] check    0x5565d509b7f8
59: [RA-] async    0x7f4bb0b4b4f0
59: [RA-] async    0x7f4bb0b4b970
59: [RA-] async    0x7f4bb0b4be70
59: [RA-] async    0x7f4bb0b4c370
59: [RA-] async    0x7f4bb0b4c870
59: [RA-] async    0x7f4bb0b4cd70
59: [RA-] async    0x7f4bb0b4d270
59: [RA-] async    0x7f4bb0b4d770
59: [RA-] async    0x7f4bb0b4dc70
59: [RA-] async    0x7f4bb0b4de20
59: [RA-] async    0x7f4bb0b4dfd0
59: [active_handles] - [base_active_handles] - [extra_active_handles] + [active_reqs] + [closing]
59:        253        -           12          -            2           +    0 [> 0]    +    0

And the other:

59: RESULT: 32 9877
59: RESULT: 32 9878
59: RESULT: 32 9879
59: RESULT: 32 9880
59: RESULT: 32 9881
59: RESULT: 32 9882
59: RESULT: 32 9883
59: RESULT: 32 9884
59: RESULT: 32 9885
59: RESULT: 32 9886
59: RESULT: 32 9887
59: RESULT: 32 9888
59: RESULT: 32 9889
59: RESULT: 32 9890
59: RESULT: 32 9891
59: RESULT: 32 9892
59: RESULT: 32 9893
59: RESULT: 32 9894
59: RESULT: 32 9895
59: RESULT: 32 9896
59: RESULT: 32 9897
59: RESULT: 32 9898
59: RESULT: 32 9899
59: RESULT: 32 9900
59: RESULT: 32 9901
59: RESULT: 32 9902
59: RESULT: 32 9903
59: RESULT: 32 9904
59: RESULT: 32 9905
59: Number of active handles: 62
59: [active_handles] - [base_active_handles] - [extra_active_handles] + [active_reqs] + [closing]
59:        61        -           12          -            2           +    0 [> 0]    +    1
59: Number of user active handles: 48
59: [-AI] async    0x7f4f1112ec70
59: [-A-] async    0x7f4f0405f4c0
59: [-A-] async    0x7f4f1112d4e0
59: [-A-] check    0x7f4f040cc0b0
59: [RA-] idle     0x7f4f040cc128
59: [-A-] prepare  0x7f4f040cc1a0
59: [-A-] check    0x7f4f040cc218
59: [-A-] async    0x7f4f040cc290
59: [RA-] async    0x7f4f041233f8
59: [RA-] async    0x7f4f04200338
59: [RA-] async    0x7f4f041f0768
59: [RA-] async    0x7f4f041f0998
59: [RA-] async    0x7f4f041f88b8
59: [RA-] async    0x7f4f041f8cf8
59: [RA-] async    0x7f4f04211f58
59: [RA-] async    0x7f4f04212418
59: [RA-] async    0x7f4f041f8ea8
59: [RA-] async    0x7f4f0421fa08
59: [RA-] async    0x7f4f0421fe98
59: [RA-] async    0x7f4f042200f8
59: [RA-] async    0x7f4f04abb4b0
59: [RA-] async    0x7f4f04abaf90
59: [RA-] async    0x7f4f04abaa90
59: [RA-] async    0x7f4f04a753c0
59: [RA-] async    0x7f4f04a53830
59: [RA-] async    0x7f4f04abdbd0
59: [RA-] async    0x7f4f04ae2540
59: [RA-] async    0x7f4f04ae4340
59: [RA-] async    0x7f4f04ae3e40
59: [RA-] async    0x7f4f04ae3940
59: [RA-] async    0x7f4f04ae3440
59: [RA-] async    0x7f4f04aea740
59: [RA-] async    0x7f4f04aeac40
59: [RA-] async    0x7f4f04aeb140
59: [RA-] async    0x7f4f04aeb640
59: [RA-] async    0x7f4f04b144d0
59: [RA-] async    0x7f4f04b14590
59: [RA-] async    0x7f4f04b14700
59: [RA-] async    0x7f4f04b14990
59: [RA-] async    0x7f4f04b14e90
59: [RA-] async    0x7f4f04b15390
59: [RA-] async    0x7f4f04b15890
59: [RA-] async    0x7f4f04aebc10
59: [RA-] async    0x7f4f04aeedb0
59: [RA-] async    0x7f4f04aef220
59: [RA-] async    0x7f4f04aebf80
59: [RA-] async    0x7f4f04aec130
59: [RA-] async    0x7f4f04aec2e0
59: [RA-] async    0x7f4f04b15c80
59: [RA-] async    0x7f4f04b15e30
59: [RA-] async    0x7f4f04b160e0
59: [RA-] async    0x7f4f04b165e0
59: [RA-] async    0x7f4f04b16ae0
59: [RA-] async    0x7f4f04b16fe0
59: [RA-] async    0x7f4f04b174e0
59: [RA-] async    0x7f4f04b179e0
59: [RA-] async    0x7f4f04b17ee0
59: [RA-] async    0x7f4f04b183e0
59: [RA-] async    0x7f4f04b188e0
59: [RA-] async    0x7f4f04b18de0
59: [RA-] async    0x7f4f04b192e0
59: [RA-] async    0x7f4f04b197e0
59: [RA-] async    0x7f4f04b19ce0
59: [RA-] prepare  0x5596f76387e0
59: [RA-] check    0x5596f7638858
59: [active_handles] - [base_active_handles] - [extra_active_handles] + [active_reqs] + [closing]
59:        61        -           12          -            2           +    0 [> 0]    +    1
59: RESULT: 32 9906
59: RESULT: 32 9907
59: RESULT: 32 9908
59: RESULT: 32 9909
59: RESULT: 32 9910
59: RESULT: 32 9911
59: RESULT: 32 9912
59: RESULT: 32 9913
59: RESULT: 32 9914
59: RESULT: 32 9915
59: RESULT: 32 9916
59: RESULT: 32 9917
59: RESULT: 32 9918
59: RESULT: 32 9919
59: RESULT: 32 9920
59: RESULT: 32 9921
59: RESULT: 32 9922
59: RESULT: 32 9923
59: RESULT: 32 9924
59: RESULT: 32 9925
59: RESULT: 32 9926
59: RESULT: 32 9927
59: RESULT: 32 9928
59: RESULT: 32 9929
59: RESULT: 32 9930
59: RESULT: 32 9931
59: RESULT: 32 9932
59: RESULT: 32 9933
59: RESULT: 32 9934
59: RESULT: 32 9935
59: RESULT: 32 9936
59: RESULT: 32 9937
59: RESULT: 32 9938
59: RESULT: 32 9939
59: RESULT: 32 9940
59: RESULT: 32 9941
59: RESULT: 32 9942
59: RESULT: 32 9943
59: RESULT: 32 9944
59: RESULT: 32 9945
59: RESULT: 32 9946
59: RESULT: 32 9947
59: RESULT: 32 9948
59: RESULT: 32 9949
59: RESULT: 32 9950
59: RESULT: 32 9951
59: RESULT: 32 9952
59: RESULT: 32 9953
59: RESULT: 32 9954
59: RESULT: 32 9955
59: RESULT: 32 9956
59: Number of active handles: 60
59: [active_handles] - [base_active_handles] - [extra_active_handles] + [active_reqs] + [closing]
59:        60        -           12          -            2           +    0 [> 0]    +    0
59: Number of user active handles: 46
59: [-AI] async    0x7f4f1112ec70
59: [-A-] async    0x7f4f0405f4c0
59: [-A-] async    0x7f4f1112d4e0
59: [-A-] check    0x7f4f040cc0b0
59: [-A-] prepare  0x7f4f040cc1a0
59: [-A-] check    0x7f4f040cc218
59: [-A-] async    0x7f4f040cc290
59: [RA-] async    0x7f4f041233f8
59: [RA-] async    0x7f4f04200338
59: [RA-] async    0x7f4f041f0768
59: [RA-] async    0x7f4f041f0998
59: [RA-] async    0x7f4f041f88b8
59: [RA-] async    0x7f4f041f8cf8
59: [RA-] async    0x7f4f04211f58
59: [RA-] async    0x7f4f04212418
59: [RA-] async    0x7f4f041f8ea8
59: [RA-] async    0x7f4f0421fa08
59: [RA-] async    0x7f4f0421fe98
59: [RA-] async    0x7f4f042200f8
59: [RA-] async    0x7f4f04abb4b0
59: [RA-] async    0x7f4f04abaf90
59: [RA-] async    0x7f4f04abaa90
59: [RA-] async    0x7f4f04a753c0
59: [RA-] async    0x7f4f04a53830
59: [RA-] async    0x7f4f04abdbd0
59: [RA-] async    0x7f4f04ae2540
59: [RA-] async    0x7f4f04ae4340
59: [RA-] async    0x7f4f04ae3e40
59: [RA-] async    0x7f4f04ae3940
59: [RA-] async    0x7f4f04ae3440
59: [RA-] async    0x7f4f04aea740
59: [RA-] async    0x7f4f04aeac40
59: [RA-] async    0x7f4f04aeb140
59: [RA-] async    0x7f4f04aeb640
59: [RA-] async    0x7f4f04b144d0
59: [RA-] async    0x7f4f04b14590
59: [RA-] async    0x7f4f04b14700
59: [RA-] async    0x7f4f04b14990
59: [RA-] async    0x7f4f04b14e90
59: [RA-] async    0x7f4f04b15390
59: [RA-] async    0x7f4f04b15890
59: [RA-] async    0x7f4f04aebc10
59: [RA-] async    0x7f4f04aeedb0
59: [RA-] async    0x7f4f04aef220
59: [RA-] async    0x7f4f04aebf80
59: [RA-] async    0x7f4f04aec130
59: [RA-] async    0x7f4f04aec2e0
59: [RA-] async    0x7f4f04b15c80
59: [RA-] async    0x7f4f04b15e30
59: [RA-] async    0x7f4f04b160e0
59: [RA-] async    0x7f4f04b165e0
59: [RA-] async    0x7f4f04b16ae0
59: [RA-] async    0x7f4f04b16fe0
59: [RA-] async    0x7f4f04b174e0
59: [RA-] async    0x7f4f04b179e0
59: [RA-] async    0x7f4f04b17ee0
59: [RA-] async    0x7f4f04b183e0
59: [RA-] async    0x7f4f04b188e0
59: [RA-] async    0x7f4f04b18de0
59: [RA-] async    0x7f4f04b192e0
59: [RA-] async    0x7f4f04b197e0
59: [RA-] async    0x7f4f04b19ce0
59: [RA-] prepare  0x5596f76387e0
59: [RA-] check    0x5596f7638858
59: [active_handles] - [base_active_handles] - [extra_active_handles] + [active_reqs] + [closing]
59:        60        -           12          -            2           +    0 [> 0]    +    0

I am going to comment out the test until we do better debugging for this because we need to tests passing for other tasks.