search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.49k stars 580 forks source link

CI Failure with stress fibers (asan crash) in `PrefixTruncateRecoveryTest.test_prefix_truncate_recovery` #12700

Closed andrwng closed 3 months ago

andrwng commented 1 year ago

https://buildkite.com/redpanda/redpanda/builds/34701#0189dbcf-f2e8-4126-a6f7-9b960026f796

This particular instance was run with a bunch of stress fibers starting up on each node before the test runs. https://github.com/redpanda-data/redpanda/pull/12611

The test may be stable on dev.

Module: rptest.tests.prefix_truncate_recovery_test
Class:  PrefixTruncateRecoveryTest
Method: test_prefix_truncate_recovery
Arguments:
{
  "acks": -1,
  "start_empty": true
}
test_id:    rptest.tests.prefix_truncate_recovery_test.PrefixTruncateRecoveryTest.test_prefix_truncate_recovery.acks=-1.start_empty=True
status:     FAIL
run time:   2 minutes 13.776 seconds

    <NodeCrash docker-rp-22: ==4701==ERROR: AddressSanitizer: heap-use-after-free on address 0x60d000033680 at pc 0x7fcb9cfe5ae8 bp 0x7fff49e16ae0 sp 0x7fff49e16ad8
>
Traceback (most recent call last):
  File "/root/tests/rptest/services/cluster.py", line 82, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/prefix_truncate_recovery_test.py", line 160, in test_prefix_truncate_recovery
    self.run_recovery(acks=acks, dst_node=self.redpanda.nodes[0])
  File "/root/tests/rptest/tests/prefix_truncate_recovery_test.py", line 132, in run_recovery
    wait_until(
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 481, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 103, in wrapped
    redpanda.raise_on_crash(log_allow_list=log_allow_list)
  File "/root/tests/rptest/services/redpanda.py", line 2845, in raise_on_crash
    raise NodeCrash(crashes)
rptest.services.utils.NodeCrash: <NodeCrash docker-rp-22: ==4701==ERROR: AddressSanitizer: heap-use-after-free on address 0x60d000033680 at pc 0x7fcb9cfe5ae8 bp 0x7fff49e16ae0 sp 0x7fff49e16ad8
>

JIRA Link: CORE-1396

andrwng commented 1 year ago

This is happening at shutdown of a node. Here is the ASAN dump

=================================================================
==4701==ERROR: AddressSanitizer: heap-use-after-free on address 0x60d000033680 at pc 0x7fcb9cfe5ae8 bp 0x7fff49e16ae0 sp 0x7fff49e16ad8
READ of size 8 at 0x60d000033680 thread T0
/usr/bin/llvm-symbolizer: /opt/redpanda_installs/ci/lib/libc.so.6: version `GLIBC_2.36' not found (required by /usr/lib/llvm-15/bin/../lib/libLLVM-15.so.1)
/usr/bin/llvm-symbolizer: /opt/redpanda_installs/ci/lib/libc.so.6: version `GLIBC_2.36' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
==4701==WARNING: Can't read from symbolizer at fd 19
==4701==WARNING: Can't write to symbolizer at fd 23
/usr/bin/llvm-symbolizer: /opt/redpanda_installs/ci/lib/libc.so.6: version `GLIBC_2.36' not found (required by /usr/lib/llvm-15/bin/../lib/libLLVM-15.so.1)
/usr/bin/llvm-symbolizer: /opt/redpanda_installs/ci/lib/libc.so.6: version `GLIBC_2.36' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
==4701==WARNING: Can't read from symbolizer at fd 19
==4701==WARNING: Can't write to symbolizer at fd 23
Reactor stalled for 173 ms on shard 0. Backtrace: 0xf1016 0x471b235 0x471a17a 0x44e02e3 0x44dd265 0x44dcd17 0x44dd4e4 0x44e24d2 0x42abf 0x15f45d 0x16a476 0x173eec 0x1733a7 0x172e6e 0x171bab 0x170236 0x170166 0x1704e4 0xb90d0 0x14a87b 0x14dd0c 0x14e9a7 0x46e0ae7 0x450e028 0x451283c 0x451845b 0x4515e35 0x3f9333c 0x3f90d36 0x6d17022 0x182f83 0x2d58f 0x2d648 0xabf74
kernel callstack: 0xffffffffffffff80 0xffffffff89090ea1 0xffffffff89091158 0xffffffff8909250d 0xffffffff8909293b 0xffffffff89092cb0 0xffffffff89974b0b 0xffffffff89a00099
/usr/bin/llvm-symbolizer: /opt/redpanda_installs/ci/lib/libc.so.6: version `GLIBC_2.36' not found (required by /usr/lib/llvm-15/bin/../lib/libLLVM-15.so.1)
/usr/bin/llvm-symbolizer: /opt/redpanda_installs/ci/lib/libc.so.6: version `GLIBC_2.36' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
==4701==WARNING: Can't read from symbolizer at fd 19
==4701==WARNING: Can't write to symbolizer at fd 23
/usr/bin/llvm-symbolizer: /opt/redpanda_installs/ci/lib/libc.so.6: version `GLIBC_2.36' not found (required by /usr/lib/llvm-15/bin/../lib/libLLVM-15.so.1)
/usr/bin/llvm-symbolizer: /opt/redpanda_installs/ci/lib/libc.so.6: version `GLIBC_2.36' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
==4701==WARNING: Can't read from symbolizer at fd 19
==4701==WARNING: Can't write to symbolizer at fd 23
==4701==WARNING: Failed to use and restart external symbolizer!
    #0 0x7fcb9cfe5ae7  (/opt/redpanda_installs/ci/lib/libseastar.so+0x46e0ae7) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #1 0x7fcb9ce13028  (/opt/redpanda_installs/ci/lib/libseastar.so+0x450e028) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #2 0x7fcb9ce1783c  (/opt/redpanda_installs/ci/lib/libseastar.so+0x451283c) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #3 0x7fcb9ce1d45b  (/opt/redpanda_installs/ci/lib/libseastar.so+0x451845b) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #4 0x7fcb9ce1ae35  (/opt/redpanda_installs/ci/lib/libseastar.so+0x4515e35) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #5 0x7fcb9c89833c  (/opt/redpanda_installs/ci/lib/libseastar.so+0x3f9333c) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #6 0x7fcb9c895d36  (/opt/redpanda_installs/ci/lib/libseastar.so+0x3f90d36) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #7 0x7fcc06c94022  (/opt/redpanda_installs/ci/lib/libv_v_application.so+0x6d17022) (BuildId: 263dad8f2673778ccc7ec18bd387fd5e8d893eeb)
    #8 0x55c618e9bf83  (/opt/redpanda_installs/ci/libexec/redpanda+0x182f83) (BuildId: 3f40e4a7ccae42ceac3df19cb50024b7a5bb6cc4)
    #9 0x7fcb97ada58f  (/opt/redpanda_installs/ci/lib/libc.so.6+0x2d58f) (BuildId: 6e7b96dfb83f0bdcb6a410469b82f86415e5ada3)
    #10 0x7fcb97ada648  (/opt/redpanda_installs/ci/lib/libc.so.6+0x2d648) (BuildId: 6e7b96dfb83f0bdcb6a410469b82f86415e5ada3)
    #11 0x55c618dc4f74  (/opt/redpanda_installs/ci/libexec/redpanda+0xabf74) (BuildId: 3f40e4a7ccae42ceac3df19cb50024b7a5bb6cc4)

0x60d000033680 is located 0 bytes inside of 144-byte region [0x60d000033680,0x60d000033710)
freed by thread T0 here:
    #0 0x55c618e9a03d  (/opt/redpanda_installs/ci/libexec/redpanda+0x18103d) (BuildId: 3f40e4a7ccae42ceac3df19cb50024b7a5bb6cc4)
    #1 0x7fcb9d198686  (/opt/redpanda_installs/ci/lib/libseastar.so+0x4893686) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #2 0x7fcb9d198454  (/opt/redpanda_installs/ci/lib/libseastar.so+0x4893454) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #3 0x7fcb9cf9710a  (/opt/redpanda_installs/ci/lib/libseastar.so+0x469210a) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #4 0x7fcb9cf96b0f  (/opt/redpanda_installs/ci/lib/libseastar.so+0x4691b0f) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #5 0x7fcb9cf96aa3  (/opt/redpanda_installs/ci/lib/libseastar.so+0x4691aa3) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #6 0x7fcb9cf96a2d  (/opt/redpanda_installs/ci/lib/libseastar.so+0x4691a2d) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #7 0x7fcb9cf9699b  (/opt/redpanda_installs/ci/lib/libseastar.so+0x469199b) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #8 0x7fcc05608440  (/opt/redpanda_installs/ci/lib/libv_v_application.so+0x568b440) (BuildId: 263dad8f2673778ccc7ec18bd387fd5e8d893eeb)
    #9 0x7fcc056082d3  (/opt/redpanda_installs/ci/lib/libv_v_application.so+0x568b2d3) (BuildId: 263dad8f2673778ccc7ec18bd387fd5e8d893eeb)
    #10 0x7fcc05608263  (/opt/redpanda_installs/ci/lib/libv_v_application.so+0x568b263) (BuildId: 263dad8f2673778ccc7ec18bd387fd5e8d893eeb)
    #11 0x7fcc05608087  (/opt/redpanda_installs/ci/lib/libv_v_application.so+0x568b087) (BuildId: 263dad8f2673778ccc7ec18bd387fd5e8d893eeb)
    #12 0x7fcc05607e60  (/opt/redpanda_installs/ci/lib/libv_v_application.so+0x568ae60) (BuildId: 263dad8f2673778ccc7ec18bd387fd5e8d893eeb)
    #13 0x7fcc05607c56  (/opt/redpanda_installs/ci/lib/libv_v_application.so+0x568ac56) (BuildId: 263dad8f2673778ccc7ec18bd387fd5e8d893eeb)
    #14 0x7fcc056079b4  (/opt/redpanda_installs/ci/lib/libv_v_application.so+0x568a9b4) (BuildId: 263dad8f2673778ccc7ec18bd387fd5e8d893eeb)
    #15 0x7fcc056073ce  (/opt/redpanda_installs/ci/lib/libv_v_application.so+0x568a3ce) (BuildId: 263dad8f2673778ccc7ec18bd387fd5e8d893eeb)
    #16 0x7fcb9ce0a0d4  (/opt/redpanda_installs/ci/lib/libseastar.so+0x45050d4) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #17 0x7fcb9ce171f1  (/opt/redpanda_installs/ci/lib/libseastar.so+0x45121f1) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #18 0x7fcb9ce1d45b  (/opt/redpanda_installs/ci/lib/libseastar.so+0x451845b) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #19 0x7fcb9ce1ae35  (/opt/redpanda_installs/ci/lib/libseastar.so+0x4515e35) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #20 0x7fcb9c89833c  (/opt/redpanda_installs/ci/lib/libseastar.so+0x3f9333c) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #21 0x7fcb9c895d36  (/opt/redpanda_installs/ci/lib/libseastar.so+0x3f90d36) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #22 0x7fcc06c94022  (/opt/redpanda_installs/ci/lib/libv_v_application.so+0x6d17022) (BuildId: 263dad8f2673778ccc7ec18bd387fd5e8d893eeb)
    #23 0x55c618e9bf83  (/opt/redpanda_installs/ci/libexec/redpanda+0x182f83) (BuildId: 3f40e4a7ccae42ceac3df19cb50024b7a5bb6cc4)
    #24 0x7fcb97ada58f  (/opt/redpanda_installs/ci/lib/libc.so.6+0x2d58f) (BuildId: 6e7b96dfb83f0bdcb6a410469b82f86415e5ada3)

previously allocated by thread T0 here:
    #0 0x55c618e997dd  (/opt/redpanda_installs/ci/libexec/redpanda+0x1807dd) (BuildId: 3f40e4a7ccae42ceac3df19cb50024b7a5bb6cc4)
    #1 0x7fcb9d006815  (/opt/redpanda_installs/ci/lib/libseastar.so+0x4701815) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #2 0x7fcb9ce47f66  (/opt/redpanda_installs/ci/lib/libseastar.so+0x4542f66) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #3 0x7fcb9cf9a51d  (/opt/redpanda_installs/ci/lib/libseastar.so+0x469551d) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #4 0x7fcb9cf99113  (/opt/redpanda_installs/ci/lib/libseastar.so+0x4694113) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #5 0x7fcb9cf989fa  (/opt/redpanda_installs/ci/lib/libseastar.so+0x46939fa) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #6 0x7fcb9cf9863a  (/opt/redpanda_installs/ci/lib/libseastar.so+0x469363a) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #7 0x7fcb9cf9839a  (/opt/redpanda_installs/ci/lib/libseastar.so+0x469339a) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #8 0x7fcb9cf982af  (/opt/redpanda_installs/ci/lib/libseastar.so+0x46932af) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #9 0x7fcb9cf97e71  (/opt/redpanda_installs/ci/lib/libseastar.so+0x4692e71) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #10 0x7fcb9cf97a19  (/opt/redpanda_installs/ci/lib/libseastar.so+0x4692a19) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #11 0x7fcb9cf9fb4f  (/opt/redpanda_installs/ci/lib/libseastar.so+0x469ab4f) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #12 0x7fcb9cf97b6b  (/opt/redpanda_installs/ci/lib/libseastar.so+0x4692b6b) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #13 0x7fcb9cf9769a  (/opt/redpanda_installs/ci/lib/libseastar.so+0x469269a) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #14 0x7fcb9cf9741b  (/opt/redpanda_installs/ci/lib/libseastar.so+0x469241b) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #15 0x7fcb9ce4c503  (/opt/redpanda_installs/ci/lib/libseastar.so+0x4547503) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #16 0x7fcb9ce4bd84  (/opt/redpanda_installs/ci/lib/libseastar.so+0x4546d84) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #17 0x7fcc06fa6685  (/opt/redpanda_installs/ci/lib/libv_v_application.so+0x7029685) (BuildId: 263dad8f2673778ccc7ec18bd387fd5e8d893eeb)
    #18 0x7fcc06c96957  (/opt/redpanda_installs/ci/lib/libv_v_application.so+0x6d19957) (BuildId: 263dad8f2673778ccc7ec18bd387fd5e8d893eeb)
    #19 0x7fcc06d01a04  (/opt/redpanda_installs/ci/lib/libv_v_application.so+0x6d84a04) (BuildId: 263dad8f2673778ccc7ec18bd387fd5e8d893eeb)
    #20 0x7fcc06d014d0  (/opt/redpanda_installs/ci/lib/libv_v_application.so+0x6d844d0) (BuildId: 263dad8f2673778ccc7ec18bd387fd5e8d893eeb)
    #21 0x7fcc06d01474  (/opt/redpanda_installs/ci/lib/libv_v_application.so+0x6d84474) (BuildId: 263dad8f2673778ccc7ec18bd387fd5e8d893eeb)
    #22 0x7fcc06d0140a  (/opt/redpanda_installs/ci/lib/libv_v_application.so+0x6d8440a) (BuildId: 263dad8f2673778ccc7ec18bd387fd5e8d893eeb)
    #23 0x7fcc06d01267  (/opt/redpanda_installs/ci/lib/libv_v_application.so+0x6d84267) (BuildId: 263dad8f2673778ccc7ec18bd387fd5e8d893eeb)
    #24 0x7fcc06d00fe2  (/opt/redpanda_installs/ci/lib/libv_v_application.so+0x6d83fe2) (BuildId: 263dad8f2673778ccc7ec18bd387fd5e8d893eeb)
    #25 0x7fcc06d00df8  (/opt/redpanda_installs/ci/lib/libv_v_application.so+0x6d83df8) (BuildId: 263dad8f2673778ccc7ec18bd387fd5e8d893eeb)
    #26 0x7fcbd92dc34d  (/opt/redpanda_installs/ci/lib/libv_v_cluster.so+0x19ff834d) (BuildId: e381b920bac3304d8d9d92bfb08a9b10e7f34edc)
    #27 0x7fcb9d4506e4  (/opt/redpanda_installs/ci/lib/libseastar.so+0x4b4b6e4) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #28 0x7fcb9d450063  (/opt/redpanda_installs/ci/lib/libseastar.so+0x4b4b063) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f)
    #29 0x7fcb97b0582f  (/opt/redpanda_installs/ci/lib/libc.so.6+0x5882f) (BuildId: 6e7b96dfb83f0bdcb6a410469b82f86415e5ada3)

SUMMARY: AddressSanitizer: heap-use-after-free (/opt/redpanda_installs/ci/lib/libseastar.so+0x46e0ae7) (BuildId: 5ce1cc03efe4bc9743936954ac2b2aae2639716f) 
Shadow bytes around the buggy address:
  0x60d000033400: fa fa fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x60d000033480: fd fd fd fd fa fa fa fa fa fa fa fa fd fd fd fd
  0x60d000033500: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa fa
  0x60d000033580: fa fa fa fa fa fa fd fd fd fd fd fd fd fd fd fd
  0x60d000033600: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
=>0x60d000033680:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x60d000033700: fd fd fa fa fa fa fa fa fa fa fd fd fd fd fd fd
  0x60d000033780: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa
  0x60d000033800: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
  0x60d000033880: fd fd fd fd fd fa fa fa fa fa fa fa fa fa fd fd
  0x60d000033900: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==4701==ABORTING
/usr/bin/llvm-symbolizer: /opt/redpanda_installs/ci/lib/libc.so.6: version `GLIBC_2.36' not found (required by /usr/lib/llvm-15/bin/../lib/libLLVM-15.so.1)
/usr/bin/llvm-symbolizer: /opt/redpanda_installs/ci/lib/libc.so.6: version `GLIBC_2.36' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
andrwng commented 1 year ago

Unless i decoded the stack incorrectly, the bad usage was

0x46e0ae7 0x450e028 0x451283c 0x451845b 0x4515e35 0x3f9333c 0x3f90d36
seastar::reactor::task_queue::indirect_compare::operator()(seastar::reactor::task_queue const*, seastar::reactor::task_queue const*) const at ??:?
seastar::reactor::insert_active_task_queue(seastar::reactor::task_queue*) at ??:?
seastar::reactor::run_some_tasks() at ??:?
seastar::reactor::do_run() at ??:?
seastar::reactor::run() at ??:?
seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) at ??:?
seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) at ??:?

It looks like maybe a task is running after seastar is being torn down. My guess is the task being run is actually a stress fiber, though the ASAN stacks are kind of unhelpful figuring it out. Regardless, put up https://github.com/redpanda-data/redpanda/pull/12701, maybe it'll do some good.

piyushredpanda commented 1 year ago

@andrwng : What's next then given #12701 is closed? Still to be triaged?

Lazin commented 1 year ago

assigned to storage because we manage stress fiber

github-actions[bot] commented 7 months ago

This issue hasn't seen activity in 3 months. If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in two weeks.

piyushredpanda commented 7 months ago

Opening this again.

piyushredpanda commented 3 months ago

Not seen in at least two months, closing