Fatal Python error: could not acquire lock at interpreter shutdown...

[meshy]

A little bit excited about this project :)

In the process playing around, and have been getting some odd results with the following program:

# main.py
from chopsticks.tunnel import Tunnel
import tasks

def main(host):
    host.call(tasks.get_time)

if __name__ == '__main__':
    main(Tunnel('user@example.com'))

# tasks.py
import subprocess

def get_time():
    return subprocess.run('time', stdout=subprocess.PIPE, shell=True).stdout.decode()

• Sometimes nothing prints, and program exits with code 0.
• Sometimes I get the following (expected) printout (and code 0):

  [user@example.com] Usage: time [-apvV] [-f format] [-o file] [--append] [--verbose]
  [user@example.com]        [--portability] [--format=format] [--output=file] [--version]
  [user@example.com]        [--quiet] [--help] command [arg...]

• ... and sometimes I get a total failure:

[user@example.com] Usage: time [-apvV] [-f format] [-o file] [--append] [--verbose]
Fatal Python error: could not acquire lock for <_io.BufferedWriter name='<stderr>'>
at interpreter shutdown, possibly due to daemon threads

Thread 0x00007f86dde4c700 (most recent call first):
  File ".../lib/python3.6/site-packages/chopsticks/ioloop.py", line 155 in println
  File ".../lib/python3.6/site-packages/chopsticks/ioloop.py", line 162 in _check
  File ".../lib/python3.6/site-packages/chopsticks/ioloop.py", line 145 in on_data
  File ".../lib/python3.6/site-packages/chopsticks/ioloop.py", line 223 in step
  File ".../lib/python3.6/site-packages/chopsticks/ioloop.py", line 242 in run
  File "/usr/lib64/python3.6/threading.py", line 864 in run
  File "/usr/lib64/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib64/python3.6/threading.py", line 884 in _bootstrap

Current thread 0x00007f86e259c4c0 (most recent call first):
[1]    6661 abort (core dumped)  ./main.py

The number of lines that successfully print before failure varies.

---

I'm running Python 3.6.1 locally, and 3.5.2 on the server. I don't know if it helps, but the result of running ssh -V is OpenSSH_7.5p1, OpenSSL 1.1.0f 25 May 2017.

Please let me know if there's any further information I could provide to help shed some light on this.

[lordmauve]

Hmm, looks like the import of subprocess (which happens both sides) is tripping over Chopsticks' IO loop somehow. May be to do with the stderr handling.

I will see if I can reproduce this when I get a few minutes.

[meshy]

Thanks :)

[lordmauve]

I have not been able to reproduce this.

Please can you let me know whether you're using Chopsticks 0.5 or master - and please could you check if it still occurs under current master.

I do understand the race condition where no stderr may be printed. That's somewhat expected, from my perspective, but I'll think what I can do to fix this or at least make it less surprising.

The crash is still very baffling.

[meshy]

I've just had another look. I was indeed using chopsticks 0.5, and it works on master \o/.

I've done a bisect, and it looks as though it was fixed in 41d7683da300e2a0f88a3a46add0eb49b3f54fce.

Thank you!

[lordmauve]

Thanks. I'll open an issue about the stderr handling specifically.