NotImplementedError raised in thread for "handle_stdin"

[pzehner]

I use Invoke v1.3.0 through Fabric v2.4.0 to execute actions on a remote server from a Docker container run by Gitlab-CI. When trying to run a simple remote command (i.e. uname -a), I get the following exception:

Traceback (most recent call last):
  File "/builds/lbm/bcm-lbm/tests/regression/../utils/utils.py", line 40, in manage_exceptions
    yield None
  File "./prepare-jss", line 53, in <module>
    prepare(os.getcwd(), args.case)
  File "/builds/lbm/bcm-lbm/tests/regression/../utils/prepare_jss.py", line 34, in prepare
    run_jss(cases)
  File "/builds/lbm/bcm-lbm/tests/regression/../utils/run_jss.py", line 1370, in run
    check_connection(c)
  File "/builds/lbm/bcm-lbm/tests/regression/../utils/run_jss.py", line 148, in check_connection
    result = connection.run("uname -a", hide=True, warn=True)
  File "<decorator-gen-3>", line 2, in run
  File "/usr/lib/python3.6/site-packages/fabric/connection.py", line 30, in opens
    return method(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/fabric/connection.py", line 702, in run
    return self._run(self._remote_runner(), command, **kwargs)
  File "/usr/lib/python3.6/site-packages/invoke/context.py", line 101, in _run
    return runner.run(command, **kwargs)
  File "/usr/lib/python3.6/site-packages/invoke/runners.py", line 291, in run
    return self._run_body(command, **kwargs)
  File "/usr/lib/python3.6/site-packages/invoke/runners.py", line 399, in _run_body
    raise ThreadException(thread_exceptions)
invoke.exceptions.ThreadException: 
Saw 1 exceptions within threads (NotImplementedError):

Thread args: {'kwargs': {'echo': None,
            'input_': <_io.TextIOWrapper name='<stdin>' mode='r' encoding='ANSI_X3.4-1968'>,
            'output': <_io.TextIOWrapper name='<stdout>' mode='w' encoding='ANSI_X3.4-1968'>},
 'target': <bound method Runner.handle_stdin of <fabric.runners.Remote object at 0x7f6d2281f438>>}

Traceback (most recent call last):

  File "/usr/lib/python3.6/site-packages/invoke/util.py", line 233, in run
    super(ExceptionHandlingThread, self).run()

  File "/usr/lib64/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)

  File "/usr/lib/python3.6/site-packages/invoke/runners.py", line 706, in handle_stdin
    self.close_proc_stdin()

  File "/usr/lib/python3.6/site-packages/invoke/runners.py", line 939, in close_proc_stdin
    raise NotImplementedError

NotImplementedError

Here are the last lines of the logs:

[2019-09-17 04:34:01,743] DEBUG    [paramiko.transport] Adding ssh-rsa host key for XXXXXXX: b'YYYYYYYYYYYY'
[2019-09-17 04:34:01,744] DEBUG    [paramiko.transport] Trying SSH agent key b'YYYYYYYYYYYY'
[2019-09-17 04:34:01,782] DEBUG    [paramiko.transport] userauth is OK
[2019-09-17 04:34:01,874] INFO     [paramiko.transport] Authentication (publickey) successful!
[2019-09-17 04:34:01,876] DEBUG    [paramiko.transport] [chan 0] Max packet in: 32768 bytes
[2019-09-17 04:34:01,886] DEBUG    [paramiko.transport] [chan 0] Max packet out: 32768 bytes
[2019-09-17 04:34:01,886] DEBUG    [paramiko.transport] Secsh channel 0 opened.
[2019-09-17 04:34:01,891] DEBUG    [paramiko.transport] [chan 0] Sesch channel 0 request ok
[2019-09-17 04:34:01,892] DEBUG    [invoke] Encountered exception NotImplementedError() in thread for 'handle_stdin'
[2019-09-17 04:34:02,109] DEBUG    [paramiko.transport] [chan 0] EOF received (0)
[2019-09-17 04:34:02,110] DEBUG    [paramiko.transport] [chan 0] EOF sent (0)

I was unable to reproduce the bug: neither on dev environment, nor on the same Docker container, nor on the same Docker container on the same host as Gitlab-CI. I have absolutely no clue on what is going wrong and why close_proc_stdin is called. I have no problem when using Invoke v1.2.0, as handle_stdin behaves differently (see for v1.2.0 and for v1.3.0)

• Dev environment: GNU/Linux Open SUSE Leap 15.0, kernel 4.12.14-lp150.12.70-default, Python 3.6.5;
• Docker image environment: GNU/Linux Open SUSE Leap 15.0, kernel 3.10.0-957.27.2.el7.x86_64, Python 3.6.5;
• Target server environment: GNU/Linux Red Hat Enterprise Linux Server release 6.7 (Santiago), kernel 2.6.32-573.el6.x86_64.

[estein-de]

Anyone ever figure out anything about this?

[andr0s]

Ouch I have the exact same issue :\ Update - I checked the code and it seems it all started since version 1.3. That method just raises NotImplementedError without even explaining what exactly is not implemented. Please fix this guys, it's been 3 years already with that pretty weird (bad?) code sitting in there

[neozenith]

Are there exact steps available for us to reproduce this issue?

Is it still happening with the latest releases of fabric 2.6 and invoke 1.7?

[andr0s]

Ehhh... Still not fixed? Omg

[MattiaCampana90b]

Are there exact steps available for us to reproduce this issue?

Is it still happening with the latest releases of fabric 2.6 and invoke 1.7?

Fabric 2.7.1 and invoke 1.7.3 works fine for me inside this Bitbucket ci/cd pipeline:

image: python:3.9-bullseye
pipelines:
  default:
    - step:
        script:
          - python --version
  branches:
    master:
      - step:
          script:
            - apt-get update && apt-get install -y unzip rsync build-essential libssl-dev libffi-dev cargo
            - curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs >> rustup.sh
            - sh rustup.sh -y
            - rustc --version
            - pip install --upgrade pip
            - pip install fabric2==2.7.1
            - pip install patchwork
            - fab2 deploy

Also my pip list output:

bcrypt       4.0.1
cffi         1.15.1
cryptography 40.0.2
fabric2      2.7.1
invoke       1.7.3
paramiko     3.1.0
patchwork    1.0.1
pathlib2     2.3.7.post1
pip          23.1.2
pycparser    2.21
PyNaCl       1.5.0
setuptools   58.1.0
six          1.16.0
wheel        0.40.0

For people landing here with the same error: patchwork forcing an invoke 1.7.3 on a fabric2 3.x.x version was the issue for me :)