Python 3.10 OpenSSL Configuration Issues

[e91f3816-b999-40f5-bc1b-dffbfd2e99d4]

BPO	46863
Nosy	@tiran, @adampinky85
Files	python_builds.tar.gz python_details.tar.gz

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['expert-SSL', 'type-bug', '3.10'] title = 'Python 3.10 OpenSSL Configuration Issues' updated_at = user = 'https://github.com/adampinky85' ``` bugs.python.org fields: ```python activity = actor = 'adam' assignee = 'none' closed = False closed_date = None closer = None components = ['SSL'] creation = creator = 'adam' dependencies = [] files = ['50653', '50654'] hgrepos = [] issue_num = 46863 keywords = [] message_count = 11.0 messages = ['414072', '414089', '414093', '414098', '414101', '414105', '414226', '414246', '414321', '414340', '414402'] nosy_count = 2.0 nosy_names = ['christian.heimes', 'adam'] pr_nums = [] priority = 'normal' resolution = None stage = None status = 'open' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue46863' versions = ['Python 3.10'] ```

[e91f3816-b999-40f5-bc1b-dffbfd2e99d4]

Python 3.10 does not appear to respecting the OpenSSL configuration within linux. Testing completed using Pyenv on both Ubuntu 20.04.4 and Centos-8. Note PEP-644 which requires OpenSSL >= 1.1.1 is released in Python 3.10.

We operate behind a corporate proxy / firewall which causes an SSL error where the Diffie-Hellman key size is too small. In previous Python versions this is resolved by updating the OpenSSL configuration, e.g. downgrading the linux crypto policies sudo update-crypto-policies --set LEGACY.

The issue is reproducible in both Ubuntu 20.04.4 and Centos-8. In both linux distributions the SSL error is resolvable in earlier Python version, using the OpenSSL configurations, but the configuration is not respected with Python 3.10.2.

See the details below on the kernel versions, linux distributions, and Openssl versions, many thanks in advance.

1. Python 3.10.2 Error: (py_3_10_2) ➜ py_3_10_2 pip install --upgrade pip WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:997)'))': /simple/pip/

2. Ubuntu details uname -a Linux Horatio 5.13.0-30-generic #33\~20.04.1-Ubuntu SMP Mon Feb 7 14:25:10 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.4 LTS Release: 20.04 Codename: focal

openssl version -a OpenSSL 1.1.1f 31 Mar 2020 built on: Wed Nov 24 13:20:48 2021 UTC platform: debian-amd64 options: bn(64,64) rc4(16x,int) des(int) blowfish(ptr) compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -fdebug-prefix-map=/build/openssl-dnfdFp/openssl-1.1.1f=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_TLS_SECURITY_LEVEL=2 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAESNI_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2 OPENSSLDIR: "/usr/lib/ssl" ENGINESDIR: "/usr/lib/x86_64-linux-gnu/engines-1.1" Seeding source: os-specific

2. Centos-8 details uname -a Linux localhost.localdomain 5.4.181-1.el8.elrepo.x86_64 #1 SMP Tue Feb 22 10:00:15 EST 2022 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/centos-release CentOS Stream release 8

openssl version -a OpenSSL 1.1.1k FIPS 25 Mar 2021 built on: Thu Dec 2 16:40:48 2021 UTC platform: linux-x86_64 options: bn(64,64) md2(char) rc4(16x,int) des(int) idea(int) blowfish(ptr) compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -Wa,--noexecstack -Wa,--generate-missing-build-notes=yes -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAESNI_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -DZLIB -DNDEBUG -DPURIFY -DDEVRANDOM="\"/dev/urandom\"" -DSYSTEM_CIPHERS_FILE="/etc/crypto-policies/back-ends/openssl.config" OPENSSLDIR: "/etc/pki/tls" ENGINESDIR: "/usr/lib64/engines-1.1" Seeding source: os-specific engines: rdrand dynamic

[tiran]

How did you build Python 3.10? Neither CentOS 8 nor Ubuntu 20.04 come with Python 3.10. Does your build of Python use system's OpenSSL build?

[e91f3816-b999-40f5-bc1b-dffbfd2e99d4]

Thanks for the quick reply. On both Ubuntu and Centos, I’m installing Python using Pyenv, testing with 3.9.10 and 3.10.2. Pyenv provides a verbose install flag, I can rebuild the Python versions and review the build commands, if helpful? I’m testing with clean Linux distributions and I believe there is only one OpenSSL installed and available. I don’t know if it’s possible to gain more details from the Python ssl module to confirm? I did confirm the OpenSSL versions aligns using ssl.OPENSSL_VERSION.

Command: pyenv install 3.10.2 --verbose

https://github.com/pyenv/pyenv

[e91f3816-b999-40f5-bc1b-dffbfd2e99d4]

I found the Python build recipes and Pyenv does appear to install OpenSSL from source. The only difference I can see, aside from the Python version, is an update on the OpenSSL versions; openssl-1.1.1l (3.9.10) to openssl-1.1.1k (3.10.2). The OpenSSL release notes do not appear to suggest anything relevant.

https://github.com/pyenv/pyenv/blob/master/plugins/python-build/share/python-build/3.10.2

https://github.com/pyenv/pyenv/blob/master/plugins/python-build/share/python-build/3.9.10

https://github.com/pyenv/pyenv/blob/master/plugins/python-build/bin/python-build

https://www.openssl.org/news/openssl-1.1.1-notes.html

[tiran]

This is a pyenv issue, not a Python issue. Custom builds of OpenSSL typically do not and cannot use global settings like crypto policies. They are missing distro downstream patches and use different config files.

[e91f3816-b999-40f5-bc1b-dffbfd2e99d4]

Yes agreed, it may well be a Pyenv issue. Interestingly we can demonstrate that the global OpenSSL crypto policies is respected with the 3.9.10 version, through adjusting the policy. The ssl error occurs with the default policy setting and is resolved with the legacy policy setting. With 3.10.2 this is no longer the case. I can’t see any obvious changes to the build recipe that would cause this.

[e91f3816-b999-40f5-bc1b-dffbfd2e99d4]

Update, the Pyenv team confirmed that they do not install OpenSSL in linux, its only installed for MacOS, and it should be built using the system OpenSSL within Linux.

We're investigating further to attempt to debug the issue. Interestingly the OpenSSL build flags for both Python versions appear to be the same.

Trying link with OPENSSL_LDFLAGS=; OPENSSL_LIBS=-lssl -lcrypto; OPENSSL_INCLUDES=

I've attached the build logs for both the Python 3.9.10 and 3.10.2 build, in case you're able to review. Many thanks.

https://github.com/pyenv/pyenv/issues/2257

[tiran]

Could you please provide the outputs of the following commands:

python3 -c "import _ssl; print(ssl.\_file__)"

ldd $(python3.10 -c "import _ssl; print(ssl.\_file__)")

strace -e openat python3.10 -c "from urllib.request import urlopen; urlopen('https://www.python.org')"

The outputs look like this on my computer:

$ python3 -c "import _ssl; print(_ssl.__file__)"
/usr/lib64/python3.10/lib-dynload/_ssl.cpython-310-x86_64-linux-gnu.so
$ ldd $(python3.10 -c "import _ssl; print(_ssl.__file__)")
        linux-vdso.so.1 (0x00007ffd1a10e000)
        libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007ff838e17000)
        libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007ff838b29000)
        libc.so.6 => /lib64/libc.so.6 (0x00007ff83891f000)
        libz.so.1 => /lib64/libz.so.1 (0x00007ff838905000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ff838f0b000)

$ strace -e openat python3.10 -c "from urllib.request import urlopen; urlopen('https://www.python.org')"
...
openat(AT_FDCWD, "/etc/pki/tls/openssl.cnf", O_RDONLY) = 3
openat(AT_FDCWD, "/etc/crypto-policies/back-ends/opensslcnf.config", O_RDONLY) = 4
openat(AT_FDCWD, "/etc/crypto-policies/back-ends/openssl.config", O_RDONLY) = 3
openat(AT_FDCWD, "/etc/pki/tls/cert.pem", O_RDONLY) = 3
...

[e91f3816-b999-40f5-bc1b-dffbfd2e99d4]

Many thanks Christian, see the attached for the output of the commands on Python 3.9.10 and 3.10.2, along with a diff removing version numbers and memory addresses.

I've run the commands on the Ubuntu distribution, we can also run the same for the Centos VM, if helpful.

There are a few differences in the outputs but nothing that appears obviously the cause.

[tiran]

pyenv uses default value for ./configure --with-ssl-default-suites. You have to use --with-ssl-default-suites=openssl so your build uses the system's crypto policy correctly.

[e91f3816-b999-40f5-bc1b-dffbfd2e99d4]

Many thanks Christian, that resolved the issue! I really appreciate your efforts here.

[Doubletop12]

I have been going around in circles trying to find a solution to this but there is nothing that seems to work. Here is the result of help() modules ssl in Python 3.10.7 `

help> modules ssl

Here is a list of modules whose name or summary contains 'ssl'.
If there are any, enter a module name to get more help.

asyncio.sslproto 
ssl - This module provides some more Pythonic support for SSL.
test.make_ssl_certs - Make the custom certificate and private key files used by test_ssl
test.ssl_servers 
test.ssltests 
test.test_asyncio.test_sslproto - Tests for asyncio/sslproto.py.
test.test_ssl 
pip._vendor.urllib3.contrib.pyopenssl - TLS with SNI_-support for Python 2. Follow these instructions if you would
pip._vendor.urllib3.util.ssl_ 
pip._vendor.urllib3.util.ssl_match_hostname - The match_hostname() function from Python 3.3.3, essential when using SSL.
pip._vendor.urllib3.util.ssltransport 

help>

` wheras for Python 3.9

`

help> modules ssl

Here is a list of modules whose name or summary contains 'ssl'.
If there are any, enter a module name to get more help.

asyncio.sslproto 
ssl - This module provides some more Pythonic support for SSL.
_ssl 
pip._vendor.urllib3.contrib.pyopenssl - TLS with SNI_-support for Python 2. Follow these instructions if you would
pip._vendor.urllib3.util.ssl_ 
pip._vendor.urllib3.util.ssl_match_hostname - The match_hostname() function from Python 3.3.3, essential when using SSL.
pip._vendor.urllib3.util.ssltransport 
pyopenssl 
pyopenssl.openssl 
OpenGL.GLES2.OES.EGL_image_external_essl3 
Expected Tk Togl installation in /usr/lib/python3/dist-packages/OpenGL/Tk/togl-linux
Failure loading Togl package: can't find package Togl, on debian systems this is provided by `libtogl2`
OpenGL.raw.GLES2.OES.EGL_image_external_essl3 
OpenSSL - pyOpenSSL - A simple wrapper around the OpenSSL library
OpenSSL.SSL 
OpenSSL._util 
OpenSSL.crypto 
OpenSSL.debug 
OpenSSL.rand - PRNG management routines, thin wrappers.
OpenSSL.version - pyOpenSSL - A simple wrapper around the OpenSSL library
cryptography.hazmat.backends.openssl 
cryptography.hazmat.backends.openssl.aead 
cryptography.hazmat.backends.openssl.backend 
cryptography.hazmat.backends.openssl.ciphers 
cryptography.hazmat.backends.openssl.cmac 
cryptography.hazmat.backends.openssl.decode_asn1 
cryptography.hazmat.backends.openssl.dh 
cryptography.hazmat.backends.openssl.dsa 
cryptography.hazmat.backends.openssl.ec 
cryptography.hazmat.backends.openssl.ed25519 
cryptography.hazmat.backends.openssl.ed448 
cryptography.hazmat.backends.openssl.encode_asn1 
cryptography.hazmat.backends.openssl.hashes 
cryptography.hazmat.backends.openssl.hmac 
cryptography.hazmat.backends.openssl.ocsp 
cryptography.hazmat.backends.openssl.poly1305 
cryptography.hazmat.backends.openssl.rsa 
cryptography.hazmat.backends.openssl.utils 
cryptography.hazmat.backends.openssl.x25519 
cryptography.hazmat.backends.openssl.x448 
cryptography.hazmat.backends.openssl.x509 
cryptography.hazmat.bindings._openssl 
cryptography.hazmat.bindings.openssl 
cryptography.hazmat.bindings.openssl._conditional 
cryptography.hazmat.bindings.openssl.binding 

` Clearly something is missing and I cannot find how to get them included in 3.10. Any suggestions please, or at least the openSSL libraries?

Pete

[ghost]

Interesting thread/problem!

it also seems that the issue propagated itself into python 3.11 as well, affecting programs like certbot:

smart@open-neurosecurity:~$ sudo certbot renew --dry-run -v Traceback (most recent call last): File "/usr/bin/certbot", line 33, in sys.exit(load_entry_point('certbot==2.1.0', 'console_scripts', 'certbot')()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/bin/certbot", line 25, in importlib_load_entry_point return next(matches).load() ^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/importlib/metadata/init.py", line 202, in load module = import_module(match.group('module')) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 1204, in _gcd_import File "", line 1176, in _find_and_load File "", line 1147, in _find_and_load_unlocked File "", line 690, in _load_unlocked File "", line 940, in exec_module File "", line 241, in _call_with_frames_removed File "/usr/lib/python3/dist-packages/certbot/main.py", line 6, in from certbot._internal import main as internal_main File "/usr/lib/python3/dist-packages/certbot/_internal/main.py", line 19, in import josepy as jose File "/usr/lib/python3/dist-packages/josepy/init.py", line 40, in from josepy.json_util import ( File "/usr/lib/python3/dist-packages/josepy/json_util.py", line 14, in from OpenSSL import crypto File "/usr/lib/python3/dist-packages/OpenSSL/init.py", line 8, in from OpenSSL import SSL, crypto File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 9, in from OpenSSL._util import ( File "/usr/lib/python3/dist-packages/OpenSSL/_util.py", line 6, in from cryptography.hazmat.bindings.openssl.binding import Binding File "/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 228, in Binding.init_static_locks() File "/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 188, in init_static_locks cls._ensure_ffi_initialized() File "/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 176, in _ensure_ffi_initialized _openssl_assert( File "/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 90, in _openssl_assert raise InternalError( cryptography.exceptions.InternalError: Unknown OpenSSL error. This error is commonly encountered when another library is > not cleaning up the OpenSSL error stack. If you are using cryptography with another library that uses OpenSSL try disabling > it before reporting a bug. Otherwise please file an issue at https://github.com/pyca/cryptography/issues with information on > > how to reproduce this. ([_OpenSSLErrorWithText(code=310378599, lib=37, reason=103, reason_text=b'error:12800067:DSO > support routines::could not load the shared library'), _OpenSSLErrorWithText(code=310378599, lib=37, reason=103, > reason_text=b'error:12800067:DSO support routines::could not load the shared library'), _OpenSSLErrorWithText(code=126353445, lib=15, reason=524325, reason_text=b'error:07880025:common libcrypto routines::reason(524325)')])

[jamiegau]

Python is a good glue language for talking to legacy devices, however, I assume this issue I will describe is connected to this issue as well.

I have an embedded device running an old https server.

If I do a basic request such as in the following code..

import requests

def fetch_and_print_webpage(url):
    response = requests.get(url, verify=False)
    print(response.text)

def main():
    url = 'https://10.30.25.11:43758/?wsdl'
    fetch_and_print_webpage(url)

if __name__ == "__main__":
    main()

If running under python 3.9, it runs as expected. However, If I run it under 3.10 or 3.11. I get the following error.

python Test2.py
Traceback (most recent call last):
  File "/Users/jamieg/p310env/lib/python3.10/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/Users/jamieg/p310env/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
    conn.connect()
  File "/Users/jamieg/p310env/lib/python3.10/site-packages/urllib3/connection.py", line 653, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
  File "/Users/jamieg/p310env/lib/python3.10/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
  File "/Users/jamieg/p310env/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
  File "/Users/jamieg/p310env/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/opt/homebrew/Cellar/python@3.10/3.10.13_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 513, in wrap_socket
    return self.sslsocket_class._create(
  File "/opt/homebrew/Cellar/python@3.10/3.10.13_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 1104, in _create
    self.do_handshake()
  File "/opt/homebrew/Cellar/python@3.10/3.10.13_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 1375, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] ssl/tls alert handshake failure (_ssl.c:1007)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jamieg/p310env/lib/python3.10/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
  File "/Users/jamieg/p310env/lib/python3.10/site-packages/urllib3/connectionpool.py", line 491, in _make_request
    raise new_e
urllib3.exceptions.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] ssl/tls alert handshake failure (_ssl.c:1007)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/jamieg/p310env/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/Users/jamieg/p310env/lib/python3.10/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
  File "/Users/jamieg/p310env/lib/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.30.25.11', port=43758): Max retries exceeded with url: /?wsdl (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] ssl/tls alert handshake failure (_ssl.c:1007)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jamieg/Test2.py", line 12, in <module>
    main()
  File "/Users/jamieg/Test2.py", line 9, in main
    fetch_and_print_webpage(url)
  File "/Users/jamieg/Test2.py", line 4, in fetch_and_print_webpage
    response = requests.get(url, verify=False)
  File "/Users/jamieg/p310env/lib/python3.10/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/Users/jamieg/p310env/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/jamieg/p310env/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/jamieg/p310env/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/Users/jamieg/p310env/lib/python3.10/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='10.30.25.11', port=43758): Max retries exceeded with url: /?wsdl (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] ssl/tls alert handshake failure (_ssl.c:1007)')))

From digging up on this, this error is because the later version of Python are compiled against the newer version of OpenSSL. The newer version having cyphers depricated and REMOVED that the old embedded https server needs.

In effect, if I go over version 3.9 I lock myself out of talking to certain older embedded devices.

I was jumping to 3.11 as I wanted the speed boost. But this currently holds me back.

This could be a serious issue for those using Python to talk to older embedded devices.

I wanted to point this out and also see if I could get an expert opinion on if it would be possible to get a python 3.11 with older OpenSSL libraries in it? Or any other input in how to deal with this situation.