TypeError: cannot use a string pattern on a bytes-like object

[jumpnett]

Description

When launching pgcli, I get the following error:

Exception in thread completion_refresh:
Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.11/threading.py", line 975, in run-mode  [F5] Explain: OFF      Refreshing completions...                                                                            
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3/dist-packages/pgcli/completion_refresher.py", line 68, in _bg_refresh
    refresher(completer, executor)
  File "/usr/lib/python3/dist-packages/pgcli/completion_refresher.py", line 109, in refresh_schemata
    completer.set_search_path(executor.search_path())
  File "/usr/lib/python3/dist-packages/pgcli/pgcompleter.py", line 306, in set_search_path
    self.search_path = self.escaped_names(search_path)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pgcli/pgcompleter.py", line 150, in escaped_names
    return [self.escape_name(name) for name in names]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pgcli/pgcompleter.py", line 150, in <listcomp>
    return [self.escape_name(name) for name in names]
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pgcli/pgcompleter.py", line 131, in escape_name
    (not self.name_pattern.match(name))
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: cannot use a string pattern on a bytes-like object

This error occurs when connecting to a PostgreSQL 9.3.4, but not a PostgreSQL 9.3.20 server.

Your environment

OS: Ubuntu 23.04 x86_64 pgcli Version: 3.5.0 pip freeze:

asn1crypto==1.5.1
attrs==22.2.0
Babel==2.10.3
blinker==1.5
Brlapi==0.8.4
certifi==2022.9.24
chardet==5.1.0
cli-helpers==2.3.0
click==8.1.3
cloud-init==23.1.2
colorama==0.4.6
command-not-found==0.3
configobj==5.0.8
cryptography==38.0.4
cupshelpers==1.0
dbus-python==1.3.2
defer==1.0.6
distro==1.8.0
distro-info==1.5
html5lib==1.1
httplib2==0.20.4
idna==3.3
importlib-metadata==4.12.0
jaraco.classes==3.2.1
jeepney==0.8.0
Jinja2==3.1.2
jsonpatch==1.32
jsonpointer==2.0
jsonschema==4.6.0
keyring==23.9.3
language-selector==0.1
launchpadlib==1.11.0
lazr.restfulclient==0.14.5
lazr.uri==1.0.6
louis==3.24.0
Mako==1.2.4.dev0
markdown-it-py==2.1.0
MarkupSafe==2.1.2
mdurl==0.1.2
mechanize==0.4.8
more-itertools==8.10.0
netifaces==0.11.0
oauthlib==3.2.2
olefile==0.46
pendulum==2.1.2
pexpect==4.8.0
pgcli==3.5.0
pgspecial==2.0.1
Pillow==9.4.0
prompt-toolkit==3.0.36
psycopg==3.1.7
psycopg-pool==3.1.5
psycopg2==2.9.5
ptyprocess==0.7.0
pycairo==1.20.1
pycups==2.0.1
Pygments==2.14.0
PyGObject==3.44.1
PyJWT==2.6.0
pyparsing==3.0.9
pyrsistent==0.18.1
pyserial==3.5
python-apt==2.5.3+ubuntu1
python-dateutil==2.8.2
python-debian==0.1.49+ubuntu2
pytz==2022.7.1
pytzdata==2020.1
pyxdg==0.28
PyYAML==6.0
reportlab==3.6.12
requests==2.28.1
rich==13.3.1
SecretStorage==3.3.3
setproctitle==1.3.1
six==1.16.0
sqlparse==0.4.2
systemd-python==235
tabulate==0.8.9
terminaltables==3.1.10
typing_extensions==4.4.0
ubuntu-advantage-tools==8001
ubuntu-drivers-common==0.0.0
ufw==0.36.1
unattended-upgrades==0.1
urllib3==1.26.12
wadllib==1.3.6
wcwidth==0.2.5
webencodings==0.5.1
xdg==5
xkit==0.0.0
zipp==1.0.0

[j-bennet]

Thanks for reporting @jumpnett. If you're able to submit a PR, it will be very welcome.

[pomo-mondreganto]

I'm experiencing the same error. Adding a few logs, I can see that pg_catalog name is bytes in my Postgres (v15) installation.

[nimbletactician]

I can replicate the error on my setup too. Thank you. @j-bennet

Details: Server: PostgreSQL 12.11 (Ubuntu 12.11-0ubuntu0.20.04.1) PgCli version: 3.5.0

[NasaGeek]

This may be occurring because your database/client's encoding is set to SQL_ASCII (which will prevent the underlying postgres library, psycopg, from decoding strings to Python str objects). Quick example:

import psycopg

# Assuming you have PG* env vars set
with psycopg.connect() as conn:
    result = conn.execute("SHOW SERVER_ENCODING").fetchone()
    print(result)
    result = conn.execute("SELECT 'str'").fetchone()[0]
    print(conn.info.encoding, result, type(result))

with psycopg.connect("client_encoding=utf8") as conn:
    result = conn.execute("SHOW SERVER_ENCODING").fetchone()
    print(result)
    result = conn.execute("SELECT 'str'").fetchone()[0]
    print(conn.info.encoding, result, type(result))

(b'SQL_ASCII',)
ascii b'str' <class 'bytes'>
('SQL_ASCII',)
utf-8 str <class 'str'>

[NasaGeek]

A workaround for the issue was to force the client encoding to utf8 via an environment variable when launching pgcli like so

PGCLIENTENCODING=utf8 pgcli

This isn't really a permanent solution, as it could result in trying to decode undecodable text from your database (since SQL_ASCII means it could contain anything, that's why psycopg won't decode it by default).

As a full fix, It's probably reasonable for pgcli to force its own decoding of returned bytes in specific cases like this where it's essentially fetching postgres identifiers (in this case, the elements of the search path) which I would always expect to be decodable. Instead of doing its own decoding, pgcli could also temporarily set the client encoding to utf8 or similar (e.g. SET client_encoding TO utf8) so that psycopg decodes the results