joseph-scu commented 11 months ago

Hello,

I am trying to install my Slack Bolt app to a certain workspace that contains a Japanese workspace as its name (for testing purposes I used 撮影課). I am using OAuth for authentication. When I attempt to install the app I get a 502 Bad Gateway Error. There doesn't seem to be any errors in the logs, however I noticed if I tried to use the app in Slack I do see an error (see below).

My code for hanlding OAUth is as below (I believe its taken mostly from the docs)


load_dotenv()

try:
    with engine.connect() as conn:
        conn.execute("select count(*) from slack_bots")
except Exception as e:
    installation_store.metadata.create_all(engine)
    oauth_state_store.metadata.create_all(engine)

oauth_settings = OAuthSettings(
    client_id=os.environ.get("SLACK_CLIENT_ID"),
    client_secret=os.environ.get("SLACK_CLIENT_SECRET"),
    scopes=["channels:history", "channels:join", "channels:read",
            "chat:write", "commands", "groups:history", "im:history", "users:read"],
    installation_store=installation_store,
    state_store=oauth_state_store,
    install_page_rendering_enabled=False
)

app = App(
    signing_secret=os.environ.get("SLACK_SIGNING_SECRET"),
    oauth_settings=oauth_settings
)

Reproducible in:

As below

The `slack_bolt` version

alembic==1.9.2 apt-xapian-index==0.49 attrs==22.2.0 autopep8==2.0.1 awscli==2.9.19 awscrt==1.0.0.dev0 catfish==4.16.4 certifi==2022.12.7 chardet==5.1.0 charset-normalizer==3.0.1 colorama==0.4.6 cryptography==38.0.4 cupshelpers==1.0 dbus-python==1.3.2 deprecation==2.0.7 distro==1.8.0 docutils==0.19 dtrx==8.4.0 gattlib==0.20210616 greenlet==2.0.1 gyp==0.1 idna==3.3 iniconfig==1.1.1 jmespath==1.0.1 lightdm-gtk-greeter-settings==1.2.2 lxml==4.9.2 Mako==1.2.4 MarkupSafe==2.1.2 more-itertools==8.10.0 mugshot==0.4.3 mypy==0.991 mypy-extensions==0.4.3 notify2==0.3 ntpsec==1.2.2 onboard==1.4.1 packaging==23.0 pdfarranger==1.9.2 pexpect==4.8.0 pikepdf==6.0.0+dfsg Pillow==9.4.0 pluggy==1.0.0+repack prompt-toolkit==3.0.38 ptyprocess==0.7.0 py==1.11.0 pyasn1==0.4.8 PyBluez==0.23 pycairo==1.20.1 pycodestyle==2.10.0 pycups==2.0.1 PyGObject==3.42.2 pysmbc==1.0.23 pytest==7.2.1 python-apt==2.6.0 python-dateutil==2.8.2 python-debian==0.1.49 pyxdg==0.28 PyYAML==6.0 requests==2.28.2 roman==3.3 rpl==1.14 ruamel.yaml==0.17.21 ruamel.yaml.clib==0.2.7 six==1.16.0 SQLAlchemy==2.0.0 systemd-python==235 typing_extensions==4.4.0 ufw==0.36.2 urllib3==1.26.12 wcwidth==0.2.5 xdg==5

Python runtime version

Python 3.11.4

OS info

1 SMP PREEMPT_DYNAMIC Debian 6.0.10-2 (2022-12-01)

Steps to reproduce:

(Share the commands to run, source code, and project settings (e.g., setup.py))

Have a workspace with Japanese characters.
Ensure you are using OAuth for authentication
Attempt to install the application (via /slack/install)
Notice that you will get a 502 Bad Gateway Error. The app will be visible in your list of apps however, but it won't be accessible

Expected result:

App installs to workspace

Actual result:

Application installs to the workspace, but not successfully. There is a 502 Bad Gateway Error, but the app is visible in workspaces list of apps. If you try to navigate to it you may see the following error in the logs:

return codecs.charmap_encode(input,errors,encoding_table)
| UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-2: character maps to <undefined>

seratch commented 11 months ago

Hi @joseph-scu, thanks for asking the question! However, with the limited information, we may not be able to help you out. I'd suggest enabling debug-level logging to see more details of the error first: https://slack.dev/bolt-python/concepts#logging

Also, you can share more details such as:

slack-bolt, slack-sdk versions
python runtime version / OS version
the infra you're running the app (e.g., AWS, Linux, whatever could be helpful for identifying the cause)
the code snippet you're using now

You can use our issue template to fill out these information (for future issue submissions, please use the template): https://github.com/slackapi/bolt-python/blob/main/.github/ISSUE_TEMPLATE/01_question.md

seratch commented 11 months ago

For your information, I have a Slack workspace with Japanese name for my personal use and I've installed a few bolt-python apps into it. However, I've never experienced the issue. Thus, just having the Japanese name may not be the cause of your issue.

joseph-scu commented 11 months ago

@seratch Thanks for that, I've updated my question. The reason I suspected it was due to the Japanese name was because the app installs fine if it is in English.

seratch commented 11 months ago

Would it be possible for you to identify where the UnicodeEncodeError is raised?

joseph-scu commented 11 months ago

This was the best I could do, I apologize for the long stack trace:

Exception happened during processing of request from ('172.20.0.1', 36920)
ku-slack-1  | Traceback (most recent call last):
ku-slack-1  |   File "/usr/local/lib/python3.8/socketserver.py", line 316, in _handle_request_noblock
ku-slack-1  |     self.process_request(request, client_address)
ku-slack-1  |   File "/usr/local/lib/python3.8/socketserver.py", line 347, in process_request
ku-slack-1  |     self.finish_request(request, client_address)
ku-slack-1 |   File "/usr/local/lib/python3.8/socketserver.py", line 360, in finish_request
ku-slack-1  |     self.RequestHandlerClass(request, client_address, self)
ku-slack-1  |   File "/usr/local/lib/python3.8/http/server.py", line 664, in __init__
ku-slack-1  |     super().__init__(*args, **kwargs)
ku-slack-1  |   File "/usr/local/lib/python3.8/socketserver.py", line 747, in __init__
ku-slack-1  |     self.handle()
ku-slack-1  |   File "/usr/local/lib/python3.8/http/server.py", line 435, in handle
ku-slack-1  |     self.handle_one_request()
ku-slack-1  |   File "/usr/local/lib/python3.8/http/server.py", line 423, in handle_one_request
ku-slack-1  |     method()
ku-slack-1  |   File "/usr/local/lib/python3.8/site-packages/slack_bolt/app/app.py", line 1349, in do_GET
ku-slack-1  |     bolt_resp = _bolt_oauth_flow.handle_callback(bolt_req)
ku-slack-1  |   File "/usr/local/lib/python3.8/site-packages/slack_bolt/oauth/oauth_flow.py", line 278, in handle_callback
ku-slack-1  |     self.store_installation(request, installation)
ku-slack-1  |   File "/usr/local/lib/python3.8/site-packages/slack_bolt/oauth/oauth_flow.py", line 361, in store_installation
ku-slack-1  |     self.settings.installation_store.save(installation)
ku-slack-1  |   File "/usr/local/lib/python3.8/site-packages/slack_sdk/oauth/installation_store/sqlalchemy/__init__.py", line 158, in save
ku-slack-1  |     conn.execute(self.installations.insert(), i)
ku-slack-1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1380, in execute
ku-slack-1  |     return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
ku-slack-1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
ku-slack-1  |     return connection._execute_clauseelement(
ku-slack-1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
ku-slack-1  |     ret = self._execute_context(
ku-slack-1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
ku-slack-1  |     self._handle_dbapi_exception(
ku-slack-1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2128, in _handle_dbapi_exception
ku-slack-1  |     util.raise_(exc_info[1], with_traceback=exc_info[2])
ku-slack-1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
ku-slack-1  |     raise exception
ku-slack-1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
ku-slack-1  |     self.dialect.do_execute(
ku-slack-1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
ku-slack-1  |     cursor.execute(statement, parameters)
ku-slack-1  |   File "/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 199, in execute
ku-slack-1  |     args = tuple(map(db.literal, args))
ku-slack-1  |   File "/usr/local/lib/python3.8/site-packages/MySQLdb/connections.py", line 275, in literal
ku-slack-1  |     s = self.string_literal(o.encode(self.encoding))
ku-slack-1  |   File "/usr/local/lib/python3.8/encodings/cp1252.py", line 12, in encode
ku-slack-1  |     return codecs.charmap_encode(input,errors,encoding_table)
ku-slack-1  | UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-2: character maps to <undefined>

seratch commented 11 months ago

It seems that the error occurs during SQLAlchemy's database operations with MySQL. You can try the answer here: https://stackoverflow.com/questions/63220132/sqlalchemy-insert-to-mysql-db-unicodeencodeerror-for-cyrlic-data

joseph-scu commented 11 months ago

@seratch Thank you very much for providing the link!

I changed my engine_url to the following:


engine_url = URL.create(
    os.environ.get("DB_DRIVER", ""),
    username=os.environ.get("DB_USERNAME"),
    password=os.environ.get("DB_PASSWORD"),
    host=os.environ.get("DB_HOST"),
    database=os.environ.get("DB_DATABASE")
)

engine_url = engine_url.update_query_pairs([("charset", "utf8mb4")])

However, this alone was not enough. I also had to modify the character set of team_name in both the slack_bots and slack_installation columns to utf8mb4, as well as collation to utf8mb4_unicode_ci. With these changes it seems to install as it should.

seratch commented 11 months ago

@joseph-scu Thanks for sharing this. It seems that the built-in SQLAlchemy-based installation store (https://github.com/slackapi/python-slack-sdk/blob/v3.23.0/slack_sdk/oauth/installation_store/sqlalchemy/__init__.py) can be improved to better support MySQL / MariaDB in future versions. I will check it out when I have the bandwidth for it.

Since you've identified the cause and resolved the issue, let me close this issue now. Thanks again for writing in!

slackapi / bolt-python

Slack bolt installation to workspace fails if workspace name is in Japanese #974

Reproducible in:

The `slack_bolt` version

Python runtime version

OS info

1 SMP PREEMPT_DYNAMIC Debian 6.0.10-2 (2022-12-01)

Steps to reproduce:

Expected result:

Actual result:

slackapi / bolt-python

Slack bolt installation to workspace fails if workspace name is in Japanese #974

Reproducible in:

The slack_bolt version

Python runtime version

OS info

1 SMP PREEMPT_DYNAMIC Debian 6.0.10-2 (2022-12-01)

Steps to reproduce:

Expected result:

Actual result:

The `slack_bolt` version