When outlier has Chinese characters, sqlite3 will raise ProgrammingError

[caitinggui]

log:

  File "/home/hadoop/project/lawplatform/env/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
ProgrammingError: (sqlite3.ProgrammingError) You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings. [SQL: u'INSERT INTO outliers (endpoint, request_values, request_headers, request_environment, request_url, cpu_percent, memory, stacktrace, execution_time, time) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)'] [parameters: ('llt.search', '{}', 'Referer: http://localhost:8082/\r\nOrigin: http://localhost:8082\r\nX-Real-Ip: 10.41.1.59\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWe ... (84 characters truncated) ... ose\r\nHost: 127.0.0.1:8182\r\nAccept: application/json, text/plain, */*\r\nAccept-Language: zh-CN,zh;q=0.9\r\nAccept-Encoding: gzip, deflate\r\n\r\n', '{\'HTTP_REFERER\': \'http://localhost:8082/\', \'SCRIPT_NAME\': \'\', \'REQUEST_METHOD\': \'GET\', \'PATH_INFO\': \'/llt/search\', \'HTTP_ORIGIN\': \ ... (1121 characters truncated) ... CN,zh;q=0.9\', \'uwsgi.version\': \'2.0.15\', \'wsgi.file_wrapper\': <built-in function uwsgi_sendfile>, \'HTTP_ACCEPT_ENCODING\': \'gzip, deflate\'}', 'http://127.0.0.1:8182/llt/search?category=book&page=0&q={"all"%3A"\xe7\x9b\x97\xe7\xaa\x83"}', '', '', '', 26.47709846496582, '2018-05-30 09:06:22.096822')] (Background on this error at: http://sqlalche.me/e/f405)

So, should we specifying utf-8 encoding when connect to sqlite3 with sqlalchemy?

[FlyingBird95]

Ideally, yes. If you have the name of the database configured in the configuration-file, you only have to append the name with '?charset=utf8'.

So, for example:

DATABASE=sqlite:////<path to your project>/dashboard.db

becomes

DATABASE=sqlite:////<path to your project>/dashboard.db?charset=utf8

Let me know if this works..

Patrick

[caitinggui]

Sorry, it can't work:

File "/home/hadoop/project/lawplatform/env/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 477, in __init__
    connection = pool._invoke_creator(self)
  File "/home/hadoop/project/lawplatform/env/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 106, in connect
    self.__connect(first_connect_check=True)
  File "/home/hadoop/project/lawplatform/env/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 671, in __connect
    connection = pool._invoke_creator(self)
  File "/home/hadoop/project/lawplatform/env/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 106, in connect
        return dialect.connect(*cargs, **cparams)
return dialect.connect(*cargs, **cparams)
  File "/home/hadoop/project/lawplatform/env/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect
    return self.dbapi.connect(*cargs, **cparams)
TypeError: 'charset' is an invalid keyword argument for this function

I use engine.connect().connection.text_factory = str in flask_monitoringdashboard/database/__init__.py, and this can help. But I don't know if it will affect other databases like MySQL and so on.

Reference links in stackoverflow: SQLite3 : You must not use 8-bit bytestrings unless you use a text_factory sqlalchemy - (ProgrammingError) can interpret 8-bit bytestrings

[bogdanp05]

We just released a new major version, 2.0.0 in #164. Let us know if you still get the issue in the new version.

[caitinggui]

@bogdanp05 sorry, I still get the issue in the new version. Flask-MonitoringDashboard==2.0.0