SpiderClub / weibospider

:zap: A distributed crawler for weibo, building with celery and requests.
MIT License
4.81k stars 1.21k forks source link

运行 python3 config/create_all.py 报错 #218

Closed Shallowhave closed 1 year ago

Shallowhave commented 2 years ago

开发者你好! 我使用的是centos7系统,已经按照该项目要求安装了mysql数据库,redis,并修改了spider.yaml配置文件 我在使用你的项目时运行python3 config/create_all.py报错如下


  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool.py", line 1122, in _do_get
    return self._pool.get(wait, self._timeout)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/queue.py", line 145, in get
    raise Empty
sqlalchemy.util.queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "config/create_all.py", line 16, in <module>
    create_all_table()
  File "config/create_all.py", line 12, in create_all_table
    metadata.create_all()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/schema.py", line 3949, in create_all
    tables=tables)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1928, in _run_visitor
    with self._optional_conn_ctx_manager(connection) as conn:
  File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1921, in _optional_conn_ctx_manager
    with self.contextual_connect() as conn:
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2112, in contextual_connect
    self._wrap_pool_connect(self.pool.connect, None),
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2147, in _wrap_pool_connect
    return fn()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool.py", line 387, in connect
    return _ConnectionFairy._checkout(self)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool.py", line 766, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool.py", line 516, in checkout
    rec = pool._do_get()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool.py", line 1138, in _do_get
    self._dec_overflow()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 187, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool.py", line 1135, in _do_get
    return self._create_connection()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool.py", line 333, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool.py", line 461, in __init__
    self.__connect(first_connect_check=True)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool.py", line 651, in __connect
    connection = pool._invoke_creator(self)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 393, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.7/site-packages/pymysql/__init__.py", line 90, in Connect
    return Connection(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/pymysql/connections.py", line 706, in __init__
    self.connect()
  File "/usr/local/lib/python3.7/site-packages/pymysql/connections.py", line 931, in connect
    self._get_server_information()
  File "/usr/local/lib/python3.7/site-packages/pymysql/connections.py", line 1269, in _get_server_information
    self.server_charset = charset_by_id(lang).name
  File "/usr/local/lib/python3.7/site-packages/pymysql/charset.py", line 38, in by_id
    return self._by_id[id]
KeyError: 255 ```
如你有空请帮我看看,万分感谢!
thekingofcity commented 2 years ago

看起来像是数据库字符集的问题,尝试升级mysql到最新版本并在创建database的时候指定字符集为utf8mb4。

另:项目的关键登录功能已不可用,除非以学习目的,否则不建议再投入更多精力。

Shallowhave commented 2 years ago

感谢你的答复!

Shallowhave commented 2 years ago

查询数据库的编码如下


+--------------------------------------+--------------------------------+
| Variable_name                        | Value                          |
+--------------------------------------+--------------------------------+
| character_set_client                 | utf8mb4                        |
| character_set_connection             | utf8mb4                        |
| character_set_database               | utf8mb4                        |
| character_set_filesystem             | binary                         |
| character_set_results                | utf8mb4                        |
| character_set_server                 | utf8mb4                        |
| character_set_system                 | utf8mb3                        |
| character_sets_dir                   | /usr/share/mysql-8.0/charsets/ |
| validate_password.special_char_count | 1                              |
+--------------------------------------+--------------------------------+
9 rows in set (0.00 sec)

发现默认采用的就是utf8mb4编码方式,另外重新设置weibo库的编码方式为utf8mb4,执行python3 config/create_all.py 依旧报该错误。
thekingofcity commented 2 years ago

好吧,那我也没有什么建议了

wrongww78319 commented 1 year ago

占楼致歉,我纯小白,首先感谢开发者的项目以及教程,但是演示视频的网盘链接失效了!!!呜呜呜

thekingofcity commented 1 year ago

占楼致歉,我纯小白,首先感谢开发者的项目以及教程,但是演示视频的网盘链接失效了!!!呜呜呜

😂就算你跑起来了微博现在也改版了,登录以及爬取的功能应该都失效了