dataabc / weibo-crawler

新浪微博爬虫,用python爬取新浪微博数据,并下载微博图片和微博视频
3.22k stars 737 forks source link

微博内容(Weibo.text)最大长度报错问题 #423

Closed wendywjhu closed 3 months ago

wendywjhu commented 3 months ago

weibo.text上限现在是5000了,代码建表的时候限制2000,超出的时候会报错,建议调整一下DB脚本。

image

报错信息:

(1406, "Data too long for column 'text' at row 1") Traceback (most recent call last): File "weibo-crawler-20240330/weibo.py", line 1515, in mysql_insert cursor.executemany(sql, [tuple(data.values()) for data in data_list]) File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pymysql/cursors.py", line 191, in executemany self.rowcount = sum(self.execute(query, arg) for arg in args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pymysql/cursors.py", line 191, in self.rowcount = sum(self.execute(query, arg) for arg in args) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pymysql/cursors.py", line 153, in execute result = self._query(query) ^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pymysql/cursors.py", line 322, in _query conn.query(q) File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pymysql/connections.py", line 558, in query self._affected_rows = self._read_query_result(unbuffered=unbuffered) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pymysql/connections.py", line 822, in _read_query_result result.read() File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pymysql/connections.py", line 1200, in read first_packet = self.connection._read_packet() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pymysql/connections.py", line 772, in _read_packet packet.raise_for_error() File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pymysql/protocol.py", line 221, in raise_for_error err.raise_mysql_exception(self._data) File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pymysql/err.py", line 143, in raise_mysql_exception raise errorclass(errno, errval) pymysql.err.DataError: (1406, "Data too long for column 'text' at row 1")

dataabc commented 3 months ago

感谢提醒。