search

JackTheMico / ruia-peewee-async

A Ruia plugin that uses the peewee-async to store data to MySQL or PostgreSQL or both of them
Other
1 stars 1 forks source link

When I run the douban example , aiopg warned that a connection not close,aiomysql warned that event loop is closed #17

Closed RuixiangS closed 1 year ago

RuixiangS commented 2 years ago

With the same package version in ruia-peewee-async.When I run the example douban.py,there have some issue. System: Win 10 pro 19043.2130

aiopg warned that Unclosed 1 connections.

[2022:10:17 12:16:51] INFO Ruia Spider started! [2022:10:17 12:16:51] WARNING Ruia Ruia tried to use loop.add_signal_handler but it is not implemented on this platform. [2022:10:17 12:16:51] WARNING Ruia Ruia tried to use loop.add_signal_handler but it is not implemented on this platform. [2022:10:17 12:16:51] INFO Ruia Worker started: 2213062427328 [2022:10:17 12:16:51] INFO Ruia Worker started: 2213062427488 [2022:10:17 12:16:51] INFO Request <GET: https://movie.douban.com/chart> [2022:10:17 12:16:52] INFO Ruia <RuiaPeeweeAsync: Success insert {'url': 'https://movie.douban.com/subject/35118954/', 'title': '子弹列车'} into database: ['POSTGRES']> ........ [2022:10:17 12:16:52] INFO Ruia <RuiaPeeweeAsync: Success insert {'url': 'https://movie.douban.com/subject/2042941/', 'title': '养鬼吃人'} into database: ['POSTGRES']> [2022:10:17 12:16:52] INFO Ruia Stopping spider: Ruia [2022:10:17 12:16:52] INFO Ruia Total requests: 1 [2022:10:17 12:16:52] INFO Ruia Time usage: 0:00:00.629152 [2022:10:17 12:16:52] INFO Ruia Spider finished! D:\Anaconda\envs\dash_feffery\lib\site-packages\aiopg\pool.py:478: ResourceWarning: Unclosed 1 connections in <aiopg.pool.Pool object at 0x0000020344D632E0>

aiomysql warned that event loop is closed in advance.

Exception ignored in: <function Connection.__del__ at 0x00000274394ADA60> Traceback (most recent call last): File "D:\Anaconda\envs\dash_feffery\lib\site-packages\aiomysql\connection.py", line 1120, in __del__ File "D:\Anaconda\envs\dash_feffery\lib\site-packages\aiomysql\connection.py", line 339, in close File "D:\Anaconda\envs\dash_feffery\lib\asyncio\selector_events.py", line 692, in close File "D:\Anaconda\envs\dash_feffery\lib\asyncio\base_events.py", line 719, in call_soon File "D:\Anaconda\envs\dash_feffery\lib\asyncio\base_events.py", line 508, in _check_closed RuntimeError: Event loop is closed

The data could be inserted into the table,but occur these error.

JackTheMico commented 2 years ago

Sorry for the delay of response, I'll look into those when I was free from work. Thank you for creating the issue, I'm very happy because this is the first time some create an issue in my repo.😊

RuixiangS commented 1 year ago

Sorry for the delay of response, I'll look into those when I was free from work. Thank you for creating the issue, I'm very happy because this is the first time some create an issue in my repo.😊

HaHa，I have found your project and I test it, It could store the data in the database,but there are some issue occurs when the ruia's spider finished. It seems that aiopg/aiomysql was not closed in advance.Your code is helpful for someone who want to store data in MySQL/Postgresql by ruia.Thank for your plugin's devlopment!

JackTheMico commented 1 year ago

@RuixiangS It's fixed, I add a before_stop function to close database managers properly and the error or warning was gone.

See the douban.py for details.

RuixiangS commented 1 year ago

OK, I 'm preparing to test it.Thanks a lot!