search

infiniflow / ragflow

RAGFlow is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding.
https://ragflow.io
Apache License 2.0
12.68k stars 1.24k forks source link

[Bug]: [ERROR]Internal server error: [Errno 5] Input/output error #1425

Open acbp1379 opened 2 weeks ago

acbp1379 commented 2 weeks ago

Is there an existing issue for the same bug?

Branch name

main

Commit ID

13

Other environment information

No response

Actual behavior

Task has been received. Page(1~13): OCR is running... Page(1~13): OCR finished Page(1~13): Layout analysis finished. Page(1~13): Table analysis finished. Page(1~13): Text merging finished Page(1~13): Finished slicing files(9). Start to embedding the content. Page(1~13): Finished embedding(1.02)! Start to build index! Page(1~13): Done! Task has been received. Page(13~25): OCR is running... Page(13~25): OCR finished Page(13~25): [ERROR]Internal server error: [Errno 5] Input/output error Task has been received. Page(25~37): OCR is running... Page(25~37): OCR finished Page(25~37): [ERROR]Internal server error: [Errno 5] Input/output error Task has been received.

Expected behavior

No response

Steps to reproduce

本地已经搭建好了xinference的embedding服务，然后上传一个pdf文件，解析过程中出现问题

Additional information

No response

KevinHuSh commented 2 weeks ago

Could you show the error stack in log? tail -f docker/ragflow-logs/api/*.log

shuaills commented 2 weeks ago

Same Issue here

shuaills commented 2 weeks ago

2024-07-09T10:51:19.940495Z 9211 [Note] Aborted connection 9211 to db: 'rag_flow' user: 'root' host: '10.115.0.43' (Got an error reading communication packets)

2024-07-09T10:51:26.572197Z 9209 [Note] Aborted connection 9209 to db: 'rag_flow' user: 'root' host: '10.115.0.43' (Got an error reading communication packets)

2024-07-09T10:51:32.749717Z 9210 [Note] Aborted connection 9210 to db: 'rag_flow' user: 'root' host: '10.115.0.43' (Got an error reading communication packets)

It seems a problem with embedded data format and mysql

KevinHuSh commented 2 weeks ago

Embeddings are not stored in Mysql. It seems mysql is down, doesn't it?

shuaills commented 2 weeks ago

Embeddings are not stored in Mysql. It seems mysql is down, doesn't it?

MySQL worked well on other tasks

shuaills commented 2 weeks ago

2024-07-09T11:13:38.611687Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2024-07-09T11:13:38.613493Z 0 [Note] mysqld (mysqld 5.7.18) starting as process 1 ... 2024-07-09T11:13:38.616649Z 0 [Note] InnoDB: PUNCH HOLE support available 2024-07-09T11:13:38.616670Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2024-07-09T11:13:38.616673Z 0 [Note] InnoDB: Uses event mutexes 2024-07-09T11:13:38.616675Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 2024-07-09T11:13:38.616676Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.3 2024-07-09T11:13:38.616678Z 0 [Note] InnoDB: Using Linux native AIO 2024-07-09T11:13:38.617271Z 0 [Note] InnoDB: Number of pools: 1 2024-07-09T11:13:38.617413Z 0 [Note] InnoDB: Using CPU crc32 instructions 2024-07-09T11:13:38.619226Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M 2024-07-09T11:13:38.626670Z 0 [Note] InnoDB: Completed initialization of buffer pool 2024-07-09T11:13:38.629281Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2024-07-09T11:13:38.641357Z 0 [Note] InnoDB: Highest supported file format is Barracuda. 2024-07-09T11:13:38.653713Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2024-07-09T11:13:38.653815Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2024-07-09T11:13:38.711710Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2024-07-09T11:13:38.713805Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active. 2024-07-09T11:13:38.713829Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active. 2024-07-09T11:13:38.715388Z 0 [Note] InnoDB: Waiting for purge to start 2024-07-09T11:13:38.765598Z 0 [Note] InnoDB: 5.7.18 started; log sequence number 19274240976 2024-07-09T11:13:38.766014Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2024-07-09T11:13:38.766203Z 0 [Note] Plugin 'FEDERATED' is disabled. 2024-07-09T11:13:38.771945Z 0 [Note] InnoDB: Buffer pool(s) load completed at 240709 19:13:38 2024-07-09T11:13:38.776035Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them. 2024-07-09T11:13:38.776096Z 0 [Warning] Failed to set up SSL because of the following SSL library error: TLS version is invalid 2024-07-09T11:13:38.776934Z 0 [Note] Server hostname (bind-address): ''; port: 3306 2024-07-09T11:13:38.776990Z 0 [Note] IPv6 is available. 2024-07-09T11:13:38.777001Z 0 [Note] - '::' resolves to '::'; 2024-07-09T11:13:38.777034Z 0 [Note] Server socket created on IP: '::'. 2024-07-09T11:13:38.780464Z 0 [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode. 2024-07-09T11:13:38.780531Z 0 [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode. 2024-07-09T11:13:38.780577Z 0 [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode. 2024-07-09T11:13:38.780598Z 0 [Warning] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode. 2024-07-09T11:13:38.786359Z 0 [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode. 2024-07-09T11:13:38.797007Z 0 [Note] Event Scheduler: Loaded 0 events 2024-07-09T11:13:38.797336Z 0 [Note] Execution of init_file '/data/application/init.sql' started. 2024-07-09T11:13:38.800510Z 0 [Note] Execution of init_file '/data/application/init.sql' ended. 2024-07-09T11:13:38.800701Z 0 [Note] mysqld: ready for connections. Version: '5.7.18' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server (GPL) 2024-07-09T11:13:38.800719Z 0 [Note] Executing 'SELECT FROM INFORMATION_SCHEMA.TABLES;' to get a list of tables using the deprecated partition engine. You may use the startup option '--disable-partition-engine-check' to skip this check. 2024-07-09T11:13:38.800723Z 0 [Note] Beginning of list of non-natively partitioned tables 2024-07-09T11:13:38.821667Z 0 [Note] End of list of non-natively partitioned tables 2024-07-09T11:15:30.510330Z 21 [Note] Aborted connection 21 to db: 'rag_flow' user: 'root' host: '10.115.0.43' (Got an error reading communication packets) 2024-07-09T11:15:32.792134Z 19 [Note] Aborted connection 19 to db: 'rag_flow' user: 'root' host: '10.115.0.43' (Got an error reading communication packets) 2024-07-09T11:15:47.176978Z 28 [Note] Aborted connection 28 to db: 'rag_flow' user: 'root' host: '10.115.0.43' (Got an error reading communication packets) 2024-07-09T11:15:57.088738Z 31 [Note] Aborted connection 31 to db: 'rag_flow' user: 'root' host: '10.115.0.43' (Got an error reading communication packets) 2024-07-09T13:45:20.127619Z 1240 [Note] Aborted connection 1240 to db: 'rag_flow' user: 'root' host: '10.115.0.43' (Got an error reading communication packets) 2024-07-09T19:15:52.833863Z 27 [Note] Aborted connection 27 to db: 'rag_flow' user: 'root' host: '10.115.0.43' (Got timeout reading communication packets) 2024-07-10T09:30:10.725104Z 10661 [Note] Aborted connection 10661 to db: 'rag_flow' user: 'root' host: '10.115.0.43' (Got an error reading communication packets) 2024-07-10T09:37:44.212633Z 10726 [Note] Aborted connection 10726 to db: 'rag_flow' user: 'root' host: '10.115.0.43' (Got an error reading communication packets) 2024-07-10T09:38:01.281744Z 10730 [Note] Aborted connection 10730 to db: 'rag_flow' user: 'root' host: '10.115.0.43' (Got an error reading communication packets)

acbp1379 commented 2 weeks ago

Could you show the error stack in log? tail -f docker/ragflow-logs/api/*.log

Fail put 1720a0503fac11efb10f78ac4456d7f0/1.txt: S3 operation failed; code: NoSuchKey, message: Object does not exist, resource: /1720a0503fac11efb10f78ac4456d7f0/%E6%B6%88%E9%98%B2%E9%9A%90%E6%82%A3.txt, request_id: 17E13C5030E80368, host_id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8, bucket_name: 1720a0503fac11efb10f78ac4456d7f0, object_name: 1.txt

Hello, I checked the logs and tested several times and found that the file uploading process is very unstable, the same txt file, the first upload doesn't work, a few more uploads and it uploads. In this case, it is very difficult to handle in large files, in the middle of a part of the parsing failure will report an error. The error message seems to be related to minio. Looking forward to your reply

shuaills commented 1 week ago

Problem solved.

[ERROR]Internal server error: [Errno 5] Input/output error. The reason for this error is some outdated task_executor.py are taking tasks but they won't process them.

You need to terminate the entrypoint.sh. It has

for ((i=0;i<WS;i++))
do
  task_exe  &
done

which would restart those task_executor.py all the time, including the outdated ones.

grep those entrypoint.sh and task_executor.py, kill them. terminate both the backend and frontend, restart them.

Then it should work.