Sist2-admin indexing freezes

[Badlee2020]

Device Information (please complete the following information):

• Deployment: Docker
• Browser (if relevant): chrome
• SIST2 Version: 3.0.6
• Elasticsearch Version (if relevant) : 7.14.9

Command with arguments Sist2-admin indexing

Describe the bug While indexing i receive the following error multiple times then the indexing "freezes". After waiting 3 hours, there is no change. 2023-04-25 07:52:12 [DEBUG cli.c] arg es_url=http://elasticsearch:9200 2023-04-25 07:52:12 [DEBUG cli.c] arg es_index=sist2 2023-04-25 07:52:12 [DEBUG cli.c] arg es_insecure_ssl=0 2023-04-25 07:52:12 [DEBUG cli.c] arg index_path=/sist2-admin/scan-Jessika-2023-04-25 07:08:37.108885.sist2 2023-04-25 07:52:12 [DEBUG cli.c] arg script_path=(null) 2023-04-25 07:52:12 [DEBUG cli.c] arg async_script=0 2023-04-25 07:52:12 [DEBUG cli.c] arg print=0 2023-04-25 07:52:12 [DEBUG cli.c] arg es_mappings_path=(null) 2023-04-25 07:52:12 [DEBUG cli.c] arg es_mappings=(null) 2023-04-25 07:52:12 [DEBUG cli.c] arg es_settings_path=(null) 2023-04-25 07:52:12 [DEBUG cli.c] arg es_settings=(null) 2023-04-25 07:52:12 [DEBUG cli.c] arg batch_size=100 2023-04-25 07:52:12 [DEBUG cli.c] arg force_reset=0 2023-04-25 07:52:12 [INFO elastic.c] Elasticsearch version is 7.17.9 (supported=1, legacy=0) 2023-04-25 07:52:12 [DEBUG database.c] Opening database /sist2-admin/scan-Jessika-2023-04-25 07:08:37.108885.sist2 (0) 2023-04-25 07:52:12 [DEBUG database.c] Closing database /sist2-admin/scan-Jessika-2023-04-25 07:08:37.108885.sist2 2023-04-25 07:52:12 [DEBUG main.c] Index version 3.0.6 2023-04-25 07:52:12 [DEBUG database.c] Initializing database /dev/shm/sist2-ipc-64.sqlite 2023-04-25 07:52:12 [INFO tpool.c] Starting thread pool with 5 threads 2023-04-25 07:52:12 [DEBUG database.c] Opening database /dev/shm/sist2-ipc-64.sqlite (1) 2023-04-25 07:52:12 [DEBUG database.c] Opening database /dev/shm/sist2-ipc-64.sqlite (1) 2023-04-25 07:52:12 [DEBUG database.c] Opening database /dev/shm/sist2-ipc-64.sqlite (1) 2023-04-25 07:52:12 [DEBUG database.c] Opening database /dev/shm/sist2-ipc-64.sqlite (1) 2023-04-25 07:52:12 [DEBUG database.c] Opening database /dev/shm/sist2-ipc-64.sqlite (1) 2023-04-25 07:52:12 [DEBUG database.c] Opening database /dev/shm/sist2-ipc-64.sqlite (2) 2023-04-25 07:52:12 [DEBUG database.c] Opening database /sist2-admin/scan-Jessika-2023-04-25 07:08:37.108885.sist2 (0) 2023-04-25 07:52:13 [DEBUG elastic.c] Indexed 100 documents (401kB) <200> 2023-04-25 07:52:13 [DEBUG elastic.c] Indexed 100 documents (592kB) <200> 2023-04-25 07:52:13 [DEBUG elastic.c] Indexed 100 documents (533kB) <200> 2023-04-25 07:52:13 [ERROR elastic.c] { "index": { "_index": "sist2", "_type": "_doc", "_id": "000d7b5a287bf69e368336533a1c9236", "_version": 2, "result": "updated", "_shards": { "total": 1, "successful": 1, "failed": 0 }, "_seq_no": 113402, "_primary_term": 5, "status": 200 } } 2023-04-25 07:52:13 [ERROR elastic.c] { "index": { "_index": "sist2", "_type": "_doc", "_id": "071596b62ca58c8917750573803b1fce", "_version": 2, "result": "updated", "_shards": { "total": 1, "successful": 1, "failed": 0 }, "_seq_no": 114195, "_primary_term": 5, "status": 200 } }

[simon987]

Thanks for flagging this

Could you try the following:

1. Create this dockerfile:

   
   FROM simon987/sist2:3.0.6-x64-linux

RUN curl -o /root/sist2 https://files.simon987.net/.gate/sist2/simon987_sist2/master_308_35cfd3b3b1fd2d4e70f8dbcaa8bcd50e53b66675/sist2-x64-linux-debug

`docker build . -t sist2-debug`

2. In your compose file, replace the sist2 image like this:

```diff
- image: simon987/sist2:3.0.4-x64-linux
+ image: sist2-debug

3. Re-run the index task

[Badlee2020]

Hi, everything seemed to work for the scan. At the end, it took a while then I got the following error:

[ERROR ] ================================================================= [ERROR ] ==18==ERROR: LeakSanitizer: detected memory leaks [ERROR ] Direct leak of 9052 byte(s) in 24 object(s) allocated from: [ERROR ] #0 0x55608046a280 in interceptor_malloc (/root/sist2+0x311280) [ERROR ] #1 0x5560814d83a9 in xmalloc /drone/src/third-party/libscan/third-party/antiword/src/xmalloc.c:37 [ERROR ] #2 0x5560814cc8a3 in pucAnalyseSummaryInfoHeader /drone/src/third-party/libscan/third-party/antiword/src/summary.c:414 [ERROR ] #3 0x5560814cca73 in vSetSummaryInfoOLE /drone/src/third-party/libscan/third-party/antiword/src/summary.c:631 [ERROR ] #4 0x5560814f68f7 in vGetPropertyInfo /drone/src/third-party/libscan/third-party/antiword/src/properties.c:147 [ERROR ] #5 0x5560814d7747 in iInitDocumentOLE /drone/src/third-party/libscan/third-party/antiword/src/wordole.c:780 [ERROR ] #6 0x5560814d0b20 in bWordDecryptor /drone/src/third-party/libscan/third-party/antiword/src/word2text.c:665 [ERROR ] #7 0x5560806b333e in parse_msdoc_text /drone/src/third-party/libscan/libscan/msdoc/msdoc.c:51 [ERROR ] #8 0x5560806b353d in parse_msdoc /drone/src/third-party/libscan/libscan/msdoc/msdoc.c:88 [ERROR ] #9 0x5560804dd413 in parse /drone/src/src/parsing/parse.c:217 [ERROR ] #10 0x5560804cfe01 in worker_thread_loop /drone/src/src/tpool.c:88 [ERROR ] #11 0x5560804d0b84 in tpool_worker /drone/src/src/tpool.c:245 [ERROR ] #12 0x7fdcf54fab42 (/lib/x86_64-linux-gnu/libc.so.6+0x94b42) [ERROR ] Direct leak of 9052 byte(s) in 24 object(s) allocated from: [ERROR ] #0 0x55608046a280 in interceptor_malloc (/root/sist2+0x311280) [ERROR ] #1 0x5560814d83a9 in xmalloc /drone/src/third-party/libscan/third-party/antiword/src/xmalloc.c:37 [ERROR ] #2 0x5560814cc8a3 in pucAnalyseSummaryInfoHeader /drone/src/third-party/libscan/third-party/antiword/src/summary.c:414 [ERROR ] #3 0x5560814cca73 in vSetSummaryInfoOLE /drone/src/third-party/libscan/third-party/antiword/src/summary.c:631 [ERROR ] #4 0x5560814f68f7 in vGetPropertyInfo /drone/src/third-party/libscan/third-party/antiword/src/properties.c:147 [ERROR ] #5 0x5560814d7747 in iInitDocumentOLE /drone/src/third-party/libscan/third-party/antiword/src/wordole.c:780 [ERROR ] #6 0x5560806b3143 in parse_msdoc_text /drone/src/third-party/libscan/libscan/msdoc/msdoc.c:39 [ERROR ] #7 0x5560806b353d in parse_msdoc /drone/src/third-party/libscan/libscan/msdoc/msdoc.c:88 [ERROR ] #8 0x5560804dd413 in parse /drone/src/src/parsing/parse.c:217 [ERROR ] #9 0x5560804cfe01 in worker_thread_loop /drone/src/src/tpool.c:88 [ERROR ] #10 0x5560804d0b84 in tpool_worker /drone/src/src/tpool.c:245 [ERROR ] #11 0x7fdcf54fab42 (/lib/x86_64-linux-gnu/libc.so.6+0x94b42) [ERROR ] Direct leak of 380 byte(s) in 1 object(s) allocated from: [ERROR ] #0 0x55608046a280 in interceptor_malloc (/root/sist2+0x311280) [ERROR ] #1 0x5560814d83a9 in xmalloc /drone/src/third-party/libscan/third-party/antiword/src/xmalloc.c:37 [ERROR ] #2 0x5560814cc8a3 in pucAnalyseSummaryInfoHeader /drone/src/third-party/libscan/third-party/antiword/src/summary.c:414 [ERROR ] #3 0x5560814cca73 in vSetSummaryInfoOLE /drone/src/third-party/libscan/third-party/antiword/src/summary.c:631 [ERROR ] #4 0x5560814f68f7 in vGetPropertyInfo /drone/src/third-party/libscan/third-party/antiword/src/properties.c:147 [ERROR ] #5 0x5560814d7747 in iInitDocumentOLE /drone/src/third-party/libscan/third-party/antiword/src/wordole.c:780 [ERROR ] #6 0x5560814d0b20 in bWordDecryptor /drone/src/third-party/libscan/third-party/antiword/src/word2text.c:665 [ERROR ] #7 0x5560806b333e in parse_msdoc_text /drone/src/third-party/libscan/libscan/msdoc/msdoc.c:51 [ERROR ] #8 0x5560806b353d in parse_msdoc /drone/src/third-party/libscan/libscan/msdoc/msdoc.c:88 [ERROR ] #9 0x5560804dd413 in parse /drone/src/src/parsing/parse.c:217 [ERROR ] #10 0x5560806946e8 in parse_archive /drone/src/third-party/libscan/libscan/arc/arc.c:234 [ERROR ] #11 0x5560804dd2c7 in parse /drone/src/src/parsing/parse.c:200 [ERROR ] #12 0x5560804cfe01 in worker_thread_loop /drone/src/src/tpool.c:88 [ERROR ] #13 0x5560804d0b84 in tpool_worker /drone/src/src/tpool.c:245 [ERROR ] #14 0x7fdcf54fab42 (/lib/x86_64-linux-gnu/libc.so.6+0x94b42) [ERROR ] Direct leak of 380 byte(s) in 1 object(s) allocated from: [ERROR ] #0 0x55608046a280 in __interceptor_malloc (/root/sist2+0x311280) [ERROR ] #1 0x5560814d83a9 in xmalloc /drone/src/third-party/libscan/third-party/antiword/src/xmalloc.c:37 [ERROR ] #2 0x5560814cc8a3 in pucAnalyseSummaryInfoHeader /drone/src/third-party/libscan/third-party/antiword/src/summary.c:414 [ERROR ] #3 0x5560814cca73 in vSetSummaryInfoOLE /drone/src/third-party/libscan/third-party/antiword/src/summary.c:631 [ERROR ] #4 0x5560814f68f7 in vGetPropertyInfo /drone/src/third-party/libscan/third-party/antiword/src/properties.c:147 [ERROR ] #5 0x5560814d7747 in iInitDocumentOLE /drone/src/third-party/libscan/third-party/antiword/src/wordole.c:780 [ERROR ] #6 0x5560806b3143 in parse_msdoc_text /drone/src/third-party/libscan/libscan/msdoc/msdoc.c:39 [ERROR ] #7 0x5560806b353d in parse_msdoc /drone/src/third-party/libscan/libscan/msdoc/msdoc.c:88 [ERROR ] #8 0x5560804dd413 in parse /drone/src/src/parsing/parse.c:217 [ERROR ] #9 0x5560806946e8 in parse_archive /drone/src/third-party/libscan/libscan/arc/arc.c:234 [ERROR ] #10 0x5560804dd2c7 in parse /drone/src/src/parsing/parse.c:200 [ERROR ] #11 0x5560804cfe01 in worker_thread_loop /drone/src/src/tpool.c:88 [ERROR ] #12 0x5560804d0b84 in tpool_worker /drone/src/src/tpool.c:245 [ERROR ] #13 0x7fdcf54fab42 (/lib/x86_64-linux-gnu/libc.so.6+0x94b42) [ERROR ] Direct leak of 160 byte(s) in 10 object(s) allocated from: [ERROR ] #0 0x55608046a280 in interceptor_malloc (/root/sist2+0x311280) [ERROR ] #1 0x5560804d1e69 in tpool_start /drone/src/src/tpool.c:359 [ERROR ] #2 0x5560804b535c in sist2_scan /drone/src/src/main.c:255 [ERROR ] #3 0x5560804bb8df in main /drone/src/src/main.c:598 [ERROR ] #4 0x7fdcf548fd8f (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) [ERROR ] Direct leak of 48 byte(s) in 3 object(s) allocated from: [ERROR ] #0 0x55608046a280 in __interceptor_malloc (/root/sist2+0x311280) [ERROR ] #1 0x556080605bdd in database_create_treemap_iterator /drone/src/src/database/database_stats.c:22 [ERROR ] #2 0x556080606ff5 in database_generate_stats /drone/src/src/database/database_stats.c:139 [ERROR ] #3 0x5560804b577d in sist2_scan /drone/src/src/main.c:284 [ERROR ] #4 0x5560804bb8df in main /drone/src/src/main.c:598 [ERROR ] #5 0x7fdcf548fd8f (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) [ERROR ] SUMMARY: AddressSanitizer: 19072 byte(s) leaked in 63 allocation(s). [ADMIN ] Process returned non-zero exit code (1)

the scanning task then showed as failing without going to indexing. When i tried a task that previously was sucessfull, it also failed.

[Badlee2020]

ran it again, and recieved the following error:

[ERROR ] AddressSanitizer:DEADLYSIGNAL [ERROR ] ================================================================= [ERROR ] ==14==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000028 (pc 0x564ed57762e3 bp 0x6070004be820 sp 0x7f53935d4f80 T2) [ERROR ] ==14==The signal is caused by a READ memory access. [ERROR ] ==14==Hint: address points to the zero page. [ERROR ] #0 0x564ed57762e2 in fz_free (/root/sist2+0xcd22e2) [ERROR ] #1 0x564ed59100b7 in opj_t1_clbl_decode_processor /vcpkg/buildtrees/openjpeg/src/cda136d811-8d2d9326d7.clean/src/lib/openjp2/t1.c:1832 [ERROR ] #2 0x564ed58e6755 in opj_thread_pool_submit_job /vcpkg/buildtrees/openjpeg/src/cda136d811-8d2d9326d7.clean/src/lib/openjp2/thread.c:835 [ERROR ] #3 0x564ed5910404 in opj_t1_decode_cblks /vcpkg/buildtrees/openjpeg/src/cda136d811-8d2d9326d7.clean/src/lib/openjp2/t1.c:1943 [ERROR ] #4 0x564ed58e3097 in opj_tcd_t1_decode /vcpkg/buildtrees/openjpeg/src/cda136d811-8d2d9326d7.clean/src/lib/openjp2/tcd.c:2000 [ERROR ] #5 0x564ed58e2596 in opj_tcd_decode_tile /vcpkg/buildtrees/openjpeg/src/cda136d811-8d2d9326d7.clean/src/lib/openjp2/tcd.c:1654 [ERROR ] #6 0x564ed58c8856 in opj_j2k_decode_tile /vcpkg/buildtrees/openjpeg/src/cda136d811-8d2d9326d7.clean/src/lib/openjp2/j2k.c:9862 [ERROR ] #7 0x564ed58cd051 in opj_j2k_decode_tiles /vcpkg/buildtrees/openjpeg/src/cda136d811-8d2d9326d7.clean/src/lib/openjp2/j2k.c:11664 [ERROR ] #8 0x564ed58c6978 in opj_j2k_exec /vcpkg/buildtrees/openjpeg/src/cda136d811-8d2d9326d7.clean/src/lib/openjp2/j2k.c:9006 [ERROR ] #9 0x564ed58cde00 in opj_j2k_decode /vcpkg/buildtrees/openjpeg/src/cda136d811-8d2d9326d7.clean/src/lib/openjp2/j2k.c:12010 [ERROR ] #10 0x564ed58d3396 in opj_jp2_decode /vcpkg/buildtrees/openjpeg/src/cda136d811-8d2d9326d7.clean/src/lib/openjp2/jp2.c:1607 [ERROR ] #11 0x564ed58b555c in opj_decode /vcpkg/buildtrees/openjpeg/src/cda136d811-8d2d9326d7.clean/src/lib/openjp2/openjpeg.c:521 [ERROR ] #12 0x564ed5766c51 in jpx_read_image (/root/sist2+0xcc2c51) [ERROR ] #13 0x564ed5767cf9 in fz_load_jpx (/root/sist2+0xcc3cf9) [ERROR ] #14 0x564ed586956d in pdf_load_jpx (/root/sist2+0xdc556d) [ERROR ] #15 0x564ed5868bd1 in pdf_load_image_imp (/root/sist2+0xdc4bd1) [ERROR ] #16 0x564ed586973d in pdf_load_image (/root/sist2+0xdc573d) [ERROR ] #17 0x564ed586bf65 in pdf_process_Do (/root/sist2+0xdc7f65) [ERROR ] #18 0x564ed586d4d2 in pdf_process_keyword (/root/sist2+0xdc94d2) [ERROR ] #19 0x564ed586ddb9 in pdf_process_stream (/root/sist2+0xdc9db9) [ERROR ] #20 0x564ed586e3c9 in pdf_process_contents (/root/sist2+0xdca3c9) [ERROR ] #21 0x564ed58945b2 in pdf_run_page_contents_with_usage_imp (/root/sist2+0xdf05b2) [ERROR ] #22 0x564ed5894d48 in pdf_run_page_contents_with_usage (/root/sist2+0xdf0d48) [ERROR ] #23 0x564ed57159df in fz_run_page_contents (/root/sist2+0xc719df) [ERROR ] #24 0x564ed5715be6 in fz_run_page (/root/sist2+0xc71be6) [ERROR ] #25 0x564ed4fe365c in load_pixmap /drone/src/third-party/libscan/libscan/ebook/ebook.c:82 [ERROR ] #26 0x564ed4fe382d in render_cover /drone/src/third-party/libscan/libscan/ebook/ebook.c:108 [ERROR ] #27 0x564ed4fe4490 in parse_ebook_mem /drone/src/third-party/libscan/libscan/ebook/ebook.c:295 [ERROR ] #28 0x564ed4fe52a1 in parse_ebook /drone/src/third-party/libscan/libscan/ebook/ebook.c:484 [ERROR ] #29 0x564ed4e28192 in parse /drone/src/src/parsing/parse.c:188 [ERROR ] #30 0x564ed4e1ae01 in worker_thread_loop /drone/src/src/tpool.c:88 [ERROR ] #31 0x564ed4e1bb84 in tpool_worker /drone/src/src/tpool.c:245 [ERROR ] #32 0x7f53a2f1bb42 (/lib/x86_64-linux-gnu/libc.so.6+0x94b42) [ERROR ] #33 0x7f53a2facbb3 in clone (/lib/x86_64-linux-gnu/libc.so.6+0x125bb3) [ERROR ] AddressSanitizer can not provide additional info. [ERROR ] SUMMARY: AddressSanitizer: SEGV (/root/sist2+0xcd22e2) in fz_free [ERROR ] Thread T2 created by T0 here: [ERROR ] #0 0x564ed4d1cd10 in interceptor_pthread_create (/root/sist2+0x278d10) [ERROR ] #1 0x564ed4e1cf07 in tpool_start /drone/src/src/tpool.c:363 [ERROR ] #2 0x564ed4e0035c in sist2_scan /drone/src/src/main.c:255 [ERROR ] #3 0x564ed4e068df in main /drone/src/src/main.c:598 [ERROR ] #4 0x7f53a2eb0d8f (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) [ERROR ] ==14==ABORTING [ADMIN ] Process returned non-zero exit code (1)

[simon987]

Could you try again with this dockerfile?

FROM simon987/sist2:3.1.2-x64-linux

RUN curl -o /root/sist2 https://files.simon987.net/.gate/sist2/simon987_sist2/arm_master_14_ca845d80e88d27fe9b74d5c93f9f1fe5751f26e3/sist2-arm64-linux-debug

[simon987]

Actually, before testing again you can wait for 3.1.3, which should at least fix the ebook error in https://github.com/simon987/sist2/issues/353#issuecomment-1522356990

[Badlee2020]

Tested it with 3.2.0 seems to work, now just having the problem that the index is somehow hanging

[simon987]

Can you try again with the latest docker tag x64-linux (v3.3.2) ?

[Badlee2020]

I was just able to test it. Tried it with 3.3.5 and the issue is still there. It ran with two indexes doing a scheduled scan every hour (offset 30 minutes between scans) for about 20 hours. Then the index froze. Tried to have a third scan run, but the index never finished. I will wipe all the containers and elastic search storage and try again when I have more time. The logs showed that everything got killed and stopped.

2023-10-12 04:00:08 [DEBUG elastic.c] Indexed 70 documents (51kB) <200> 2023-10-12 04:00:08 [DEBUG elastic.c] Indexed 70 documents (100kB) <200> 2023-10-12 04:00:08 [DEBUG elastic.c] Indexed 70 documents (136kB) <200> 2023-10-12 04:00:08 [DEBUG elastic.c] Indexed 70 documents (49kB) <200> 2023-10-12 04:00:08 [DEBUG elastic.c] Indexed 70 documents (441kB) <200> 2023-10-12 04:00:08 [DEBUG elastic.c] Indexed 70 documents (259kB) <200> 2023-10-12 04:00:08 [DEBUG elastic.c] Indexed 70 documents (319kB) <200> 2023-10-12 04:00:08 [INFO tpool.c] Worker threads finished 2023-10-12 04:00:08 [INFO tpool.c] Destroying thread pool 2023-10-12 04:00:08 [DEBUG database.c] Closing database /dev/shm/sist2-ipc-11682.sqlite (0x55629d8ac7b8) 2023-10-12 04:00:08 [WARNING elastic.c] calling _elastic_flush with 0 in queue 2023-10-12 04:00:08 [DEBUG elastic.c] Destroying indexer 2023-10-12 04:00:08 [DEBUG database.c] Closing database /dev/shm/sist2-ipc-11682.sqlite (0x7fe7100011f8) 2023-10-12 04:00:08 [WARNING elastic.c] calling _elastic_flush with 0 in queue 2023-10-12 04:00:08 [DEBUG elastic.c] Destroying indexer 2023-10-12 04:00:08 [DEBUG database.c] Closing database /dev/shm/sist2-ipc-11682.sqlite (0x7fe7100011f8) 2023-10-12 04:00:08 [WARNING elastic.c] calling _elastic_flush with 0 in queue 2023-10-12 04:00:08 [DEBUG elastic.c] Destroying indexer 2023-10-12 04:00:08 [DEBUG database.c] Closing database /dev/shm/sist2-ipc-11682.sqlite (0x7fe7100011f8) 2023-10-12 04:00:08 [DEBUG tpool.c] Child process terminated with status code 0 2023-10-12 04:00:08 [DEBUG tpool.c] Child process terminated with status code 0 2023-10-12 04:00:08 [DEBUG tpool.c] Child process terminated with status code 0 2023-10-12 04:00:08 [DEBUG elastic.c] Indexed 68 documents (47kB) <200> 2023-10-12 04:00:08 [DEBUG elastic.c] Destroying indexer 2023-10-12 04:00:08 [DEBUG database.c] Closing database /dev/shm/sist2-ipc-11682.sqlite (0x7fe7100011f8) 2023-10-12 04:00:08 [DEBUG tpool.c] Child process terminated with status code 0 2023-10-12 04:00:08 [INFO elastic.c] Refresh index <200> 2023-10-12 04:00:09 [INFO elastic.c] Merge index <200> 2023-10-12 04:00:09 [INFO elastic.c] Set refresh interval <200>

!! After some testing, the issue seems to be with restarting the frontend to the running scan. Sadly the logs show nothing. After it started to not finish indexing, I tuned off the front ends (I have 3) and restarted the admin container. Without doing anything, it all started working normally.