Can't performe a full index with occ fulltextsearch:index

GekoCH commented 9 months ago

I'm running Nextcloud: 28.0.2.5 elasticsearch: 8.8.2

I tried to do a full index with occ fulltextsearch:index but I get this error at file 123911 of 184926.

I can open the particular file without any problem so it0s not corrupt. Can someone help me to get this working? Thx

Memory: 607 MB                                                                                                                                                ┌─ Indexing  ────                                                                                                                                             │ Action: indexDocument                                                                                                                                       │ Provider: Files                Account: admin                                                                                                               │ Document: 335204                                                                                                                                            │ Info: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet                                                                                     │ Title: xxx/yyy/test.xlsx
│ Content size: 11952924
│ Chunk:     48/1685
│ Progress: 123911/184926
└──
┌─ Results ────
│ Result: 124110/124110
│ Index: files:335196
│ Status: ok
│ Message: {"_index":"my_index","_id":"files:335196","_version":1,"result":"cr
│ eated","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":123795,"_pri
│ mary_term":1}
└──
┌─ Errors ────
│ Error:    630/630
│ Index: files:335228
│ Exception: Elastic\Elasticsearch\Exception\ClientResponseException
│ Message: unknown error
│
│
└──
## x:first result ## c/v:prec/next result ## b:last result
## f:first error ## h/j:prec/next error ## d:delete error ## l:last error
## q:quit ## p:pause

In ElasticSearchPlatform.php line 202:

  [OCA\FullTextSearch\Exceptions\PlatformTemporaryException]

gianlucagilardi commented 8 months ago

same issue here. Following.

chriscroome commented 3 months ago

Is this the same issue?


  occ fulltextsearch:index

  Memory: 82 MB
  ┌─ Indexing  ────
  │ Action: fillDocument
  │ Provider: Files                Account: animorph
  │ Document: 286977
  │ Info: httpd/unix-directory
  │ Title: Directors/BDC/Sales/Prospecting
  │ Content size: 0
  │ Chunk:      3/24
  │ Progress:    583/587
  └──
  ┌─ Results ────
  │ Result:      5/5
  │ Index: files:286977
  │ Status: ok
  │ Message: {"_index":"nextcloud_index","_id":"files:286977","_version":1,"result":"created","_shards":{"total":2,"successful":1,"fai 
  │ led":0},"_seq_no":4,"_primary_term":1}
  │
  └──
  ┌─ Errors ────
  │ Error:     42/42
  │ Index: files:6117
  │ Exception: OCA\FullTextSearch_Elasticsearch\Vendor\Elastic\Elasticsearch\Exception\ClientResponseException
  │ Message: unknown error
  │
  │
  └──
  ## x:first result ## c/v:prec/next result ## b:last result                                                                           
  ## f:first error ## h/j:prec/next error ## d:delete error ## l:last error                                                            
  ## q:quit ## p:pause                                                                                                                 
  Force Quit

trying again returns:

  occ fulltextsearch:index

  In RunningService.php line 86:

    Index is already running

davidcrds commented 3 months ago

Having the same “In ElasticSearchPlatform.php line 202:” issue on Nextcloud 29.0.4, since adding a new user with a bunch of files. No idea how to identify the culprit, probably a rogue file that grinds the indexing to a halt.

ferdiga commented 2 months ago

you may want to check [(https://github.com/nextcloud/fulltextsearch/pull/870)]

ferdiga commented 2 months ago

attention -

Be aware that if you share folders, the full-text search may index all files for every user with whom the folder is shared. This can lead to a large number of indexed files, so managing shared folders carefully is essential.

"Share Folders" was the solution before group folders were introduced.

ferdiga commented 2 months ago

Be aware that images and external files are submitted to the fulltextsearch process although config says "no image processing" [(https://github.com/nextcloud/fulltextsearch/issues/871)]

processing of images is extremely time and resource consuming

chriscroome commented 2 months ago

Thanks @ferdiga! How would you suggest deploying this pull request to test it?

ferdiga commented 2 months ago

I moved shared folders to group folders
I have no solution for not processing images and external files yet

if you have access to the source code on your machine, simple copy/install the updated Runner.php

nextcloud / fulltextsearch_elasticsearch

Can't performe a full index with occ fulltextsearch:index #346