search

nextcloud / fulltextsearch_elasticsearch

🔍 Use Elasticsearch to index the content of your Nextcloud
https://apps.nextcloud.com/apps/fulltextsearch_elasticsearch
GNU Affero General Public License v3.0
82 stars 32 forks source link

NC29:Fulltextsearch live crash #375

Open PesekLasy opened 5 months ago

PesekLasy commented 5 months ago

Hello, 2 days ago my service for fultextsearch live crash and since this day does not start. When i run occ fulltextsearch:live by hand ./occ fulltextsearch:live is returning error to stdout:

Error: Typed property OC\FullTextSearch\Model\IndexDocument::$access must not be accessed before initialization in /home/ncloud/lib/private/FullTextSearch/Model/IndexDocument.php:775
Stack trace:
#0 [internal function]: OC\FullTextSearch\Model\IndexDocument->jsonSerialize()
#1 /home/ncloud/apps/files_fulltextsearch/vendor/artificial-owl/my-small-php-tools/lib/Traits/Nextcloud/nc22/TNC22Logger.php(192): json_encode(Array)
#2 /home/ncloud/apps/files_fulltextsearch/vendor/artificial-owl/my-small-php-tools/lib/Traits/Nextcloud/nc22/TNC22Logger.php(177): OCA\Files_FullTextSearch\Provider\FilesProvider->log(0, '[debug] updateD...', false, Array)
#3 /home/ncloud/apps/files_fulltextsearch/lib/Provider/FilesProvider.php(315): OCA\Files_FullTextSearch\Provider\FilesProvider->debug('[debug] updateD...', Array)
#4 /home/ncloud/apps/fulltextsearch/lib/Service/IndexService.php(409): OCA\Files_FullTextSearch\Provider\FilesProvider->updateDocument(Object(OCA\FullTextSearch\Model\Index))
#5 /home/ncloud/apps/fulltextsearch/lib/Command/Live.php(313): OCA\FullTextSearch\Service\IndexService->updateDocument(Object(OCA\FullTextSearch_Elasticsearch\Platform\ElasticSearchPlatform), Object(OCA\Files_FullTextSearch\Provider\FilesProvider), Object(OCA\FullTextSearch\Model\Index))
#6 /home/ncloud/apps/fulltextsearch/lib/Command/Live.php(262): OCA\FullTextSearch\Command\Live->liveCycle()
#7 /home/ncloud/3rdparty/symfony/console/Command/Command.php(298): OCA\FullTextSearch\Command\Live->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#8 /home/ncloud/core/Command/Base.php(177): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#9 /home/ncloud/3rdparty/symfony/console/Application.php(1040): OC\Core\Command\Base->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#10 /home/ncloud/3rdparty/symfony/console/Application.php(301): Symfony\Component\Console\Application->doRunCommand(Object(OCA\FullTextSearch\Command\Live), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#11 /home/ncloud/3rdparty/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#12 /home/ncloud/lib/private/Console/Application.php(213): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 /home/ncloud/console.php(102): OC\Console\Application->run()
#14 /home/ncloud/occ(11): require_once('/home/ncloud...')
#15 {main}

NC version 29.0.2

PesekLasy commented 5 months ago

Hello, I was fix this issue with reset index and then is possible run again liveindex.

vasyugan commented 3 months ago

I'm seeing this error on two instances immediately after running occ fulltextsearch:reset and freshly reindexing everything.

Here is the console output:

Memory: 50 MB                                                                                                                                        
┌─ Indexing  ────                                                          
│ Action: indexing
│ Provider: Files                Account: **                                                                                                          
│ Document:                                                                                                                                           
│ Info:                                                                                                                                               
│ Title:                                                                   
│ Content size:                                                                                                                                       
└──                                                                                                                                                   
┌─ Results ────                                                                                                                                       
│ Result:      0/0                                                                                                                                    
│ Index:                                                                                                                                              
│ Status:                                                                                                                                             
│ Message:                                                                                                                                            
│                                                                                                                                                     
│                                                                                                                                                     
└──                                                                                                                                                   
┌─ Errors ────                                                             
│ Error:    144/144                                                        
│ Index: files:9176428              
│ Exception: OCA\FullTextSearch_Elasticsearch\Vendor\Elastic\Elasticsearch\Exception\ClientResponseException                                          
│ Message: unknown error             
│                                    
│                                    
└──                                  
## x:first result ## c/v:prec/next result ## b:last result                 
## f:first error ## h/j:prec/next error ## d:delete error ## l:last error  
## q:quit ## p:pause                 
An unhandled exception has been thrown:                                    
Error: Typed property OC\FullTextSearch\Model\IndexDocument::$access must not be accessed before initialization in /var/www/nextcloud/lib/private/FullTextSearch/Model/IndexDocument.php:775                                     
Stack trace:      
#0 [internal function]: OC\FullTextSearch\Model\IndexDocument->jsonSerialize()                                                                        
#1 /var/www/nextcloud/apps/files_fulltextsearch/vendor/artificial-owl/my-small-php-tools/lib/Traits/Nextcloud/nc22/TNC22Logger.php(192): json_encode()
#2 /var/www/nextcloud/apps/files_fulltextsearch/vendor/artificial-owl/my-small-php-tools/lib/Traits/Nextcloud/nc22/TNC22Logger.php(177): OCA\Files_FullTextSearch\Provider\FilesProvider->log()                                  
#3 /var/www/nextcloud/apps/files_fulltextsearch/lib/Provider/FilesProvider.php(315): OCA\Files_FullTextSearch\Provider\FilesProvider->debug()         
#4 /var/www/nextcloud/apps/fulltextsearch/lib/Service/IndexService.php(409): OCA\Files_FullTextSearch\Provider\FilesProvider->updateDocument()        
#5 /var/www/nextcloud/apps/fulltextsearch/lib/Command/Live.php(313): OCA\FullTextSearch\Service\IndexService->updateDocument()                        
#6 /var/www/nextcloud/apps/fulltextsearch/lib/Command/Live.php(262): OCA\FullTextSearch\Command\Live->liveCycle()                                     
#7 /var/www/nextcloud/3rdparty/symfony/console/Command/Command.php(298): OCA\FullTextSearch\Command\Live->execute()                                   
#8 /var/www/nextcloud/core/Command/Base.php(177): Symfony\Component\Console\Command\Command->run()                                                    
#9 /var/www/nextcloud/3rdparty/symfony/console/Application.php(1040): OC\Core\Command\Base->run()                                                     
#10 /var/www/nextcloud/3rdparty/symfony/console/Application.php(301): Symfony\Component\Console\Application->doRunCommand()                           
#11 /var/www/nextcloud/3rdparty/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun()                                  
#12 /var/www/nextcloud/lib/private/Console/Application.php(213): Symfony\Component\Console\Application->run()                                         
#13 /var/www/nextcloud/console.php(102): OC\Console\Application->run()     
#14 /var/www/nextcloud/occ(11): require_once('...')                        
#15 {main}
ralfi commented 2 months ago

yes, same here, but fulltextsearch:index seams to work

vasyugan commented 1 month ago

Still happening. After a restart of the server I had to reset and recreate the indexes for two of my instances. This seems to affect every NC instance I manage, across multiple servers.

ralfi commented 1 month ago

Complete reset and rebuild the index solves the issue at me, including PDF indexing

vasyugan commented 1 month ago

Complete reset and rebuild the index solves the issue at me, including PDF indexing

Until after a while it fails again. That's what I am experiencing.

XueSheng-GIT commented 1 month ago

I'm also seeing that issue occasionally on one of my instances (NC29.0.7). On this specific instance, there's a lot of work done in groupfolders. Some of them using ACL. My assumption is that this is part of the issue (maybe the the live process fails on files which are restricted for some users/groups!?). If the issue occurs, a reset and index does solve it (at least for a couple of hours or days... depending on the case).

pmarini-nc commented 3 weeks ago

Hello maintainers, what is the root cause of this issue? Is there a stable fix? I applied the code change proposed here https://github.com/nextcloud/fulltextsearch/issues/849#issue-2401253266, but it would be great to have a proper solution. Thanks!

PesekLasy commented 3 weeks ago

Many months since report and still doesn't fix.

XueSheng-GIT commented 3 weeks ago

I'm now able to reproduce the issue on NC30.0.2rc2. It would be great if some of you could double check whether you can reproduce it.

Steps:

  1. Create user "Max"
  2. Create Group "Group1"
  3. Assign Max to Group1
  4. Create Groupfolder "TestFolder", Group: Group1 (write, share, delete), ACL: Group1, Screenshot below
  5. Login as Max
  6. Change ACL of TestFolder (restrict share for Group1), Screenshot below
  7. Stop live index: occ fulltextsearch:stop
  8. Start live index: occ fulltextsearch:live
  9. Open TestFolder as Max in webif
  10. Create a new file (e.g. markdown).

Expected behavior: Created file is added to fulltextsearch index.

Actual behavior: Live index fails with error mentioned in original issue report. https://github.com/nextcloud/fulltextsearch_elasticsearch/issues/375#issue-2361529977

Screenshots of config: Image Image

Futher details: Generally, it seems that this is a permission issue. It seems that updateDocument (see following link) does not return a valid FilesDocument so that finally json_encode of $document fails at some point. Whereas the property $access itself does not seem to be the root cause. https://github.com/nextcloud/files_fulltextsearch/blob/d3472a4584fef46152a815c17856fad840e316bc/lib/Service/FilesService.php#L582-L590

Example how $document looks like in case of such an error (without $access to be able to get that output):

{\"id\":\"2939448\",\"providerId\":\"files\",\"modifiedTime\":0,\"title\":\"\",\"link\":\"\",\"index\":{\"ownerId\":\"Max.Mustermann\",\"providerId\":\"files\",\"collection\":\"local\",\"source\":\"\",\"documentId\":\"2939448\",\"lastIndex\":0,\"errors\":[],\"errorCount\":0,\"status\":60,\"options\":[]},\"source\":\"\",\"info\":[],\"hash\":\"\",\"contentSize\":0,\"tags\":[],\"metatags\":[],\"subtags\":[],\"more\":[],\"excerpts\":[],\"score\":\"\"}"}