search

nextcloud / fulltextsearch_elasticsearch

🔍 Use Elasticsearch to index the content of your Nextcloud
GNU Affero General Public License v3.0
81 stars 29 forks source link

TypeError: Argument 2 passed ... must be of type array, null given #170

Closed CharlesCage closed 2 years ago

CharlesCage commented 3 years ago

Happening almost instantly on indexing.

Nextcloud: 21.0.4 Full text search: 21.0.3 Full text search - Elasticsearch Platform: 21.0.2 Full text search - Files: 21.0.4

Elasticsearch: 7.14.0

Trace:


An unhandled exception has been thrown:
TypeError: Argument 2 passed to OCA\FullTextSearch_Elasticsearch\Platform\ElasticSearchPlatform::getArray() must be of the type array, null given, called in /var/www/nextcloud/apps/fulltextsearch_elasticsearch/lib/Platform/ElasticSearchPlatform.php on line 317 and defined in /var/www/nextcloud/apps/fulltextsearch_elasticsearch/vendor/daita/my-small-php-tools/lib/Traits/TArrayTools.php:252
Stack trace:
#0 /var/www/nextcloud/apps/fulltextsearch_elasticsearch/lib/Platform/ElasticSearchPlatform.php(317): OCA\FullTextSearch_Elasticsearch\Platform\ElasticSearchPlatform->getArray('error', NULL)
#1 /var/www/nextcloud/apps/fulltextsearch_elasticsearch/lib/Platform/ElasticSearchPlatform.php(287): OCA\FullTextSearch_Elasticsearch\Platform\ElasticSearchPlatform->parseIndexErrorException(Object(Elasticsearch\Common\Exceptions\BadRequest400Exception))
#2 /var/www/nextcloud/apps/fulltextsearch_elasticsearch/lib/Platform/ElasticSearchPlatform.php(236): OCA\FullTextSearch_Elasticsearch\Platform\ElasticSearchPlatform->manageIndexErrorException(Object(OCA\Files_FullTextSearch\Model\FilesDocument), Object(Elasticsearch\Common\Exceptions\BadRequest400Exception))
#3 /var/www/nextcloud/apps/fulltextsearch/lib/Service/IndexService.php(434): OCA\FullTextSearch_Elasticsearch\Platform\ElasticSearchPlatform->indexDocument(Object(OCA\Files_FullTextSearch\Model\FilesDocument))
#4 /var/www/nextcloud/apps/fulltextsearch/lib/Command/Live.php(308): OCA\FullTextSearch\Service\IndexService->updateDocument(Object(OCA\FullTextSearch_Elasticsearch\Platform\ElasticSearchPlatform), Object(OCA\Files_FullTextSearch\Provider\FilesProvider), Object(OCA\FullTextSearch\Model\Index))
#5 /var/www/nextcloud/apps/fulltextsearch/lib/Command/Live.php(261): OCA\FullTextSearch\Command\Live->liveCycle()
#6 /var/www/nextcloud/3rdparty/symfony/console/Command/Command.php(255): OCA\FullTextSearch\Command\Live->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#7 /var/www/nextcloud/core/Command/Base.php(169): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#8 /var/www/nextcloud/3rdparty/symfony/console/Application.php(1009): OC\Core\Command\Base->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#9 /var/www/nextcloud/3rdparty/symfony/console/Application.php(273): Symfony\Component\Console\Application->doRunCommand(Object(OCA\FullTextSearch\Command\Live), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#10 /var/www/nextcloud/3rdparty/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#11 /var/www/nextcloud/lib/private/Console/Application.php(215): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#12 /var/www/nextcloud/console.php(100): OC\Console\Application->run()
#13 /var/www/nextcloud/occ(11): require_once('/var/www/nextcl...')
#14 {main}
CharlesCage commented 3 years ago

Looking a little deeper $e->getmessage() is returning "Unknown 413 error from Elasticsearch"

CharlesCage commented 3 years ago

This is an issue with the index insert exceeding Elasticsearch http.max_content_length

CharlesCage commented 3 years ago

Ok, so I'm still testing, but it looks like setting http.max_content_length to 200mb in the elasticsearch config field resolves this for me.

I think part of the trick here is that the default for http.max_content_length is 100mb, but with the full text search maximum files size in Nextcloud settings set to 100mb (mine was/is), we're still triggering this.

Maybe it's just that the overhead of the request itself causes problems close to the limit? Maybe we should either recommend setting to 110mb (or 200mb or something) or default to max size in the installed product to 90mb or something to account for the overhead.

Also it looks like there's a possibility for $e->getmessage() to return something other than json, so we probably need a quick check there to see if there's text in it (at least this message, maybe?) before applyijng json_decode?

Thoughts?

CharlesCage commented 3 years ago

One more update: it looks like Elasticsearch used to fail ugly on an oversized http call, but a while back they put in catching to provide the 413 error. That may account for why we're not handling it.

james-cook commented 3 years ago

Thanks for documenting this - solved the problem I was having :) You need to set the value in e.g. /etc/elasticsearch/elasticsearch.yml: http.max_content_length: 1000mb This (above) is absolutely necessary.

I also set this "same" value in the nextcloud ES settings (may not be necessary - didn't test without): /index.php/settings/admin/fulltextsearch: Maximum file size: Maximum file size to index (in Mb) [1000] <-- value set in form...

R0Wi commented 3 years ago

I think this should be fixed by https://github.com/nextcloud/fulltextsearch_elasticsearch/commit/9c5649f6834b9cd047a1d4b71f6660853b475f6b which is currently only available in the master branch. Although if you want to index larger files you should increment the ElasticSearch http.max_content_length settings like proposed in https://github.com/nextcloud/fulltextsearch_elasticsearch/issues/170#issuecomment-915180198. According to the docs the default value for this is 100MB like stated above.

R0Wi commented 3 years ago

Edit: seems it was backported to version 22 in https://github.com/nextcloud/fulltextsearch_elasticsearch/pull/165 so fulltextsearch_elasticsearch v22.0.1 should already contain the fix. Not sure if it will be backported to 21 so you might have to update or request @ArtificialOwl .

R0Wi commented 2 years ago

@CharlesCage could you please check if the issue still exists in fulltextsearch_elasticsearch app version v22.0.1? Otherwise i think this can be closed.

ArtificialOwl commented 2 years ago

You can also set a maximum size for your files to be indexed ? It should be fixed in 22, do you still need a backport to 21 ?