setup:upgrade fails - "Return value must be of type string, null returned"

Stoyvo commented 1 year ago

Preconditions

Magento Version : 2.4.4-p2

ElasticSuite Version : 2.10.10

Environment : Container (GCP Build)

Third party modules : No 3rd party modules

Steps to reproduce

New Magento 2.4.4-p2 instance
composer require elasticsuite 2.10.10
Run magento setup:upgrade

Expected result

setup:upgrade successful

Actual result

vahonc commented 1 year ago

Hello @Stoyvo,

Most likely your issue is specific only to your environment, especially to using some specific containers.

Are you sure that all is correctly configured, and Elasticsearch is installed and working?

Provide us the outputs/screenshots of these commands:

$ curl -XGET 'http://localhost:9200' //or change to your alias name

$ curl http://localhost:9200/_cat/indices?v

BR, Vadym

Stoyvo commented 1 year ago

In my pipeline I have added the CURL requests with authorization headers, it appears it's working.

Commands in order:

curl --user elastic:MYPASSWORD -XGET 'https://MYDOMAIN.es.northamerica-northeast1.gcp.elastic-cloud.com:9243'
curl --user elastic:MYPASSWORD 'https://MYDOMAIN.es.northamerica-northeast1.gcp.elastic-cloud.com:9243/_cat/indices?v'

php bin/magento cache:flush
php bin/magento setup:upgrade

Output:

Step #1:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Step #1:                                  Dload  Upload   Total   Spent    Left  Speed
Step #1: 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   571  100   571    0     0   5636      0 --:--:-- --:--:-- --:--:--  5653
Step #1: {
Step #1:   "name" : "instance-0000000001",
Step #1:   "cluster_name" : "428hhjagas2361512hf696",
Step #1:   "cluster_uuid" : "sk_DbshayG63AHnsahmLg",
Step #1:   "version" : {
Step #1:     "number" : "7.16.3",
Step #1:     "build_flavor" : "default",
Step #1:     "build_type" : "docker",
Step #1:     "build_hash" : "4e6e4eab229e7947edad834e688dad46290d018022",
Step #1:     "build_date" : "2022-01-06T23:43:02.825887787Z",
Step #1:     "build_snapshot" : false,
Step #1:     "lucene_version" : "8.10.1",
Step #1:     "minimum_wire_compatibility_version" : "6.8.0",
Step #1:     "minimum_index_compatibility_version" : "6.0.0-beta1"
Step #1:   },
Step #1:   "tagline" : "You Know, for Search"
Step #1: }
Step #1:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Step #1:                                  Dload  Upload   Total   Spent    Left  Speed
Step #1: 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  1524  100  1524    0     0  20245      0 --:--:-- --:--:-- --:--:-- 20320
Step #1: health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
Step #1: green  open   .geoip_databases                mLVGJ_CoSKSTLE_OHEbjCQ   1   0         41           41     39.2mb         39.2mb
Step #1: green  open   .kibana_7.16.3_001              IwbrJtU6SxGW2Tt9IkuxAw   1   0       2233          253      5.3mb          5.3mb
Step #1: green  open   .security-tokens-7              oos5kyeDTbWq8NKPuMO-hA   1   0          9            0     45.3kb         45.3kb
Step #1: green  open   .security-7                     OvobhTjOToSvL5mv4R2N5Q   1   0         80            0    297.6kb        297.6kb
Step #1: green  open   .apm-custom-link                EoaatdrGQ5uWDBaP05H3YA   1   0          0            0       226b           226b
Step #1: green  open   .fleet-enrollment-api-keys-7    cTPQar9eSJO1dQzZge0gnw   1   0          2            0      6.6kb          6.6kb
Step #1: green  open   .apm-agent-configuration        whAGVGYzTh-J9UQjDsbrSw   1   0          0            0       226b           226b
Step #1: green  open   .kibana_task_manager_7.16.3_001 IJsVaXmKS7qvYeg_X-Q9xQ   1   0         18          762      447kb          447kb
Step #1: green  open   .fleet-policies-7               hmXtILIkT3esvWHsAGHshw   1   0          3            0     15.7kb         15.7kb
Step #1: yellow open   magento2_product_1_v229         SNGWXfNGYzT2wDBgmgKy9Q   1   1        364            0      1.2mb          1.2mb
Step #1: green  open   .tasks                          B2vBSMuhRkiQ5Yp0ECW-JQ   1   0          6            0     35.1kb         35.1kb
Step #1: Flushed cache types:
Step #1: config
Step #1: layout
Step #1: block_html
Step #1: collections
Step #1: reflection
Step #1: db_ddl
Step #1: compiled_config
Step #1: eav
Step #1: customer_notification
Step #1: config_integration
Step #1: config_integration_api
Step #1: full_page
Step #1: config_webservice
Step #1: translate
Step #1: Updating modules:
Step #1: Cache cleared successfully
Step #1: Schema creation/updates:
Step #1: Module 'Magento_AdminAnalytics':
Step #1: Module 'Magento_Store':
...
Step #1: Installing data... 
Step #1: Enabling caches:
Step #1: Current status:
Step #1: layout: 1
Step #1: block_html: 1
Step #1: full_page: 1
Step #1: No alive nodes found in your cluster
Finished Step #1

The cluster is alive, and I can confirm Magento 2 connectivity is successful in admin Configuration -> Catalog -> Search

romainruaud commented 1 year ago

Where is your elasticsuite configuration stored ? In you app/etc/env.xml ?

Do you have configuration set for Elasticsuite ?

What is exactly the sequence of commands that are launched by your pipeline ?

Are you sure Elasticsuite is configured as the Magento default search engine ?

Btw this is a duplicated of #2716 but the author of this issue did not detail how it was fixed by his side :/

Regards

Stoyvo commented 1 year ago

@romainruaud

Where is your elasticsuite configuration stored ? In you app/etc/env.xml ? Database. I can attempt to move to app/etc/env.php but I believe db and env values are merged.

Do you have configuration set for Elasticsuite ? ElasticSearch configurations in admin are indeed set and confirmed working. Indexer and configuration test button confirms it is working successfully.

What is exactly the sequence of commands that are launched by your pipeline ? composer install --optimize-autoloader --no-dev --no-progress php bin/magento deploy:mode:set production -s php bin/magento cache:flush php bin/magento setup:upgrade (Where we fail) php bin/magento config:set dev/static/sign 1 --lock-env php bin/magento app:config:import .... Rest seems irrelevant

Are you sure Elasticsuite is configured as the Magento default search engine ? Where should I look to absolutely confirm this? I can tell index data is in ElasticSearch, in Admin Configuration I can confirm ElasticSearch7 is enabled and confirmed connection is working.

Note: I am happy to debug and chase this down, but I'm not exactly sure where to start with chasing this. Since it's blocking a pipeline I have to disable module and re-enable to test so I might be delayed in response. Could you tell me where a reindex may start with this module when it communicates with ElasticSearch? Looking for a starting point to debug.

Question: Has this module been tested with externally hosted ElasticSearch hosts?

romainruaud commented 1 year ago

Hi, you could have a look at our pipeline which runs without failing :

https://github.com/Smile-SA/elasticsuite/blob/2.10.x/.github/workflows/20-integration.yml

What is very strange for me with yours, is the fact that you never do a "setup:install" of Magento, which is supposed to be run to initialize things.

You can check an example of an install command here : https://github.com/Smile-SA/elasticsuite/wiki/ModuleInstall#magento-is-not-installed-yet-

Important things is "--search-engine=elasticsuite"

Are you injecting a DB in your pipeline ?

If yes, you might want to refer to the installation guide "when magento is already installed" :

https://github.com/Smile-SA/elasticsuite/wiki/ModuleInstall#magento-is-already-installed-

Other than that, if you see "Elasticsearch7" in your Back-Office, that's incorrect, you should see "Elasticsuite" as being selected for the search engine. It's done by default here : https://github.com/Smile-SA/elasticsuite/blob/2.10.x/src/module-elasticsuite-catalog/etc/config.xml#L20

But this will not be applied if a configuration exists "elsewhere" (in app/etc/env.php or in DB) to enforce that the engine is rather "elasticsearch7".

You can enforce Elasticsuite being the search engine with this command :

bin/magento config:set -le catalog/search/engine elasticsuite

And yes, Elasticsuite has been tested with externally hosted Elasticsearch host, thousands of websites are running Elasticsuite in production with such configuration.

Regards

Stoyvo commented 1 year ago

https://github.com/Smile-SA/elasticsuite/blob/2.10.x/src/module-elasticsuite-catalog/etc/config.xml#L20

Pipeline is deploying to production so we perform setup:upgrade to make changes to production DB. This is part of a "release" phase. I believe setup:install is invalid in the part.

Search Engine is configured only to Elastic Search 7 because "ElasticSuite" isn't an option yet (still releasing). Purhaps this is the issue? When running setup:upgrade in release, module is expecting "elasticsuite" to be configured?

Stoyvo commented 1 year ago

I just realized in documentation that ElasticSearch node configuration needs to be re-defined specifically for elasticsuite: https://github.com/Smile-SA/elasticsuite/wiki/ModuleInstall#magento-is-already-installed-

I already have ElasticSearch node configured with Default Magento engine, it's required for a Magento 2 install, why does elasticsuite require this to be reconfigured again? I think this is the cause, will confirm.

Stoyvo commented 1 year ago

I attempted the following:

In env.php set the default search engine to 'elasticsuite'
In env.php for smile_elasticsuite_core_base_settings/es_client/ items (servers, enable_https_mode, enable_http_auth, http_auth_user, http_auth_pwd)

setup:upgrade fails with: ElasticSuite : Unable to validate connection to Elasticsearch server : No alive nodes found in your cluster

This will require code debugging, the instance is live and these configurations are 100% correct. I can set these on a dev environment via Magento 2 admin, verified elasticsuite is functioning as expected.

Where, in code, is the entry for elasticsuite when running setup:upgrade?

romainruaud commented 1 year ago

Hi,

I'm sorry but I'll have to close this issue. That's completely project specific.

I understand that you are running a pipeline that is having the Elasticsuite modules disabled / not delivered.

We'll not be able to provide help here as part of our Open Source support policy.

If you want professional services (paid), please contact us at elasticsuite@smile.fr

Regards

Stoyvo commented 1 year ago

I'm curious how this was decided as a project issue. Magento is configured with elasticsearch credentials, and they work. This module does not install properly and is failing to connect. I hate to argue, but the error is clearly in module, there is no reason for OOB elasticsearch connection to work and this module fails.

Smile is falsely promoting open source software if this issue's resolve is paywalled, avoiding answers of open source questions.

Stoyvo commented 1 year ago

@romainruaud Regardless if you re-open this ticket or not, I will define the issue here for everyone who falls on this item.

This is elasticsuite ClientBuilder: https://github.com/Smile-SA/elasticsuite/blob/2.10.x/src/module-elasticsuite-core/Client/ClientBuilder.php

Core Magento has a Client defined as well: https://github.com/magento/magento2/blob/2.4-develop/app/code/Magento/Elasticsearch7/Model/Client/Elasticsearch.php

The key difference is how elasticsuite expects host values. Magento is forgiving, it will parse out "http://" and "https://" from host names. Managed providers will always provide an endpoint with http/https prefix, eg: elastic.co.

Due to some functionality in elasticsuite, it will automatically inherit a configuration value from core Magento configuration but never apply the same "cleaning" of the host url. Fallback functionality of importing Magento 2 configuration without cleaning hostname is here: https://github.com/Smile-SA/elasticsuite/blob/2.10.x/src/module-elasticsuite-core/Setup/ConfigOptionsList.php#L195

To resolve this, all elasticsuite configurations are expecting "server" values without any http/https prefix. Two options are:

Set this host for elasticsuite configuration via app/etc/env.php or bin/magento config:set smile_elasticsuite_core_base_settings/es_client/servers or
Remove any URL protocol from Magento 2 configuration for host. Eg: https://my.elastic.instance:1234 to become my.elastic.instance:1234

I hope this helps!

romainruaud commented 1 year ago

Smile is falsely promoting open source software if this issue's resolve is paywalled, avoiding answers of open source questions.

?????

This issue is not reproduced elsewhere except in your very specific pipeline. Check our CI actions, the pipeline is installing Magento + Elasticsuite, in a nominal way, and this just work.

We are providing this module for free, for several years, and fixing all the issues that people report, if we are able to reproduce them on standard (native) conditions. Once again, we do this for free.

How much did we charge you and/or your customer for all the features brought by Elasticsuite to him ? $0

So please, think to yourself before posting something like this. We cannot involve time in having a look to something we cannot reproduce. We are completely blind about what could possibly be happening in your particular situation, and we are not intended to jump into your project to work on it for free.

I do think you are misunderstanding what Open Source means....

That being said, and according to your last reply, it looks like we now have a proper way to reproduce this issue.

Let me rephrase it :

if you install Magento with option like --elasticsearch-server="https://elastic.instance" --elasticsearch-port="1234" this will then fail when it's passed to Elasticsuite.

Is this correct ?

Regards

romainruaud commented 1 year ago

@Stoyvo is my rephrasing of your problem correct ?

Stoyvo commented 1 year ago

@romainruaud My apologies for my comment. That was a personal low point, I do greatly value the effort of the team.

I did not use any options. These options were defined via config (app/etc/env.xml) and by CLI bin/magento config:set smile_elasticsuite_core_base_settings/es_client/servers

The issue is applying any server/host with "https://". Base Magento filters this out of URLs (https://github.com/magento/magento2/blob/2.4-develop/app/code/Magento/Elasticsearch7/Model/Client/Elasticsearch.php)

If a user copies Magento 2 URL with "https://", and then attempts to setup this module, it will fail.

Smile-SA / elasticsuite