Elasticsearch8 does not work when Server has Basic authentication

lory-to commented 1 year ago

Preconditions and environment

2.4.6 CE
Elasticsearch server 8.X

Steps to reproduce

Deploy elastic-cloud with Elasticsearch 8.X
Configure Magento to use elasticsearch8, see https://experienceleague.adobe.com/docs/commerce-operations/upgrade-guide/prepare/prerequisites.html?lang=en#upgrade-elasticsearch

Configure Catalog > Catalog > Catalog Search as follows

string(75) "https://*****.*****.azure.elastic-cloud.com"
["port"]=>
NULL
["index"]=>
string(8) "magento2"
["enableAuth"]=>
string(1) "1"
["username"]=>
string(7) "*****"
["password"]=>
string(24) "*****"
["timeout"]=>
string(2) "15"

Run bin/magento setup:upgrade
Run bin/magento indexer:reindex

Expected result

bin/magento setup:upgrade ends without errors bin/magento indexer:reindex ends without errors

Actual result

bin/magento setup:upgrade ends with Could not validate a connection to Elasticsearch. Verify that the Elasticsearch host and port are configured correctly.
bin/magento indexer:reindex ends with Indexer handler is not available: elasticsearch8

Additional information

The versions required for elastic/transport and elasticsearch/elasticsearch ignore the settings for username and password The Transport then does not have such credentials and according to the code they are not recorvered from the available nodes. (the nodes have the credentials as the URL is created prepending username:password@ but this URL is not used. \Elastic\Elasticsearch\Client::sendRequest is then sent without credentials.

Suggested change \Magento\Elasticsearch8\Model\Client\Elasticsearch::buildESConfig

add the following

$options['basicAuthentication'] = [$options['username'], $options['password']];

Release note

No response

Triage and priority

[ ] Severity: S0 - Affects critical data or functionality and leaves users without workaround.
[X] Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
[ ] Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
[ ] Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
[ ] Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

m2-assistant[bot] commented 1 year ago

Hi @lory-to. Thank you for your report. To speed up processing of this issue, make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, Add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release
For more details, review the Magento Contributor Assistant documentation.
Add a comment to assign the issue: @magento I am working on this
To learn more about issue processing workflow, refer to the Code Contributions.

Join Magento Community Engineering Slack and ask your questions in #github channel. :warning: According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting. :clock10: You can find the schedule on the Magento Community Calendar page. :telephone_receiver: The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, join the Community Contributions Triage session to discuss the appropriate ticket.

m2-assistant[bot] commented 1 year ago

Hi @engcom-Bravo. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

[ ] 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
[ ] 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue.
[ ] 3. Add Area: XXXXX label to the ticket, indicating the functional areas it may be related to.
[ ] 4. Verify that the issue is reproducible on 2.4-develop branch
Details
- Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
- If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
- If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!
[ ] 5. Add label Issue: Confirmed once verification is complete.
[ ] 6. Make sure that automatic system confirms that report has been added to the backlog.

engcom-Bravo commented 1 year ago

Hi @lory-to,

Thank you for reporting and collaboration.

Verified the issue on Magento 2.4.6 instance and the issue is not reproducible.Kindly refer the screenshots.

We have Configured Magento with elasticsearch8 using https://experienceleague.adobe.com/docs/commerce-operations/upgrade-guide/prepare/prerequisites.html?lang=en#upgrade-elasticsearch.

Both the bin/magento setup:upgrade and bin/magento indexer:reindex ends without errors.

Kindly recheck the behavior on Magento 2.4.6 instance and elaborate steps to reproduce if the issue is still reproducible.

Thanks.

lory-to commented 1 year ago

Hi @lory-to, We have Configured Magento with elasticsearch8 using https://experienceleague.adobe.com/docs/commerce-operations/upgrade-guide/prepare/prerequisites.html?lang=en#upgrade-elasticsearch.

Both the bin/magento setup:upgrade and bin/magento indexer:reindex ends without errors.

Kindly recheck the behavior on Magento 2.4.6 instance and elaborate steps to reproduce if the issue is still reproducible.

Thanks.

Good morning @engcom-Bravo Does your Elasticsearch instance have Authentication enabled?

This is happening only when I'm required to imput username and password.

engcom-Bravo commented 1 year ago

Hi @lory-to,

Thanks for your update.

Verified the issue on Magento 2.4.6 instance and the issue is not reproducible.Kindly refer the screenshots.

Please try to install elasticsearch with setup : install command.Kindly check this document https://experienceleague.adobe.com/docs/commerce-operations/installation-guide/advanced.html

bin/magento setup:install --base-url=[http://magento246.local](http://magento2.local/) --db-host=localhost --db-name=magento246 --db-user=global --db-password=Global@123 --admin-firstname="admin" --admin-lastname=admin --admin-email=admin@admin.com --admin-user=admin --admin-password=admin123 --language=en_US --currency="USD" --timezone=America/Chicago --use-rewrites=1 --backend-frontname=admin --search-engine=elasticsearch8 --elasticsearch-host=localhost --elasticsearch-port=9200 --elasticsearch-username=magento --elasticsearch-password=guest

We have configured Magento instance with Elasticsearch8 using this document https://experienceleague.adobe.com/docs/commerce-operations/upgrade-guide/prepare/prerequisites.html?lang=en#upgrade-elasticsearch

https://user-images.githubusercontent.com/51680745/235614003-9b47f854-09c4-4553-914e-3dbf5bcce37e.mov

Both the bin/magento setup:upgrade and bin/magento indexer:reindex ends without errors.

Kindly recheck the behavior on Magento 2.4.6 instance and elaborate steps to reproduce if the issue is still reproducible.

Please let us know if you are still facing any issue.

Thanks.

engcom-Bravo commented 1 year ago

Hi @lory-tom,

We have noticed that this issue has not been updated for a period of 14 Days. Hence we assume that this issue is fixed now, so we are closing it. Please raise a fresh ticket or reopen this ticket if you need more assistance on this.

Thanks.

jtolhurst-trellis commented 1 year ago

I have also ran into this issue on 2.4.6 and found @lory-to 's proposed solution to work. This seem to happen with external hosts. I also am using an externally hosted instance and not one hosted on localhost. It is possible this could play a role in how you are testing.

In either case, it would be beneficial for Magento to use the Elasticsearch library correctly. Here is a demonstration without Magento that shows how Magento's config array that it builds does not work, but adding the correct configurations allows a connection.

composer require elasticsearch/elasticsearch

Create a file test.php

<?php

include 'vendor/autoload.php';

use Elastic\Elasticsearch\ClientBuilder;

// How Magento would build an options array
$options1 = [
    'hostname' => 'https://my-instance.es.us-east-1.aws.found.io',
    'port' => '443',
    'index' => 'myindex',
    'enableAuth' => '1',
    'username' => 'username',
    'password' => 'password',
    'timeout' => '15',
    'hosts' => [
        'https://username:password@my-instance.es.us-east-1.aws.found.io:443'
    ]
];

$client1 = ClientBuilder::fromConfig($options1, true);

// Adding in `basicAuthentication` to the config
$options2 = [
    'hostname' => 'https://my-instance.es.us-east-1.aws.found.io',
    'port' => '443',
    'index' => 'myindex',
    'enableAuth' => '1',
    'username' => 'username',
    'password' => 'password',
    'timeout' => '15',
    'basicAuthentication' => [
        'username',
        'password',
    ],
    'hosts' => [
        'https://username:password@my-instance.es.us-east-1.aws.found.io:443'
    ]
];

$client2 = ClientBuilder::fromConfig($options2, true);

// Info API
$response1 = $client1->ping()->asBool();
$response2 = $client2->ping()->asBool();

Execute the file

php test.php

And you will get the following output

bool(false)
bool(true)

Where the first ping() fails because of a 401 Unauthorized response from the server, but the second will succeed because it properly authenticates. So it seems that Magento could do better to correctly configure the Elasticsearch client to ensure a connection to all clients.

As an additional note, the current array that Magento passes is largely discarded. If you do not silence the errors, fromConfig($options, false), the Elasticsearch library will emit an error showing that it discards all but the hosts key from the options because they are not valid configurations.

magento / magento2