search

elastic / azure-marketplace

Elasticsearch Azure Marketplace offering + ARM template
https://www.elastic.co/guide/en/elastic-stack-deploy/current/index.html
MIT License
120 stars 161 forks source link

Unable to authenticate user [elastic] for rest request #358

Closed AnilSunkesula closed 4 years ago

AnilSunkesula commented 4 years ago

I have launched Elasticsearch(Self-Managed) cluster using Azure marketplace. But not able to access elastic search. I tried with default user elastic and password which I gave while deploying the cluster. Getting error "unable to authenticate user [elastic] for REST request"

Here are the steps that I followed,

Selected Existing Virtual Network and Subnet Selected No for Kibana, logstash and jump box selected load balance as internal Selected yes for all data nodes are master eligible Choose no to configure TLS security

Once deployment is done, I am trying to access internal load balance on 9200 port http://:9200 But getting error "unable to authenticate user [elastic] for REST request" after I entered the username and password

I tried login through browser, using curl command and also through postman. Getting same error in all cases. curl -X GET 'http://:9200/_cat/indices?v' -u elastic:

I also tried redeploying the cluster with a different password, But still getting same error.

Could you please help me?

russcam commented 4 years ago

@AnilSunkesula, please use https://discuss.elastic.co/c/elastic-stack/elasticsearch/6 for questions in the future; the forum is designed for questions and answers. GitHub issues should be used for confirmed bugs and feature requests.

Once deployment is done, I am trying to access internal load balance on 9200 port http://:9200

From where are you accessing the internal load balancer? The internal load balancer is addressable only from a resource connected to the same vnet (address on a subnet, vnet-to-vnet gateway, etc.). It sounds like access is being attempted from the same vnet, based on the error, but be good to confirm. Can you connect with any of the other configured built-in user accounts, like kibana,?

russcam commented 4 years ago

I've spent some time investigating user supplied passwords with special characters, and have opened https://github.com/elastic/azure-marketplace/pull/359 to fix a bug if a password contains one or more double quotes. Does the password you're using for elastic superuser contain a double quote, or any symbols (if so, which ones)?

AnilSunkesula commented 4 years ago

Thanks for spending your time. My password consists (capital letter, small letters and special symbols @ !).

I am accessing the internal load balancer from a resource which is connected to the Elasticsearch Vnet through peering. I am able to login to the elastic data nodes from this resource. But getting error while trying to connect load balancer IP on 9200.

From each data node, I am able to do telnet on 9200 and 9300 for other two data nodes.

russcam commented 4 years ago

Can you access the cluster through the internal loadbalancer, from one of the elastic nodes?

AnilSunkesula commented 4 years ago

I am getting same error when I access the internal load balancer from one of the elastic nodes. Please find the attached images.

elastic_curl.PNG: contains error message while accessing load balancer from one of the elastic nodes and also /etc/elasticsearch/elasticsearch.yml file content

elastic_curl

arm-install-log.PNG: contains /var/log/arm-install.log. Here I can see built in elastic user password is updated.

arm-install-log

Sorry I couldn't copy paste the content. So sending the screenshots

russcam commented 4 years ago

Please can you copy/paste the contents in future. Please can you copy/paste the contents of /var/lib/waagent/custom-script/download/0/stdout and /var/lib/waagent/custom-script/download/0/stderr. My suspicion is that the elastic user password has not been updated successfully, despite what /var/log/arm-install.log states.

AnilSunkesula commented 4 years ago

I can see password is successfully updated message in /var/log/arm-install.log file.

Could you please let us know what are the inbound and outbound security rules that need to be created for network security group? I am suspecting there might be NSG issue.

We selected existing Vnet and existing subnet. Existing subnet has NSG.

russcam commented 4 years ago

Please can you copy/paste the contents of /var/lib/waagent/custom-script/download/0/stdout and /var/lib/waagent/custom-script/download/0/stderr. My suspicion is that the elastic user password has not been updated successfully, despite what /var/log/arm-install.log states.

Please can you post the contents of these files (from all nodes). I don't believe the elastic user password has been successfully updated, and the contents of those files should help to determine this.

Could you please let us know what are the inbound and outbound security rules that need to be created for network security group? I am suspecting there might be NSG issue.

I don't think this is the case because you can successfully reach Elasticsearch through the internal load balancer, but receive a 401 response because the credentials supplied cannot authenticate.

AnilSunkesula commented 4 years ago

First of all sorry the for the delay in reply. We have some network issues to take the log files into our local systems

Please find the content for stdout and stderr files

/var/lib/waagent/custom-script/download/0/stdout

[18082020-18:08:48] Begin execution of Elasticsearch script extension on epp02data-0 [18082020-18:08:48] epp02data-0 not found in /etc/hosts [18082020-18:08:48] hostname epp02data-0 added to /etc/hosts [18082020-18:08:48] Option l set [18082020-18:08:48] Option n set [18082020-18:08:48] Option v set [18082020-18:08:48] Option m set [18082020-18:08:48] Option A set [18082020-18:08:48] Option R set [18082020-18:08:48] Option K set [18082020-18:08:48] Option S set [18082020-18:08:48] Option F set [18082020-18:08:48] Option M set [18082020-18:08:48] Option B set [18082020-18:08:48] Option Z set [18082020-18:08:48] Option p set [18082020-18:08:48] Option a set [18082020-18:08:48] Option k set [18082020-18:08:48] Option E set [18082020-18:08:48] Option L set [18082020-18:08:48] Option C set [18082020-18:08:48] Option D set [18082020-18:08:48] Option H set [18082020-18:08:48] Option G set [18082020-18:08:48] Option V set [18082020-18:08:48] Option J set [18082020-18:08:48] Option T set [18082020-18:08:48] Option W set [18082020-18:08:48] Option N set [18082020-18:08:48] Option O set [18082020-18:08:48] Option P set [18082020-18:08:48] using bootstrap password as the seed password [18082020-18:08:48] bootstrapping an Elasticsearch 7.8.0 cluster named 'LMAPAZ1ESPPDSHL02' with minimum_master_nodes set to 3 [18082020-18:08:48] cluster uses dedicated master nodes is set to 0 and unicast goes to ["epp02data-0","epp02data-1","epp02data-2","epp02data-3","epp02data-4"] [18082020-18:08:48] cluster install X-Pack plugin is set to 1 [18082020-18:08:48] cluster basic security is set to 1 [18082020-18:08:48] [format_data_disks] checking node role [18082020-18:08:48] [format_data_disks] data node, data disks may be attached [18082020-18:08:48] [format_data_disks] starting partition and format attached disks [18082020-18:08:48] [format_and_partition_disks] Option s set with value [18082020-18:08:49] [format_and_partition_disks] installing or updating mdadm [18082020-18:09:00] [format_and_partition_disks] apt-get updated installing mdadm now Reading package lists... Building dependency tree... Reading state information... mdadm is already the newest version (3.3-2ubuntu7.6). The following package was automatically installed and is no longer required: grub-pc-bin Use 'apt autoremove' to remove it. 0 upgraded, 0 newly installed, 0 to remove and 9 not upgraded. [18082020-18:09:01] [format_and_partition_disks] apt-get installed mdadm and can be found returns: 0 [18082020-18:09:01] [format_and_partition_disks] Begin creating striped volume [18082020-18:09:01] [format_and_partition_disks] Disks are /dev/sdc [18082020-18:09:01] [format_and_partition_disks] Working on /dev/sdc [18082020-18:09:01] [format_and_partition_disks] /dev/sdc is not partitioned, partitioning [18082020-18:09:01] [format_and_partition_disks] create partition for /dev/sdc with fdisk

Welcome to fdisk (util-linux 2.27.1). Changes will remain in memory only, until you decide to write them. Be careful before using the write command.

Device does not contain a recognized partition table. Created a new DOS disklabel with disk identifier 0x86a0d721.

Command (m for help): Partition type p primary (0 primary, 0 extended, 4 free) e extended (container for logical partitions) Select (default p): Partition number (1-4, default 1): First sector (2048-268435455, default 2048): Last sector, +sectors or +size{K,M,G,T,P} (2048-268435455, default 268435455): Created a new partition 1 of type 'Linux' and of size 128 GiB.

Command (m for help): Selected partition 1 Partition type (type L to list all types): Changed type of partition 'Linux' to 'Linux raid autodetect'.

Command (m for help): The partition table has been altered. Calling ioctl() to re-read partition table. Syncing disks.

[18082020-18:09:06] [format_and_partition_disks] Using 1 partitions /dev/sdc1 [18082020-18:09:06] [format_and_partition_disks] Next mount point appears to be /datadisks/disk1 [18082020-18:09:06] [format_and_partition_disks] only one disk (/dev/sdc) attached. mount it Creating filesystem with 33554176 4k blocks and 8388608 inodes Filesystem UUID: 66386890-7c5f-40d5-9f54-8f64f259b19f Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: 0/1024 done
Writing inode tables: 0/1024 done
Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: 0/1024 done

[18082020-18:09:08] [format_and_partition_disks] attempting to get UUID from /dev/sdc1 [18082020-18:09:08] [format_and_partition_disks] adding UUID: 66386890-7c5f-40d5-9f54-8f64f259b19f to fstab /dev/sdc1 [18082020-18:09:08] [format_and_partition_disks] calling fstab with UUID: 66386890-7c5f-40d5-9f54-8f64f259b19f and mount point: /datadisks/disk1 [18082020-18:09:08] [format_data_disks] finished partition and format attached disks [18082020-18:09:08] [apt-get] updating apt-get [18082020-18:09:10] [apt-get] updated apt-get [18082020-18:09:10] [install_ntp] installing ntp Reading package lists... Building dependency tree... Reading state information... The following package was automatically installed and is no longer required: grub-pc-bin Use 'apt autoremove' to remove it. The following additional packages will be installed: libopts25 Suggested packages: ntp-doc The following NEW packages will be installed: libopts25 ntp 0 upgraded, 2 newly installed, 0 to remove and 9 not upgraded. Need to get 575 kB of archives. After this operation, 1792 kB of additional disk space will be used. Get:1 http://azure.archive.ubuntu.com/ubuntu xenial/main amd64 libopts25 amd64 1:5.18.7-3 [57.8 kB] Get:2 http://azure.archive.ubuntu.com/ubuntu xenial-updates/main amd64 ntp amd64 1:4.2.8p4+dfsg-3ubuntu5.10 [517 kB] Fetched 575 kB in 1s (537 kB/s) Selecting previously unselected package libopts25:amd64.

(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 53967 files and directories currently installed.)

Preparing to unpack .../libopts25_1%3a5.18.7-3_amd64.deb ...

Unpacking libopts25:amd64 (1:5.18.7-3) ...

Selecting previously unselected package ntp.

Preparing to unpack .../ntp_1%3a4.2.8p4+dfsg-3ubuntu5.10_amd64.deb ...

Unpacking ntp (1:4.2.8p4+dfsg-3ubuntu5.10) ...

Processing triggers for libc-bin (2.23-0ubuntu11.2) ...

Processing triggers for man-db (2.7.5-1) ...

Processing triggers for ureadahead (0.100.0-19.1) ...

Processing triggers for systemd (229-4ubuntu21.28) ...

Setting up libopts25:amd64 (1:5.18.7-3) ...

Setting up ntp (1:4.2.8p4+dfsg-3ubuntu5.10) ...

sent invalidate(group) request, exiting

sent invalidate(passwd) request, exiting

sent invalidate(group) request, exiting

sent invalidate(group) request, exiting

sent invalidate(group) request, exiting

sent invalidate(passwd) request, exiting

sent invalidate(group) request, exiting

sent invalidate(passwd) request, exiting

sent invalidate(group) request, exiting

sent invalidate(passwd) request, exiting

sent invalidate(group) request, exiting

sent invalidate(passwd) request, exiting

sent invalidate(group) request, exiting

sent invalidate(passwd) request, exiting

sent invalidate(group) request, exiting

sent invalidate(passwd) request, exiting

sent invalidate(group) request, exiting

sent invalidate(passwd) request, exiting

sent invalidate(group) request, exiting

Processing triggers for libc-bin (2.23-0ubuntu11.2) ...

Processing triggers for ureadahead (0.100.0-19.1) ...

Processing triggers for systemd (229-4ubuntu21.28) ...

[18082020-18:09:21] [install_ntp] installed ntp [18082020-18:09:21] [install_ntpdate] installing ntpdate Reading package lists... Building dependency tree... Reading state information... The following package was automatically installed and is no longer required: grub-pc-bin Use 'apt autoremove' to remove it. The following NEW packages will be installed: ntpdate 0 upgraded, 1 newly installed, 0 to remove and 9 not upgraded. Need to get 48.7 kB of archives. After this operation, 173 kB of additional disk space will be used. Get:1 http://azure.archive.ubuntu.com/ubuntu xenial-updates/main amd64 ntpdate amd64 1:4.2.8p4+dfsg-3ubuntu5.10 [48.7 kB] Fetched 48.7 kB in 0s (94.1 kB/s) Selecting previously unselected package ntpdate.

(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 54029 files and directories currently installed.)

Preparing to unpack .../ntpdate_1%3a4.2.8p4+dfsg-3ubuntu5.10_amd64.deb ...

Unpacking ntpdate (1:4.2.8p4+dfsg-3ubuntu5.10) ...

Processing triggers for man-db (2.7.5-1) ...

Setting up ntpdate (1:4.2.8p4+dfsg-3ubuntu5.10) ...

[18082020-18:09:24] [install_ntpdate] installed ntpdate [18082020-18:09:33] [install_java] Option v set with value 7.8.0 [18082020-18:09:33] [install_java] updating apt-get [18082020-18:09:35] [install_java] updated apt-get [18082020-18:09:35] [install_java] not installing java, using JDK bundled with distribution [18082020-18:09:35] [install_es] installing Elasticsearch 7.8.0 [18082020-18:09:35] [install_es] download location - https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.8.0-amd64.deb?ultron=msft&gambit=azure [18082020-18:09:44] [install_es] downloaded Elasticsearch 7.8.0 elasticsearch-7.8.0-amd64.deb: OK Selecting previously unselected package elasticsearch. (Reading database ... 54043 files and directories currently installed.) Preparing to unpack elasticsearch-7.8.0-amd64.deb ... Creating elasticsearch group... OK Creating elasticsearch user... OK Unpacking elasticsearch (7.8.0) ... Setting up elasticsearch (7.8.0) ... Created elasticsearch keystore in /etc/elasticsearch/elasticsearch.keystore Processing triggers for ureadahead (0.100.0-19.1) ... Processing triggers for systemd (229-4ubuntu21.28) ... [18082020-18:10:08] [install_es] installed Elasticsearch 7.8.0 [18082020-18:10:08] [setup_data_disk] configuring disk /datadisks/disk1/elasticsearch/data [18082020-18:10:08] [setup_bootstrap_password] adding bootstrap.password to keystore [18082020-18:10:09] [setup_bootstrap_password] added bootstrap.password to keystore [18082020-18:10:09] [configure_elasticsearch_yaml] configure elasticsearch.yml file [18082020-18:10:09] [configure_elasticsearch_yaml] update configuration with data path list of /datadisks/disk1/elasticsearch/data [18082020-18:10:09] [configure_elasticsearch_yaml] update configuration with discovery.seed_hosts and cluster.initial_master_nodes set to ["epp02data-0","epp02data-1","epp02data-2","epp02data-3","epp02data-4"] [18082020-18:10:09] [configure_elasticsearch_yaml] configure master/client/data node type flags only master-0 only data-0 [18082020-18:10:09] [configure_elasticsearch_yaml] configure node as master and data [18082020-18:10:09] [install_jq] installing jq Reading package lists... Building dependency tree... Reading state information... The following package was automatically installed and is no longer required: grub-pc-bin Use 'apt autoremove' to remove it. The following additional packages will be installed: libonig2 The following NEW packages will be installed: jq libonig2 0 upgraded, 2 newly installed, 0 to remove and 9 not upgraded. Need to get 231 kB of archives. After this operation, 797 kB of additional disk space will be used. Get:1 http://azure.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 libonig2 amd64 5.9.6-1ubuntu0.1 [86.7 kB] Get:2 http://azure.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 jq amd64 1.5+dfsg-1ubuntu0.1 [144 kB] Fetched 231 kB in 0s (279 kB/s) Selecting previously unselected package libonig2:amd64.

(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 55086 files and directories currently installed.)

Preparing to unpack .../libonig2_5.9.6-1ubuntu0.1_amd64.deb ...

Unpacking libonig2:amd64 (5.9.6-1ubuntu0.1) ...

Selecting previously unselected package jq.

Preparing to unpack .../jq_1.5+dfsg-1ubuntu0.1_amd64.deb ...

Unpacking jq (1.5+dfsg-1ubuntu0.1) ...

Processing triggers for libc-bin (2.23-0ubuntu11.2) ...

Processing triggers for man-db (2.7.5-1) ...

Setting up libonig2:amd64 (5.9.6-1ubuntu0.1) ...

Setting up jq (1.5+dfsg-1ubuntu0.1) ...

Processing triggers for libc-bin (2.23-0ubuntu11.2) ...

[18082020-18:10:13] [install_jq] installed jq [18082020-18:10:13] [configure_awareness_attributes] configure fault and update domain attributes [18082020-18:10:13] [configure_awareness_attributes] configure shard allocation awareness using fault_domain and update_domain [18082020-18:10:13] [configure_elasticsearch_yaml] Set generated license type to trial [18082020-18:10:13] [configure_elasticsearch_yaml] Set X-Pack Security enabled [18082020-18:10:13] [configure_elasticsearch_yaml] setting bootstrap.memory_lock: true [18082020-18:10:13] [configure_elasticsearch] configuring elasticsearch default configuration [18082020-18:10:13] [configure_elasticsearch] configuring heap size from available memory [18082020-18:10:13] [configure_elasticsearch] configure elasticsearch heap size - 8009 megabytes [18082020-18:10:13] [configure_os_properties] configuring operating system level configuration [18082020-18:10:13] [configure_os_properties] configure systemd to start Elasticsearch service automatically when system boots [18082020-18:10:14] [configure_os_properties] configured operating system level configuration [18082020-18:10:14] [port_forward] setting up port forwarding from 9201 to 9200 [18082020-18:10:14] [port_forward] installing iptables-persistent Reading package lists... Building dependency tree... Reading state information... The following package was automatically installed and is no longer required: grub-pc-bin Use 'apt autoremove' to remove it. The following additional packages will be installed: netfilter-persistent The following NEW packages will be installed: iptables-persistent netfilter-persistent 0 upgraded, 2 newly installed, 0 to remove and 9 not upgraded. Need to get 13.3 kB of archives. After this operation, 79.9 kB of additional disk space will be used. Get:1 http://azure.archive.ubuntu.com/ubuntu xenial/universe amd64 netfilter-persistent all 1.0.4 [6786 B] Get:2 http://azure.archive.ubuntu.com/ubuntu xenial/universe amd64 iptables-persistent all 1.0.4 [6540 B] Preconfiguring packages ... Fetched 13.3 kB in 0s (19.3 kB/s) Selecting previously unselected package netfilter-persistent.

(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 55098 files and directories currently installed.)

Preparing to unpack .../netfilter-persistent_1.0.4_all.deb ...

Unpacking netfilter-persistent (1.0.4) ...

Selecting previously unselected package iptables-persistent.

Preparing to unpack .../iptables-persistent_1.0.4_all.deb ...

Unpacking iptables-persistent (1.0.4) ...

Processing triggers for man-db (2.7.5-1) ...

Processing triggers for ureadahead (0.100.0-19.1) ...

Processing triggers for systemd (229-4ubuntu21.28) ...

Setting up netfilter-persistent (1.0.4) ...

update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults

Setting up iptables-persistent (1.0.4) ...

Processing triggers for ureadahead (0.100.0-19.1) ...

Processing triggers for systemd (229-4ubuntu21.28) ...

/var/lib/waagent/custom-script/download/0/stderr

mke2fs 1.42.13 (17-May-2015) 18 Aug 18:09:33 ntpdate[4285]: no server suitable for synchronization found sent invalidate(group) request, exiting sent invalidate(passwd) request, exiting sent invalidate(group) request, exiting sent invalidate(group) request, exiting sent invalidate(group) request, exiting sent invalidate(passwd) request, exiting sent invalidate(group) request, exiting sent invalidate(passwd) request, exiting sent invalidate(group) request, exiting sent invalidate(passwd) request, exiting sent invalidate(group) request, exiting sent invalidate(passwd) request, exiting sent invalidate(group) request, exiting sent invalidate(passwd) request, exiting sent invalidate(group) request, exiting sent invalidate(passwd) request, exiting sent invalidate(group) request, exiting sent invalidate(passwd) request, exiting sent invalidate(group) request, exiting Synchronizing state of elasticsearch.service with SysV init with /lib/systemd/systemd-sysv-install... Executing /lib/systemd/systemd-sysv-install enable elasticsearch Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service. run-parts: executing /usr/share/netfilter-persistent/plugins.d/15-ip4tables save run-parts: executing /usr/share/netfilter-persistent/plugins.d/25-ip6tables save % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 292 100 265 100 27 990 100 --:--:-- --:--:-- --:--:-- 992 HTTP 503 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 476 100 449 100 27 3731 224 --:--:-- --:--:-- --:--:-- 3741 HTTP 401 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 494 100 467 100 27 3799 219 --:--:-- --:--:-- --:--:-- 3827 HTTP 401 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 488 100 461 100 27 3902 228 --:--:-- --:--:-- --:--:-- 3906 HTTP 401 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 484 100 457 100 27 3631 214 --:--:-- --:--:-- --:--:-- 3656 HTTP 401 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 508 100 481 100 27 4356 244 --:--:-- --:--:-- --:--:-- 4372 HTTP 401

russcam commented 4 years ago

No problem, thank you for sharing these.

As I suspected, the supplied passwords for the built-in users have not been applied successfully, which the following log lines show

[18082020-18:11:07] [apply_security_settings] start updating roles and users
{"error":{"root_cause":[{"type":"status_exception","reason":"Cluster state has not been recovered yet, cannot write to the [null] index"}],"type":"status_exception","reason":"Cluster state has not been recovered yet, cannot write to the [null] index"},"status":503}
[18082020-18:11:07] [apply_security_settings] updated built-in elastic superuser password
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/_security/user/kibana/_password]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/_security/user/kibana/_password]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}
[18082020-18:11:07] [apply_security_settings] updated built-in kibana user password
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/_security/user/logstash_system/_password]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/_security/user/logstash_system/_password]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}
[18082020-18:11:07] [apply_security_settings] updated built-in logstash_system user password
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/_security/user/beats_system/_password]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/_security/user/beats_system/_password]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}
[18082020-18:11:07] [apply_security_settings] updated built-in beats_system user password
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/_security/user/apm_system/_password]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/_security/user/apm_system/_password]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}
[18082020-18:11:07] [apply_security_settings] updated built-in apm_system user password
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/_security/user/remote_monitoring_user/_password]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/_security/user/remote_monitoring_user/_password]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}
[18082020-18:11:07] [apply_security_settings] updated built-in remote_monitoring_user user password

The first log line indicates that the cluster state was not ready when an attempt was made to set the elastic built-in user password.

The PR https://github.com/elastic/azure-marketplace/pull/362 adds some additional checks to wait for the cluster to be ready before updating passwords and has been merged in already. A new version of the template has been submitted for approval to publish on the Azure Marketplace, but has not yet been approved, so these changes are not yet available on the Azure Marketplace. I'll let you know once they are available.

In the meantime, you can

  1. deploy the template from the GitHub repository with these changes, targeting the 7.9 branch (you can use 7.9.0 as the esVersion parameter value to deploy Elastic Stack 7.9.0).

or

  1. set the bootstrap.password in the elastic keystore using elasticsearch-keystore tool to a known value on each node, and restart the node, then use elasticsearch-setup-passwords tool to set the passwords for all built-in users.. For example
#log in to each Elasticsearch node and do the following:

# set bootstrap.password in the keystore
BOOTSTRAP_PASSWORD="<set this to a password>"
echo "$BOOTSTRAP_PASSWORD" | /usr/share/elasticsearch/bin/elasticsearch-keystore add bootstrap.password -xf

# restart the Elasticsearch service
systemctl reload-or-restart elasticsearch.service

# Once bootstrap.password has been set on each node, do the following on one of the nodes and follow the prompts:
/usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive
AnilSunkesula commented 4 years ago

Thanks for the update. I have tried 2nd option. After I set the bootstrap password, Getting below error when I restart elasticsearch service

Job for elasticsearch.service failed because the control process exited with error code. See systemctl status elasticsearch.service and journalctl -xe for details.

elasticsearch status

● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/elasticsearch.service.d └─override.conf Active: failed (Result: exit-code) since Thu 2020-08-27 06:09:28 UTC; 10min ago Docs: https://www.elastic.co Process: 79489 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE) Main PID: 79489 (code=exited, status=1/FAILURE)

Aug 27 06:09:28 epp02data-0 systemd-entrypoint[79489]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) Aug 27 06:09:28 epp02data-0 systemd-entrypoint[79489]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) Aug 27 06:09:28 epp02data-0 systemd-entrypoint[79489]: at org.elasticsearch.cli.Command.main(Command.java:90) Aug 27 06:09:28 epp02data-0 systemd-entrypoint[79489]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) Aug 27 06:09:28 epp02data-0 systemd-entrypoint[79489]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) Aug 27 06:09:28 epp02data-0 systemd-entrypoint[79489]: For complete error details, refer to the log at /var/log/elasticsearch/LMAPAZ1ESPPDSHL02.log Aug 27 06:09:28 epp02data-0 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE Aug 27 06:09:28 epp02data-0 systemd[1]: Failed to start Elasticsearch. Aug 27 06:09:28 epp02data-0 systemd[1]: elasticsearch.service: Unit entered failed state. Aug 27 06:09:28 epp02data-0 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

jornalctl shows to check /var/log/elasticsearch/clustername.log file. Please find log file content

/var/log/elasticsearch/clustername.log

[2020-08-27T06:09:27,960][INFO ][o.e.p.PluginsService ] [epp02data-0] loaded module [x-pack-analytics] [2020-08-27T06:09:27,961][INFO ][o.e.p.PluginsService ] [epp02data-0] loaded module [x-pack-async-search] [2020-08-27T06:09:27,961][INFO ][o.e.p.PluginsService ] [epp02data-0] loaded module [x-pack-autoscaling] [2020-08-27T06:09:27,961][INFO ][o.e.p.PluginsService ] [epp02data-0] loaded module [x-pack-ccr] [2020-08-27T06:09:27,961][INFO ][o.e.p.PluginsService ] [epp02data-0] loaded module [x-pack-core] [2020-08-27T06:09:27,961][INFO ][o.e.p.PluginsService ] [epp02data-0] loaded module [x-pack-deprecation] [2020-08-27T06:09:27,962][INFO ][o.e.p.PluginsService ] [epp02data-0] loaded module [x-pack-enrich] [2020-08-27T06:09:27,962][INFO ][o.e.p.PluginsService ] [epp02data-0] loaded module [x-pack-eql] [2020-08-27T06:09:27,962][INFO ][o.e.p.PluginsService ] [epp02data-0] loaded module [x-pack-graph] [2020-08-27T06:09:27,962][INFO ][o.e.p.PluginsService ] [epp02data-0] loaded module [x-pack-identity-provider] [2020-08-27T06:09:27,962][INFO ][o.e.p.PluginsService ] [epp02data-0] loaded module [x-pack-ilm] [2020-08-27T06:09:27,962][INFO ][o.e.p.PluginsService ] [epp02data-0] loaded module [x-pack-logstash] [2020-08-27T06:09:27,962][INFO ][o.e.p.PluginsService ] [epp02data-0] loaded module [x-pack-ml] [2020-08-27T06:09:27,963][INFO ][o.e.p.PluginsService ] [epp02data-0] loaded module [x-pack-monitoring] [2020-08-27T06:09:27,963][INFO ][o.e.p.PluginsService ] [epp02data-0] loaded module [x-pack-ql] [2020-08-27T06:09:27,963][INFO ][o.e.p.PluginsService ] [epp02data-0] loaded module [x-pack-rollup] [2020-08-27T06:09:27,964][INFO ][o.e.p.PluginsService ] [epp02data-0] loaded module [x-pack-security] [2020-08-27T06:09:27,964][INFO ][o.e.p.PluginsService ] [epp02data-0] loaded module [x-pack-sql] [2020-08-27T06:09:27,964][INFO ][o.e.p.PluginsService ] [epp02data-0] loaded module [x-pack-voting-only-node] [2020-08-27T06:09:27,964][INFO ][o.e.p.PluginsService ] [epp02data-0] loaded module [x-pack-watcher] [2020-08-27T06:09:27,965][INFO ][o.e.p.PluginsService ] [epp02data-0] no plugins loaded [2020-08-27T06:09:28,025][INFO ][o.e.e.NodeEnvironment ] [epp02data-0] using [1] data paths, mounts [[/datadisks/disk1 (/dev/sdc1)]], net usable_space [119.3gb], net total_space [125.8gb], types [ext4] [2020-08-27T06:09:28,026][INFO ][o.e.e.NodeEnvironment ] [epp02data-0] heap size [7.8gb], compressed ordinary object pointers [true] [2020-08-27T06:09:28,104][ERROR][o.e.b.Bootstrap ] [epp02data-0] Exception org.elasticsearch.ElasticsearchException: failed to bind service at org.elasticsearch.node.Node.(Node.java:652) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.node.Node.(Node.java:266) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) [elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) [elasticsearch-cli-7.8.0.jar:7.8.0] at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.8.0.jar:7.8.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.8.0.jar:7.8.0] Caused by: java.io.IOException: failed to test writes in data directory [/datadisks/disk1/elasticsearch/data/nodes/0] write permission is required at org.elasticsearch.env.NodeEnvironment.tryWriteTempFile(NodeEnvironment.java:1254) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.env.NodeEnvironment.assertCanWrite(NodeEnvironment.java:1217) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:315) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.node.Node.(Node.java:335) ~[elasticsearch-7.8.0.jar:7.8.0] ... 11 more Caused by: java.nio.file.FileSystemException: /datadisks/disk1/elasticsearch/data/nodes/0/.es_temp_file: Input/output error at sun.nio.fs.UnixException.translateToIOException(UnixException.java:100) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?] at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:249) ~[?:?] at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:110) ~[?:?] at java.nio.file.Files.deleteIfExists(Files.java:1185) ~[?:?] at org.elasticsearch.env.NodeEnvironment.tryWriteTempFile(NodeEnvironment.java:1250) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.env.NodeEnvironment.assertCanWrite(NodeEnvironment.java:1217) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:315) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.node.Node.(Node.java:335) ~[elasticsearch-7.8.0.jar:7.8.0] ... 11 more [2020-08-27T06:09:28,112][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [epp02data-0] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: IOException[failed to test writes in data directory [/datadisks/disk1/elasticsearch/data/nodes/0] write permission is required]; nested: FileSystemException[/datadisks/disk1/elasticsearch/data/nodes/0/.es_temp_file: Input/output error]; at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.8.0.jar:7.8.0] at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.8.0.jar:7.8.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.8.0.jar:7.8.0] Caused by: org.elasticsearch.ElasticsearchException: failed to bind service at org.elasticsearch.node.Node.(Node.java:652) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.node.Node.(Node.java:266) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.8.0.jar:7.8.0] ... 6 more Caused by: java.io.IOException: failed to test writes in data directory [/datadisks/disk1/elasticsearch/data/nodes/0] write permission is required at org.elasticsearch.env.NodeEnvironment.tryWriteTempFile(NodeEnvironment.java:1254) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.env.NodeEnvironment.assertCanWrite(NodeEnvironment.java:1217) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:315) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.node.Node.(Node.java:335) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.node.Node.(Node.java:266) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.8.0.jar:7.8.0] ... 6 more Caused by: java.nio.file.FileSystemException: /datadisks/disk1/elasticsearch/data/nodes/0/.es_temp_file: Input/output error at sun.nio.fs.UnixException.translateToIOException(UnixException.java:100) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?] at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:249) ~[?:?] at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:110) ~[?:?] at java.nio.file.Files.deleteIfExists(Files.java:1185) ~[?:?] at org.elasticsearch.env.NodeEnvironment.tryWriteTempFile(NodeEnvironment.java:1250) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.env.NodeEnvironment.assertCanWrite(NodeEnvironment.java:1217) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:315) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.node.Node.(Node.java:335) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.node.Node.(Node.java:266) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.8.0.jar:7.8.0] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.8.0.jar:7.8.0] ... 6 more

russcam commented 4 years ago

Caused by: java.io.IOException: failed to test writes in data directory [/datadisks/disk1/elasticsearch/data/nodes/0] write permission is required

The log indicates that the Elasticsearch process does not have write permissions to /datadisks/disk1/elasticsearch/data/nodes/0.

/datadisks/disk1 is mounted as part of the installation, with /datadisks/disk1/elasticsearch created and owned by elasticsearch user and group, with file mode 755:

https://github.com/elastic/azure-marketplace/blob/ab16f389c7c043a79fffa955411cdc96e5062fb3/src/scripts/elasticsearch-install.sh#L329-L348

/datadisks/disk1/elasticsearch/data is configured as path.data in elasticsearch.yml

https://github.com/elastic/azure-marketplace/blob/ab16f389c7c043a79fffa955411cdc96e5062fb3/src/scripts/elasticsearch-install.sh#L907-L920

Please check the permissions that /datadisks/disk1/elasticsearch/data has, and ensure that it's owned by elasticsearch:elasticsearch.

russcam commented 4 years ago

Hi @AnilSunkesula, have you been able to get further with this?

AnilSunkesula commented 4 years ago

Hi @russcam, Thank you very much for all your suggestions. I didn't get chance to check the permission on datadisk path as we deleted the deployment. We are working on an alternative solution to configure the Elastic cluster manually instead of using Azure Market place image. Please feel free to close the ticket for now. We will contact you if we require any assistance in future.

russcam commented 4 years ago

Thanks for letting me know, @AnilSunkesula. If you did want to try the Marketplace again, there is a new version available that should address the original issue you had.

attilabalazsy commented 2 years ago

Hi guys, @russcam, this issue is still present in elasticsearch 8.3. When the password contains exclamation marks (!), then the user cannot authenticate to ES.