Elastic Agent: failed to enroll (HTTP 401)

[mtojek]

Today I updated the testing environment and it failed enrollment with:

➜  package-registry git:(stream-archived-content-2) ✗ docker logs 51b27bd78f87
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    22  100    22    0     0      2      0  0:00:11  0:00:09  0:00:02     5
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   277  100   221  100    56   2848    721 --:--:-- --:--:-- --:--:--  2870
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    58  100    58    0     0    962      0 --:--:-- --:--:-- --:--:--   966
{"isInitialized":true}{"statusCode":400,"error":"Bad Request","message":"[request body]: types that failed validation:\n- [request body.0.admin_username]: definition for this key is missing\n- [request body.1]: expected value to equal [null]"}{"list":[],"success":true,"total":0,"page":1,"perPage":20}
null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    82  100    82    0     0    794      0 --:--:-- --:--:-- --:--:--   796
null
The Elastic Agent is currently in Experimental and should not be used in production
2020-05-18T10:46:58Z DEBUG  client.go:170   Request method: POST, path: /api/ingest_manager/fleet/agents/enroll
fail to enroll: fail to execute request to Kibana: Status code: 401, Kibana returned an error: Unauthorized, message: [security_exception] missing authentication credentials for REST request [/_security/_authenticate], with { header={ WWW-Authenticate={ 0="ApiKey" & 1="Basic realm=\"security\" charset=\"UTF-8\"" } } }```

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[michalpristas]

i have a PR https://github.com/elastic/beats/pull/18611

[ph]

Fixed in https://github.com/elastic/beats/pull/18611

[mtojek]

Let me verify it first.

[mtojek]

@michalpristas I used the image "docker.elastic.co/beats/elastic-agent:8.0.0-SNAPSHOT", according to the post: https://github.com/elastic/beats/issues/18341#issuecomment-625245040 it should be updated on daily basis (I suppose at midnight?)

Reproduce (integrations):

$ cd testing/environments
$ docker-compose -f snapshot.yml -f agent.yml -f local.yml pull
$ docker-compose -f snapshot.yml -f agent.yml -f local.yml up

agent logs:

➜  package-registry git:(stream-archived-content-2) docker logs 6f86d2142c07
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    22  100    22    0     0      2      0  0:00:11  0:00:08  0:00:03     5
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   277  100   221  100    56   2703    684 --:--:-- --:--:-- --:--:--  2728
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    58  100    58    0     0    867      0 --:--:-- --:--:-- --:--:--   878
{"isInitialized":true}{"statusCode":400,"error":"Bad Request","message":"[request body]: types that failed validation:\n- [request body.0.admin_username]: definition for this key is missing\n- [request body.1]: expected value to equal [null]"}{"list":[],"success":true,"total":0,"page":1,"perPage":20}
null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    82  100    82    0     0    959      0 --:--:-- --:--:-- --:--:--   953
null
The Elastic Agent is currently in Experimental and should not be used in production
2020-05-19T08:24:31Z DEBUG  client.go:170   Request method: POST, path: /api/ingest_manager/fleet/agents/enroll
fail to enroll: fail to execute request to Kibana: Status code: 401, Kibana returned an error: Unauthorized, message: [security_exception] missing authentication credentials for REST request [/_security/_authenticate], with { header={ WWW-Authenticate={ 0="ApiKey" & 1="Basic realm=\"security\" charset=\"UTF-8\"" } } }

[michalpristas]

change is not part of the snapshot. all commits from yesterday are missing you can verify by running ./elastic-agent version there is a commit hash you can then look which commit was the last included

[mtojek]

elastic-agent version:

Agent version is 8.0.0-SNAPSHOT (build: eaf196d10e94ad9668febadcc84610fee28e7b79 at 2020-05-18 08:45:19 +0000 UTC)

Do you know when can I expect this change to be published? I think we need to decrease the delivery time. Should I use a different image?

[ph]

it's a daily snapshot, build by the release manager from our internal ci. The way to test would be to pickup master and run mage package in the x-pack/elastic-agent folder.

We used to have our own packaging jobs in https://beats-ci.elastic.co/job/elastic+beats+master+package/, we can still trigger them manually but that won't push images to docker.

Concerning decreasing the delivery times, I don't know if we have something that builds them more often maybe @kuisathaverat knowns?

[mtojek]

I managed to pull the updated image today, the issue seems to be fixed (thanks!). Let's keep it open until @kuisathaverat responds.

[kuisathaverat]

We used to have our own packaging jobs in https://beats-ci.elastic.co/job/elastic+beats+master+package/, we can still trigger them manually but that won't push images to docker.

This job is superseded by https://beats-ci.elastic.co/job/Beats/job/packaging/, this build every time you make a merge in beats repo and the build success (master, 7.x, 7.7, and 7.8 branches) the it push to the docker.elastic.co/observability-ci namespace.

[2020-05-19T16:53:44.346Z] + docker tag docker.elastic.co/beats/elastic-agent:8.0.0-SNAPSHOT docker.elastic.co/observability-ci/elastic-agent:8.0.0-SNAPSHOT
[2020-05-19T16:53:44.346Z] + docker push docker.elastic.co/observability-ci/elastic-agent:8.0.0-SNAPSHOT
[2020-05-19T16:53:44.346Z] The push refers to repository [docker.elastic.co/observability-ci/elastic-agent]
[2020-05-19T16:53:44.346Z] cf5ea4895659: Preparing
[2020-05-19T16:53:44.346Z] 10d2a2e30a28: Preparing
[2020-05-19T16:53:44.346Z] 258e55292314: Preparing
[2020-05-19T16:53:44.346Z] f22531109fcd: Preparing
[2020-05-19T16:53:44.346Z] 0015282596da: Preparing
[2020-05-19T16:53:44.346Z] 4fe65ce74344: Preparing
[2020-05-19T16:53:44.346Z] edf3aa290fb3: Preparing
[2020-05-19T16:53:44.346Z] 4fe65ce74344: Waiting
[2020-05-19T16:53:44.346Z] edf3aa290fb3: Waiting
[2020-05-19T16:53:46.881Z] f22531109fcd: Pushed
[2020-05-19T16:53:46.881Z] 258e55292314: Pushed
[2020-05-19T16:53:46.881Z] edf3aa290fb3: Layer already exists
[2020-05-19T16:53:46.881Z] 10d2a2e30a28: Pushed
[2020-05-19T16:53:53.465Z] cf5ea4895659: Pushed
[2020-05-19T16:53:53.465Z] 4fe65ce74344: Pushed
[2020-05-19T16:53:53.465Z] 0015282596da: Pushed
[2020-05-19T16:53:55.996Z] 8.0.0-SNAPSHOT: digest: sha256:95fb4c0e35cb839ab03fab263c70a346ce7dea39185ea4337aeebd64d4a2ea7c size: 1786
[2020-05-19T16:53:55.996Z] + docker tag docker.elastic.co/beats/elastic-agent:8.0.0-SNAPSHOT docker.elastic.co/observability-ci/elastic-agent:b43c1dbb84c42ef24baf2ca8b0abccd9ca184d80
[2020-05-19T16:53:55.996Z] + docker push docker.elastic.co/observability-ci/elastic-agent:b43c1dbb84c42ef24baf2ca8b0abccd9ca184d80
[2020-05-19T16:53:55.996Z] The push refers to repository [docker.elastic.co/observability-ci/elastic-agent]
[2020-05-19T16:53:55.996Z] cf5ea4895659: Preparing
[2020-05-19T16:53:55.996Z] 10d2a2e30a28: Preparing
[2020-05-19T16:53:55.996Z] 258e55292314: Preparing
[2020-05-19T16:53:55.996Z] f22531109fcd: Preparing
[2020-05-19T16:53:55.996Z] 0015282596da: Preparing
[2020-05-19T16:53:55.996Z] 4fe65ce74344: Preparing
[2020-05-19T16:53:55.996Z] edf3aa290fb3: Preparing
[2020-05-19T16:53:55.996Z] 4fe65ce74344: Waiting
[2020-05-19T16:53:55.996Z] edf3aa290fb3: Waiting
[2020-05-19T16:53:56.255Z] 258e55292314: Layer already exists
[2020-05-19T16:53:56.255Z] 0015282596da: Layer already exists
[2020-05-19T16:53:56.255Z] 10d2a2e30a28: Layer already exists
[2020-05-19T16:53:56.255Z] f22531109fcd: Layer already exists
[2020-05-19T16:53:56.255Z] cf5ea4895659: Layer already exists
[2020-05-19T16:53:56.513Z] 4fe65ce74344: Layer already exists
[2020-05-19T16:53:56.513Z] edf3aa290fb3: Layer already exists
[2020-05-19T16:53:57.081Z] b43c1dbb84c42ef24baf2ca8b0abccd9ca184d80: digest: sha256:95fb4c0e35cb839ab03fab263c70a346ce7dea39185ea4337aeebd64d4a2ea7c size: 1786

[mtojek]

I understand that it's pushed to "observability-ci" namespace after merge and build succeeds. What about the "docker.elastic.co/beats/elastic-agent:8.0.0-SNAPSHOT"? How often is this one pushed?

[kuisathaverat]

this image is pushed by elastic+release-manager+master+unified-snapshot this job from the internal-ci is scheduled daily, it put the snapshot on S3 and Nexus, but I am not sure how the Docker images are pushed to docker.elastic.co/beats I can ask the infra team that it is the one how manages that process.

[kuisathaverat]

Ok, I found it elastic+infra+master+release-tekton-snapshot-docker-artifacts is a Tekton process the one how pushed the Docker images. That is trigger by elastic+infra+master+release-multijob-post-snapshot-master that it is trigger by elastic+release-manager+master+unified-snapshot, it is build daily. Right now is building, because is a whole products release build it takes about 6h, and it is trigger about 7.30 AM UTC all days, so if everything goes fine, it release the Docker image every day about 13:30-14:00 PM UTC

[ph]

@kuisathaverat @mtojek I am not sure we need to solve it yet, we can create an issue in the robots repository and track it there. Hopefully, as more stable, we get the less of these issues happen.

going to close this.