Support authorization using api keys in the agents

[estolfo]

The APM server has added support for Api keys and agents should support it for stack version 7.6. This will allow finer authorization rules going forward, as Api keys can be created for specific use cases (e.g. upload sourcemaps but not ingest data). In the future, they can also be created for specific instrumented services. The use of Api keys also does not have the disadvantage of secret tokens: inevitable downtime/lost events when they need to be changed.

Supporting Api keys for agents requires allowing the user to configure an option, api_key. The api_key must be a Base64 encoded string, corresponding to the id:key. Currently, the apm server does not have tooling to allow a user to generate an api key, but when it does, the user will obtain the Base64 encoded string and can set it directly to the api_key setting of the agent.

Simultaneous support for a Secret token and Api key

The apm server supports both a secret token and api key. When doing end-to-end testing, you should observe the following behavior:

apm-server.api_key.enabled	apm-server.secret_token configured	Valid API Key	Invalid API Key	Valid secret token	Invalid secret token	No auth header
true	true	allowed	denied	allowed	denied	denied
true	false	allowed	denied	denied	denied	denied
false	true	denied	denied	allowed	denied	denied
false	false	allowed	allowed	allowed	allowed	allowed

If the user sets both a secret_token and api_key for the agent, the api_key should be used.

Testing

As the apm server does not yet have tooling for generating an Api key, the key must be created manually in Elasticsearch.

Here is an example set of commands to generate an Api key:

PUT /_security/privilege
{
 "apm": {
   "write_sourcemap": {
     "actions": [ "sourcemap:write" ]
   },
   "write_event": {
     "actions": [ "event:write" ]
   },
   "read_agent_config": {
     "actions": [ "config_agent:read" ]
   } }
}

POST _security/api_key
{
   "name": "apm-backend",
   "role_descriptors": {
     "apm-backend": {
        "applications": [
           {
             "application": "apm",
             "privileges": ["*"],
             "resources": ["*"]
            }
          ]
        }
    }
}

The id:key then must be converted to a base64 encoded string before being configured as the api_key for the agent. For example, the last command above would return:

{
  "id" : "OEXB_m4B3SbfbZmPv8vb",
  "name" : "apm-backend",
  "api_key" : "XkZVRlCSSPWXM0IR0QdXDQ"
}

The api key to set in the agent would then be: api_key = Base64.encode(“OEXB_m4B3SbfbZmPv8vb:XkZVRlCSSPWXM0IR0QdXDQ”)

Functional specifications

The Ruby client has implemented support for the Api key in this PR. There are also functional specs written using Cucumber/Gherkin. The Gherkin specs can be shared amongst agents to ensure that we are consistent in behavior on this feature.

Note: at this time, the agent takes the api_key as a Base64 encoded string. Tooling has not yet been added to allow creating an api key via the apm server that is returned as a Base64 encoded string. That means, as agents test right now, they must manually create the api key via the Elasticsearch commands written above and encode the id:key to set as the agent api_key config option.

cc @elastic/apm-agent-devs

[SergeyKleyman]

@estolfo Thank you for the detailed spec.

(Note: at this time, the agent takes the id:key as the api_key configuration and does the conversion to a Base64 string itself. It will be changed when the apm server tooling for api keys is added)

It would be great if agents won't need to deal with Base64 encoding - it has a few variations and agent devs will have to make sure that whatever implementation they use is compatible with the one used by the backend.

[estolfo]

I agree with you, @SergeyKleyman. I plan on running it by all the agent engineers in our meeting this week to make sure there's no advantage of having agents do the encoding that I might be missing. In the meantime, I'm going to update my implementation to require the api_key configuration to be a base64-encoded string, as it's highly likely that will be the final approach.

[mikker]

Very nice work, Emily! Also this shows how the shared feature specs can help a lot with aligning how the agents handle all the little details.

Do we have a target stack version for this to be in the agents?

[estolfo]

Both the tooling and API key support will be available from 7.6 on.

[gregkalapos]

Do we plan to have any integration test with a real apm server on this? I feel that'd be an important test.

[estolfo]

@gregkalapos Are you asking about including a test in our integration tests or manual end-to-end testing? I did end-to-end testing manually myself when developing the feature.

[gregkalapos]

@gregkalapos Are you asking about including a test in our integration tests or manual end-to-end testing? I did end-to-end testing manually myself when developing the feature.

I'm asking about automated tests - so, yes, basically in our integration tests. I know we recently turned on HTTPS in the integration tests - I just thought it'd be nice to somehow also cover this to make sure we don't break it later - or if we break we'll know it immediately.

Manual testing is definitely an option, I'd be ok to merge the PR implementing this in .NET after manual testing, so not that this'd be a blocker, just asking, because I always prefer to also have an automated test.

[estolfo]

Yes, I agree @elastic/observablt-robots Can we add tests using API key auth into our automated integration tests?

[kuisathaverat]

sure, I understand that we have to follow these steps:

• Start the Elastic Stack (Elasticsearch+Kibana+APM)
• Execute a couple of commands to grab an ApiKey
• Transform the ApiKey to base64
• Pass the ApiKey to all the agents in some way

I have a doubt in the last step, Is there an environment variable to pass this ApiKey? I did not see any on the PR, What would be the way to pass this ApiKey to all agents?

[gregkalapos]

I have a doubt in the last step, Is there an environment variable to pass this ApiKey?

ELASTIC_APM_API_KEY I guess - that's what we have in the .NET PR.

[estolfo]

I have a doubt in the last step, Is there an environment variable to pass this ApiKey?

For Ruby, ELASTIC_APM_API_KEY would also work.

[kuisathaverat]

cool, I've opened an issue to the integration test we will add a new parameter to enable ApiKey authentication on every agent, we have to figure out how to generate the ApiKey in runtime and pass it to every agent.

[estolfo]

Thanks @kuisathaverat!

[SylvainJuge]

Just in case someone else struggles when testing this manually:

When using base64 command line tool, don't forget to add -n to the echo command, otherwise you'll have an extra EOL character included in the base64 key.

echo -n 'OEXB_m4B3SbfbZmPv8vb:XkZVRlCSSPWXM0IR0QdXDQ' | base64
# T0VYQl9tNEIzU2JmYlptUHY4dmI6WGtaVlJsQ1NTUFdYTTBJUjBRZFhEUQ==
# --> this one works -----------------[ without EOL ] ------^

echo -n 'OEXB_m4B3SbfbZmPv8vb:XkZVRlCSSPWXM0IR0QdXDQ' | base64
# T0VYQl9tNEIzU2JmYlptUHY4dmI6WGtaVlJsQ1NTUFdYTTBJUjBRZFhEUQo=
# --> this one won't work ----------[ thanks to EOL ] ------^

[SylvainJuge]

Also, I've made a few minor changes to the original spec https://github.com/elastic/apm/pull/226

There is no impact on production code, only test suites should be lightly impacted when updating.

[graphaelli]

This is complete