[Request] Document Developer Setup Docker local

[ollyhowell]

Draft tutorial from @danielkhan:

Developer Setup Docker local

[Intro … E.g. Elastic observability is free and open, can be installed locally, supports many languages and OpenTelemetry. […]
This documentation shows how to get started with Elastic Observability in 5 minutes. […]]
​ Install Get Docker | Docker Documentation and Install Docker Compose | Docker Documentation . Please make sure to dedicate ​ [Alternatively to the instructions provided here, could we also provide some artefact on e.g. GitHub to simply clone?]
​ Create a directory elastic-observability somewhere on your system. ​ In this directory, create a file docker/compose.yml with this content: ​

version: '2.2'
services:
elasticsearch:
container_name: elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
environment:
- bootstrap.memory_lock=true
- cluster.name=docker-cluster
- cluster.routing.allocation.disk.threshold_enabled=false
- discovery.type=single-node
- xpack.security.enabled=true
- xpack.security.authc.api_key.enabled=true
- ES_JAVA_OPTS=-Xms750m -Xmx750m
ulimits:
memlock:
hard: -1
soft: -1
volumes:
- esdata:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- esnet
healthcheck:
interval: 20s
retries: 10
test: curl -s http://localhost:9200/_cluster/health | grep -vq '"status":"red"'
​
kibana:
container_name: kibana
image: docker.elastic.co/kibana/kibana:${VERSION}
depends_on:
elasticsearch:
condition: service_healthy
environment:
- ELASTICSEARCH_URL=http://elasticsearch:9200
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
- ELASTICSEARCH_USERNAME=kibana_system
- ELASTICSEARCH_PASSWORD=${ELASTICSEARCH_KIBANA_PASSWORD}
ports:
- 5601:5601
networks:
- esnet
healthcheck:
interval: 10s
retries: 20
test: curl -u ‘kibana_system:${ELASTICSEARCH_KIBANA_PASSWORD}’ —write-out ‘HTTP %{http_code}’ —fail —silent —output /dev/null http://localhost:5601/api/status

apm-server: image: docker.elastic.co/apm/apm-server:${VERSION} depends_on: elasticsearch: condition: service_healthy kibana: condition: service_healthy cap_add: [“CHOWN”, “DAC_OVERRIDE”, “SETGID”, “SETUID”] cap_drop: [“ALL”] ports:

• 8200:8200 networks:
• esnet command: > apm-server -e -E apm-server.rum.enabled=true -E setup.kibana.host=kibana:5601 -E apm-server.kibana.username=elastic -E apm-server.kibana.password=${ELASTICSEARCH_PASSWORD} -E setup.template.settings.index.number_of_replicas=0 -E apm-server.kibana.enabled=true -E apm-server.kibana.host=kibana:5601 -E output.elasticsearch.hosts=[“elasticsearch:9200”] -E output.elasticsearch.username=“elastic” -E output.elasticsearch.password=${ELASTICSEARCH_PASSWORD} ​ healthcheck: interval: 10s retries: 12 test: curl —write-out ‘HTTP %{http_code}’ —fail —silent —output /dev/null http://localhost:8200/ ​ volumes: esdata: name: esdata driver: local ​ networks: esnet: name: esnet driver: bridge

  ​
  Additionally, create a file `.env` that will hold your configuration:
  ​

  VERSION=7.16.0-SNAPSHOT // Needs to change depending on the doc page viewed ELASTICSEARCH_KIBANA_PASSWORD= ELASTICSEARCH_PASSWORD=

  ​
  **ATTENTION:** When using Git, please make sure to add `.env` to your `.gitignore` file to not accidentally push credentials to GitHub.
  ​
  Now in this directory, run `docker compose up elasticsearch` to download and start Elasticsearch.
  Once Elasticsearch is running , run `docker exec elasticsearch /bin/bash -c ‘bin/elasticsearch-setup-passwords auto —batch —url http://elasticsearch:9200'`
  ​
  This will create and store a set of passwords in Elastic.
  Store the output of this command at a safe place.
  ​
  Open your `.env` file and fill copy the password for the`kibana_system` and `elastic` user to their respective config variables:
  ​

  VERSION=7.16.0-SNAPSHOT ELASTICSEARCH_KIBANA_PASSWORD= ELASTICSEARCH_PASSWORD=

  
  ​
  Now stop docker and run it again using `docker compose up` (without `elasticsearch`).
  This will start Elasticsearch, Kibana and the APM Server.
  Please wait until bloglines for `app-server` show up on the console output.
  Please ignore lines, that indicate an authentication error.
  ​
  > [I’m getting this - currently trying to figure out what it is - does not have an impact]]  
  > elasticsearch  | {"type": "server", "timestamp": "2021-11-25T12:26:05,969Z", "level": "INFO", "component": "o.e.x.s.a.RealmsAuthenticator", "cluster.name": "docker-cluster", "node.name": "8d05684f22de", "message": "Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]", "cluster.uuid": "BOGGeCrEQz-1vLjiII1NBQ", "node.id": "isSa2LI4Sa2mwmXxa63y1A"    
  ​
  Now navigate to http://localhost:5601 in your browser and log in with the username `elastic` and the password you generated before.
  Click on the Observability tile and follow the instructions on screen to set up APM.
  ​
  ​
  > [In my tests, I saw that the Docker default memory setting wasn’t enough on my Silicon MBP and the Elastic server crashed with Error 137.  
  > Can we verify and propose safe memory limits? ]  
  ![](&&&SFLOCALFILEPATH&&&Capto_Capture%202021-11-25_12-50-18_PM.png)

[bmorelli25]

I don't think we want to promote the APM Server binary as the best way for users to set up a Docker dev environment. I think the Elastic Stack + Elastic Agent would be better to document. As it is, this guide really only sets up APM, not Elastic Observability. Documenting Elastic Agent instead would fix that.

FWIW, we do already have documentation that is similar to this, although perhaps not as detailed (and certainly hasn't been updated in a while). The APM Server Legacy Quick Start: https://www.elastic.co/guide/en/apm/guide/current/quick-start-overview.html

[ollyhowell]

@danielkhan what do you think?

[danielkhan]

@bmorelli25 can you describe the setup you'd recommend to a local development setup?

Also, can we test the quick start you referenced and see what needs to be updated? I created my guide months ago and as far as I remember, the solutions in our docs didn't work with 7.16. If I remember correctly, it was because security is enabled and required now by default. If this guide does not work with 8.0, how can we ensure that such artifacts are kept up-to-date in the future, @ollyhowell?

[ollyhowell]

It sounds like there's an overlap between the above quick start and the APM one but neither are ideal / up-to-date. I've not ran thru either setup myself yet but let's work out what the 'ideal' setup would be before you get too deep in this issue, @colleenmcginnis

To the general point about keeping docs up-to-date, the main problem is we're under resourced so there's a never ending To Do list and plenty of areas that don't get enough attention. We have talked about testing documentation and having 'upgrade checklists' for each release to run thru and spot these kind of issues. We don't have a great answer for this just now but it is on our radar and continuing to encourage devs, PMs etc. to flag and contribute to these issues is a no brainer.

[danielkhan]

How should we proceed with this? @bmorelli25 brought up that we want to recommend a different setup. Who do we think is responsible for providing guidance in cases like this? @alex-fedotyev, @simitt do you have a preferred way we should document here? The goal is to have a quick and easy way to spin up a local elastic setup for o11y.

[simitt]

Ensuring that docs in the APM Server guide are up-to-date has been the responsibility of the APM Server team. I suggest we keep owning that for now. So ideally, whenever we update or implement a new feature, there is a dedicated docs issue that we share with the tech writer team and that gets scheduled, or if it is just tiny updates, engineers might include the changes directly in their PR.

It becomes a bit more tricky with Elastic Agent. I'd say that ideally the Elastic Agent team (@jlind23 ) has a comprehensive documentation with examples how to get started, and keeps these docs up to date. APM Server should still have its own docs and examples, but link to the Elastic Agent docs as much as possible to avoid duplication and getting out of sync.

[simitt]

For the concrete ask to document docker with Elastic Agent and Fleet, we will need to provide extra guidance on how to add APM Server Integration to the Agent policy to which the Elastic Agent is enrolled.

[danielkhan]

I think this discussion shows an inherent problem.

This was the original claim by @bmorelli25

I don't think we want to promote the APM Server binary as the best way for users to set up a Docker dev environment. I think the Elastic Stack + Elastic Agent would be better to document.

For a dev setup, one would need to run Agent and Fleet Server in a Docker container each and, even with an ideal Docker file, then also get the configuration within Kibana right so that Agent and Fleet are enrolled and connected. In this case, all components would run isolated in a Docker container. Does it provide any benefit compared to sending telemetry to the APM server? I tried setting this up and it's really not easy.

As it is, this guide really only sets up APM, not Elastic Observability. Documenting Elastic Agent instead would fix that.

I really want this to be clarified. What would using Agent change in this particular case?

I can not see any benefit from using Agent for this particular use case and if we want to default to it we have to make it simple to set it up.

[bmorelli25]

As it is, this guide really only sets up APM, not Elastic Observability. Documenting Elastic Agent instead would fix that.

I really want this to be clarified. What would using Agent change in this particular case?

A few thoughts here. First — and Silvia, correct me if this has changed — APM Server standalone will soon be deprecated in favor of Elastic Agent + the APM integration. That's a good enough reason in itself to avoid documenting the standalone (now legacy) method of running APM. In addition, the Docker guide in this issue only sets up Elasticsearch, Kibana, and APM Server. It's a nice way to get started with Elastic APM, but it excludes the rest of Elastic Observability.

Using Elastic Agent, instead, would be more observability focussed — users can add the APM integration for APM, but also have access to our other Observability integrations: synthetics, host logs/metrics, etc.

[danielkhan]

A few thoughts here. First — and Silvia, correct me if this has changed — APM Server standalone will soon be deprecated in favor of Elastic Agent + the APM integration.

That's valid. Do we know who drives this decision and makes the call @simitt?

Using Elastic Agent, instead, would be more observability focussed — users can add the APM integration for APM, but also have access to our other Observability integrations: synthetics, host logs/metrics, etc.

The thing is that from a Docker container all you will see are the metrics from the container Agent runs on and also Synthetics isn't relevant in this scenario either.

[bmorelli25]

The thing is that from a Docker container all you will see are the metrics from the container Agent runs on and also Synthetics isn't relevant in this scenario either.

Maybe I'm misunderstanding the goal of this docker setup tutorial. Isn't this a way for users to easily test out Elastic Observability? To do that, don't we want to give them a playground where they can install whatever integrations they choose?

[danielkhan]

Cloud provides a viable playground for testing the whole range of observability. This is for folks that want to use Elastic to mostly collect traces during development. The purpose is to provide something similar as shown here: https://www.jaegertracing.io/docs/1.29/getting-started/ Run it, send traces and see them in Elastic. This can be either through an APM agent or OpenTelemetry. We support both very well and could be a viable alternative to Jaeger if we make it easy to set it up.

[bmorelli25]

Oh. In that case, let's start over.

A legacy (APM Server standalone) docker guide exists: https://www.elastic.co/guide/en/apm/guide/current/quick-start-overview.html. It's similar to what you have in this issue. We can update it and add context to turn it into a more useful developer guide. But there's a catch. Starting in 7.17, our documentation began focussing on the new method of running APM: Elastic Agent + the APM integration. Because of this integration focus, it wouldn't make sense for an APM Server standalone docker dev tutorial to live side-by-side with the other top-level docs in the APM Guide. A few reasons come to mind:

• Settings and configurations are different
• Feature support is different (ex: tail-based sampling is documented but not supported in legacy mode)
• Index lifecycle management is different
• Ingest pipelines are different

Like Silvia mentioned, I think we need a docker dev guide that is maintained by the Elastic Agent team but has instructions for how to enable and configure the APM integration. It might be more complex in the beginning, but it aligns better with where we're heading. (this is all assuming we still plan on deprecating APM Server in the near future)

[danielkhan]

I think this all makes sense. Ideally, this could also be an exercise to make the overall Agent/Fleet setup easier. I know that there is some ongoing work. We might have to connect the strings here though. FYI @mukeshelastic

[simitt]

A few thoughts here. First — and Silvia, correct me if this has changed — APM Server standalone will soon be deprecated in favor of Elastic Agent + the APM integration.

That's valid. Do we know who drives this decision and makes the call @simitt?

The decision will be driven by @axw and me in collaboration with APM PMs. We have not yet deprecated APM Server standalone, due to feature gaps and limited support of APIs (Fleet API is not GA yet) and docker setup, etc. We are going to hold back with deprecating standalone until we have a good answer for our users for all the currently supported setup that should still be supported.

@ruflin @ph can you please chime in here, how users are supposed to set up an automated dockerized dev environment with configured integrations, without any manual interaction. There was also a related request on discuss forum recently.

In case there is no good answer yet to this, @bmorelli25 and @danielkhan how about we update the currently existing docker-compose example the Brandon was referencing (we should do that anyways) and in the managed apm guide link to this docker-compose example with a note that an easy to use setup with Elastic Agent is in the works?

[danielkhan]

In case there is no good answer yet to this, @bmorelli25 and @danielkhan how about we update the currently existing docker-compose example the Brandon was referencing (we should do that anyways) and in the managed apm guide link to this docker-compose example with a note that an easy to use setup with Elastic Agent is in the works?

I like this pragmatic solution, @simitt.

[ruflin]

It seems there are 2 discussions here:

• Document the docker(-compose) setup
• Document an easy way for a dev setup

For the second one, could we use elastic-package for this purpose?

[axw]

For the second one, could we use elastic-package for this purpose?

I don't think so. Users want to be able to easily run the Observability solution alongside their application, so I don't think it's practical to push elastic-package.

We need to find a way to better fit into existing, popular, developer tools like docker-compose. Otherwise I expect we'll see more of this: https://twitter.com/theshalvah/status/1501602601208619013