[Observability Onboarding] Copy changes to the OTel flow for Kubernetes

gbamparop commented 3 weeks ago

:notebook: Summary

Some changes to the instructions and sample configurations in the onboarding flow for Kubernetes through OTel should be made to cater for the most common use cases.

✔ Acceptance criteria

The second step (Install the OpenTelemetry Operator) displays the following description: "Install the OpenTelemetry Operator using the kube-stack Helm chart and the provided values file. For automatic certificate renewal, we recommend installing the cert-manager, and customize the values.yaml file before the installation as described in our documentation."
- Use ela.st link so we can adjust at any point
The description and the sample configuration for the third step (Instrument your application (optional)) will be updated.
Update the link for manual-instrumentation once this PR is backported to 8.16
- Use ela.st link so we can adjust at any point

❓ Open questions

What should the description and the sample configuration for the thirst step be? cc @mlunadia

Dependencies

8.16 backport of https://github.com/elastic/opentelemetry/pull/34

elasticmachine commented 3 weeks ago

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

elasticmachine commented 3 weeks ago

Pinging @elastic/obs-ux-onboarding-team (Feature: Observability Onboarding)

mlunadia commented 2 weeks ago

Team to keep it consistent with the flow I think a command is much better. We can change the annotation according to the language

The UI can be as follows:

Enable automatic instrumentation for your applications by annotating the resource that manages their pods, such as a deployment, or by annotating the entire namespace.

To annotate a specific deployment kubectl annotate deployment my-deployment -n my-namespace instrumentation.opentelemetry.io/inject-java: "opentelemetry-operator-system/elastic-instrumentation"

To annotate all resources in a namespace kubectl annotate namespace my-namespace instrumentation.opentelemetry.io/inject-java: "opentelemetry-operator-system/elastic-instrumentation"

Restart your deployment kubectl rollout restart deployment my-deployment -n my-namespace

Check annotations have been applied correctly kubectl describe deployment my-deployment -n my-namespace

mlunadia commented 2 weeks ago

@jackshirazi can you pls confirm the above annotations require quotes around "true"

I got the annotations from here

mlunadia commented 2 weeks ago

@gbamparop feel free to adjust the language or example placeholder (my-deployment) to make it consistent with other steps

jackshirazi commented 2 weeks ago

Quotes are included in the reference documentation so we should

mlunadia commented 2 weeks ago

@eedugon can you sync with @jackshirazi, for docs we want an example that shows how to use a command to annotate both ns or deployment as described in the comment above. The UI should be aligned with this.

@gizas will check this once ready for both resources as part of the end to end testing.

eedugon commented 2 weeks ago

Please note that annotations at deployment level will never make auto-instrumentation to work, so avoid kubectl annotate deployment commands.

There's no command to annotate the Pod belonging to a deployment, it has to be done with kubectl edit and then save the deployment. Or if the user has the deployment manifests, just edit and apply.

cc: @mlunadia , @jackshirazi , @gizas

gizas commented 2 weeks ago

I also prefer what @eedugon is describing to edit the deployment. So to offer an example with a deployment annotation.

Moreover, just for reference, see https://www.elastic.co/observability-labs/blog/using-the-otel-operator-for-injecting-java-agents We keep referring to namespace annotation in the link in order to support the zero-config mode. But I guess it complicates more things in our case and we should avoid it

eedugon commented 2 weeks ago

@mlunadia :

@eedugon can you sync with @jackshirazi, for docs we want an example that shows how to use a command to annotate both ns or deployment as described in the comment above. The UI should be aligned with this.

I'm working on those docs (in the opentelemetry repo), and they will include examples. But this issue is not exactly about the docs, it's about the Onboarding UX and what links should be used in the UI to point in the docs, isn't it?

Please clarify, @mlunadia and @gbamparop cc: @jackshirazi

@gbamparop : Are you missing commands for any of the topics of the Onboarding UX?

For cert-manager based installation I'm planning to add a command in the docs. That issue is in a private repo but I'll try to transfer it to elastic/opentelemetry and link it here.
For instrumentation related doc, the PR is the one you have added in the description: https://github.com/elastic/opentelemetry/pull/34

flash1293 commented 2 weeks ago

@mlunadia Based on what @eedugon and @gizas are writing above, how should it look like in the UI? I can't open a PR right now because there are issues with Github, but this is how it looks like (which is wrong based on my understanding of what was written above):

Should we show a code sample how the user needs to edit their manifest.yml for a deployment?

eedugon commented 2 weeks ago

@flash1293 , as mentioned in my previous comment kubectl annotate deployment cannot be used.

Also the initial sentence of the screenshot is still misleading and technically incorrect (cc: @mlunadia ).

This is what I'm adding in the doc PR (from a sentence provided by @mlunadia where I have added a small part):

To enable auto-instrumentation, add the corresponding language annotation to the Pods template (spec.template.metadata.annotations) in your Deployment or relevant workload object (StatefulSet, Job, CronJob, etc.).

^^ That sentence is technically valid.

The reason why the other one is not valid is:

by annotating the resource that manages the Pod ---> This makes the user to think that the Deployment must be annotated, and it's not completely accurate, it's the Pod template defined in the Deployment.

flash1293 commented 2 weeks ago

After checking with @eedugon I changed it to this:

WIll create the PR once Kibana repo is up again.

eedugon commented 2 weeks ago

That looks great but we should probably add also the other parts previously suggested by Miguel

flash1293 commented 2 weeks ago

Combined version:

gizas commented 2 weeks ago

One minor on the final step: #check annotations have been applied correctly

I would advise to check the pod not the deployment

eedugon commented 2 weeks ago

@gizas , @flash1293 : the latest suggestion looks great.

I would advise to check the pod not the deployment

I completely agree with that because of two reasons:

If the user forgets to restart the Pods then they will be mislead if they describe the Deployment showing the right annotations.
If the user sets the annotation wrongly at Deployment level, the describe deployment won't help much, as they will still see the incorrect annotation, but if they describe the Pod they will never see the annotation and that should make them to realize that it was wrongly set.

So +100 to @gizas comment.

And one more thing.... The describe is useful not only to verify the annotation, but also to see if the auto-instrumentation magic has occurred (init-container created, or sidecar container, injected environment variables in the main container, etc.).

What I don't know is how to show it also in the UI.... Maybe...

# Check annotations have been applied correctly and auto-instrumentation library is injected
kubectl describe pod <myapp-pod-name> -n my-namespace

What do you think? :)

We will cover all that and the possible issues in the troubleshooting doc that will be also part of the linked document.

elastic / kibana