Closed bmorelli25 closed 2 months ago
@roshan-elastic is the SME
I'm OOO until 4th July but FYI useful contacts might be:
Engineering @andrewvc @smith @tommyers-elastic
PM @akhileshpok @chrisdistasio
Note : They might not be able to help in my absence but just sharing in case...
The actual recommendation is pretty simple, just adding a field, but I think the complexity here is in terms of how forceful we want to be. My $0.02 is to recommend it as a useful thing, but not be too heavy handed. Ultimately I don't want users to think they're somehow doing something wrong by not defining this one field.
The custom logs onboarding does give you the option of adding a service.name
(which adds an add_fields processor to the standalone agent config, which would probably be the easiest way for a new user who is adding new logs.
I would assume that adding an add_fields processor to either an integration, a standalone agent config, or filebeat config would be the easiest way. We have some docs here that could help, but I think we would want to be specific for this use case, and maybe link out to these docs:
@roshan-elastic @andrewvc please let me know if I'm missing anything here.
Hey @mdbirnstiehl,
Thanks for picking this up. Do you have any idea when this could be available? We were aiming to go live tomorrow but we're running a bit behind (cc @kpatticha).
I would assume that adding an add_fields processor to either an integration, a standalone agent config, or filebeat config would be the easiest way.
I think this broadly makes sense. Here are the main use cases I think we should highlight:
service.name
with a value to all logs via a processorservice.name
(e.g. if someone passes in a service name to myservicename
and they wanted to map that to service.name
)I was thinking the structure of the doc could be something like:
Declaring a service name
Explain they just need to declare a service.name
and there are many ways to do that:
(a) Declaring it in the data itself (with a link to how to declare a field name) (b) Using a processor via integrations, agent config or filebeat (with links to how) (c) Have call out pointing towards more advanced stuff like Logstash/ingest pipelines etc?
Perhaps show a simple end-to-end example?
Already have a service name?
Something to explain how they could map an existing field to service.name
(for customers who already pass in a service name but to a different field name).
Perhaps show a simple end-to-end example?
As this experience will evolving relatively quickly, I'm guessing it might make sense to keep this all text based (i.e. no screenshots of the actual new UI experience).
@roshan-elastic we're targeting end of week for docs completion
FYI @bmorelli25 - I'm hoping we can link to the docs from the UI using a short link (so we can change it remotely):
https://ela.st/new-experience-associate-service-logs
This should allow us to ship the code before the docs are ready. This is normally for stateful but since we now have to account for serverless, if we do this we'll need to make sure the feature is disabled via feature flag by default (so users don't see a UI pointing to the wrong docs).
Just an FYI
(a) Declaring it in the data itself (with a link to how to declare a field name)
@roshan-elastic Here are we talking about declaring the service name field in data that has already been ingested in ES? Do you have a specific process or documentation in mind?
Here are we talking about declaring the service name field in data that has already been ingested in ES? Do you have a specific process or documentation in mind?
I don't think that's what he's talking about. Just adding to a filebeat or agent config for new logs being ingested.
Sorry @mdbirnstiehl - I'm probably just confusing myself here.
Forget (a), just (b) and (c) then:
(b) Using a processor via integrations, agent config or filebeat (with links to how) (c) Have call out pointing towards more advanced stuff like Logstash/ingest pipelines etc?
Just FYI, our UI will be pointing towards somewhere in these docs which would need to explain to users how to 'associate existing service logs'* with a service:
*maybe we should rethink the UI naming here but the idea is to tell them how to take their existing logs and start declaring service.name
against them moving forwards
See 'red' links below
We might just point towards the whole doc you make or maybe there'll be a header or something specific to this but the idea is that it'll tell users how to start declaring service.name
in logs moving forwards (e.g. processors, ingest pipelines, logstash etc).
FYI the other two links will point towards these journeys:
Add APM Agent Collect new service logs
Note
Summary
All details are in https://github.com/elastic/observability-docs/issues/4027 and https://github.com/elastic/observability-docs/issues/4027#issuecomment-2183019089.