search

WIPACrepo / wipac-telemetry

WIPAC Telemetry: Monitoring/Tracing Applications, Supporting Infrastructures, and Services
MIT License
0 stars 0 forks source link

Configure Service Name #36

Closed ric-evans closed 3 years ago

ric-evans commented 3 years ago

Programmatically assign a service name to the trace.

A service name will consist of the common name and a reference name (a path to the executing code). This will make finding the traced service (in the Jaegar GUI) quick and also be able to distinguish past/future/other executions of the same script/module.

There are two distinct detection/naming schemes (for python):

  1. Modules
  2. Scripts

Exmaple: Modules

Entry Path: /home/hank/repos/my_repo_100/mymodule/__main__.py ~Service Name: "mymodule (/home/hank/repos/my_repo_100)"~ Service Name: "mymodule (v1.2.3)" new revision

Exmaple: Scripts

Script Path: /home/hank/repos/my_repo_200/myscript.py Service Name: "myscript.py (/home/hank/repos/my_repo_200)"

closes https://github.com/WIPACrepo/wipac-telemetry-prototype/issues/35

ric-evans commented 3 years ago

Screenshot from 2021-10-22 12-47-14

ric-evans commented 3 years ago

Open Question: Do we want to include pid and datetime to further distinguish traces? Including pid would stop traces from being merged if a script is run concurrently in parallel. Similarly, including datetime would separate otherwise merged traces on successive runs; that would be important if we wanted to independently look at a previous version of the File Catalog for example.

The downside is that the service name string would become larger.

dsschult commented 3 years ago

Some comments:

  1. module name: wouldn't it be better to use __name__, since that would also capture the full python name if we were in a submodule?
  2. file path: do we really want to record your system's file path? is that even relevant?

I'd recommend not using pid, since some services run multiple processes and it would be good to put those traces together.

One useful thing might be the version, unless that's recorded in the traces somewhere.

I'm also not sure it would be good to have lots of different services all named roughly the same thing, since too many makes it a headache to find the right service. But, this is probably something best discovered through trial and error while using the tool.

ric-evans commented 3 years ago

@dsschult:

  1. module name: wouldn't it be better to use __name__, since that would also capture the full python name if we were in a submodule?

__name__ will be wipac_telemetry.tracing_tools since it's a module/library import.

  1. file path: do we really want to record your system's file path? is that even relevant?

The idea here is that we want to grab as much identifying info as possible. For instance, if the script's name is server.py, that's too vague. For things like scripts, I say the whole path is useful. BUT for modules, you have a good point, the fully qualified module name could suffice.

I'd recommend not using pid, since some services run multiple processes and it would be good to put those traces together.

Sure.

One useful thing might be the version, unless that's recorded in the traces somewhere.

Good idea!

I'm also not sure it would be good to have lots of different services all named roughly the same thing, since too many makes it a headache to find the right service. But, this is probably something best discovered through trial and error while using the tool.

Yes, the goal here is to not have lots of services with the same name. So let's revisit this in the next commits.

ric-evans commented 3 years ago

image

New Revision: include version info with the module, exclude file path

I'm not sure we'll ever actually see a fully qualified module/sub-module in production (mypackage.mymodule), but supporting it is free.

ric-evans commented 3 years ago

@blinkdog what do you think about merging/separating services? Is a traced service an individual invocation of a module/script, or something else?

blinkdog commented 3 years ago

@blinkdog what do you think about merging/separating services? Is a traced service an individual invocation of a module/script, or something else?

The Jaeger UI has Service and Operation at the top.

To me, it seems Service would be very course-grained ("File Catalog", "LTA", "MOU Dashboard", etc.) and Operation gets into the specifics.

Here I think adding version numbers can help to distinguish between old vs current vs bleeding-edge versions. The bleeding-edge version might be helpful when checking on new deployments. Over time, I think it'll lead to those combo-box nightmares, like a web page where every year of birth from 1900 to 2050 is crammed into one selector. But this may encourage us to be a little more proactive about deleting old traces. :smile:

I don't think I'd go with the full path. Service seems like something to keep short and sweet, especially as the very specific stuff (i.e.: the filename) falls off the right side of the combo-box display. We'll have unique strings, but over time we may find most of them start with the same prefix, making the UI more difficult to use.

I'd suggest maybe even an environment variable so the service could name itself (or override the default name). For example:

export WIPAC_TELEMETRY_SERVICE_NAME="MOU-DASH-PR-203"

I could see that being useful for development and production in some cases.

ric-evans commented 3 years ago

@blinkdog: For environment variables, I only see this being used in practice by modules. In which case, we already have a good auto-naming convention: module (v#.#.#). I like your point on versions leading to overpopulating the list. I vote we punt this and see if it becomes a problem. The dropdown is sorted, so it might not look too crowded--it'll at least be indexable.

But this may encourage us to be a little more proactive about deleting old traces. :smile:

This is especially what I'm thinking for one-off scripts, the traces shouldn't live forever. But I'll get rid of the full path since it's not all that helpful.

I'll put in another commit with some sort-related name finessing and script stuff.

ric-evans commented 3 years ago

If we go with this version and don't like it, we can always change it. :smile_cat:

I'm sure this won't be the last time we see this topic :smiley:

dsschult commented 3 years ago

Re history: I'd encourage only keeping traces for a week or two in production. Helps save space, and keeps searching fast.

ric-evans commented 3 years ago

Screenshot from 2021-10-26 12-55-57

Newest Revision: