rowanmanning
commented
2 months ago Hiya, status update on where I've got to so far. This is hard! I'm encountering two issues:
HTTP vs Undici
I set up a test app locally which makes requests via both Undici and node-fetch – these are the two most common ways we make client requests from our apps. I can get the Undici request to log but I don't seem to be able to hook into the HTTP (node-fetch) request at all. How is this working?? @sjparkinson this is making me question everything - are we definitely getting metrics from node-fetch? 😂
Here's what I've been trying (I've also tried every other option under @opentelemetry/instrumentation-http):
getNodeAutoInstrumentations({
'@opentelemetry/instrumentation-http': {
requestHook: (span, request) => {
console.log('HTTP REQUEST HOOK', request.url);
}
},
'@opentelemetry/instrumentation-undici': {
requestHook: (span, request) => {
console.log('UNDICI REQUEST HOOK', request.path);
}
}
// ...
})When I make the following requests:
const nFetch = require('node-fetch');
const bulbasaur = await fetch('https://pokeapi.co/api/v2/pokemon/bulbasaur'); // native fetch
const squirtle = await nFetch('https://pokeapi.co/api/v2/pokemon/squirtle'); // node-fetchI get only this log:
UNDICI REQUEST HOOK /api/v2/pokemon/bulbasaurI might be missing something, maybe there's a different instrumentation library that handles HTTP client requests? Seems unlikely though.
Custom attributes on existing metrics
Turns out that the instrumentations for Node.js are quite strict. Ignoring the HTTP issue I decided to try and add custom metrics to native fetches. Attributes would get added to the http.client.request.duration metric.
If I try adding a custom attribute that isn't explicitly supported or even if I add the one experimental attribute (url.template) then I see nothing. So adding the following:
getNodeAutoInstrumentations({
// ...
'@opentelemetry/instrumentation-undici': {
startSpanHook(request) {
return {
['url.template']: '/hello-world'
};
}
}
// ...
})The metric that gets exported is like this:
{
// ...
attributes: [
{
'http.response.status_code': 200,
'http.request.method': 'GET',
'server.address': 'pokeapi.co',
'server.port': 443,
'url.scheme': 'https'
}
]
}However if I reuse one of the metrics that's already exported then it works:
getNodeAutoInstrumentations({
// ...
'@opentelemetry/instrumentation-undici': {
startSpanHook(request) {
return {
['server.address']: 'why.wont.you.work'
};
}
}
// ...
})This exports the metric:
{
// ...
attributes: [
{
'http.response.status_code': 200,
'http.request.method': 'GET',
'server.address': 'why.wont.you.work',
'server.port': 443,
'url.scheme': 'https'
}
]
}Very frustrating. If this is a hard limitation at the moment then I'm not sure what we can do to get paths into our metrics.
sjparkinson
jonnangle
We'd like to have added a hook in the Opentelemetry plugin that adds specificity to the
api.ft.comnet_peer_nameattribute. So we can select the metrics specific to an API that lives inside the API GatewayWhat problem does this feature solve?
Due to the ongoing Graphite migration, we need to replace some of the system's Graphite based alerts with Opentelemetry.
Some of these alerts do need to trigger when a specific service is not working as it should. For example, let's see this alert in
next-myft-apiGraphite health check
The metrics inside
next.heroku.myft-api.*.fetch.insights-api-topic-recommendationscorrespond to the requests done to the service in this URLhttps://api.ft.com/snr/v1/insights/topic-recommendations/, which lives inside of the API Gateway. But when creating the Opentelemetry Grafana panel for this alert, we find thatnet_peer_namedoesn't allow us to selecttopic-recommendations, onlyapi.ft.combecause this label is domain based.Link to Grafana panel
This means that if
next-myft-apiuses many services that are inside API Gateway (which it does), all the metrics are bundled together and we can't tell where the 5xx or 4xx are coming fromIdeal solution
As suggested by Sam Parkinson here, we could add a hook in the Opentelemetry plugin to add the first part of the path to the
net_peer_nameattribute. In this case it would beapi.ft.com/snr, which works for us becausenext-myft-apidoesn't use any other API fromsnr. But maybe we could consider adding the last part of the path, eg :api.ft.com/topic-recommendationsAnyways, this is a problem shared in many repositories (and probably teams), and it'd be amazing to have this solved in the Opentelemetry plugin
Alternatives
An alternative is to code this OTel hook in every system that needs this granularity.