Create basic opentelemetry example

[onx2]

I thought this might be a useful starting point for some people interested in seeing opentelemetry working with async-graphql.

It's a pretty bare bones example with a hello world query and requires the user to have curl to see the output of opentelemetry. I would have preferred to use graphql playground but I don't know how to hide the introspection query in opentelemetry and thought it would be too distracting for a basic example.

Hope this is helpful! If it is I can also make an example for using Jaegar UI.

I can also add an apollo studio example here or in the extension repo but I'm currently waiting for some help to get it working: https://github.com/async-graphql/async_graphql_apollo_studio_extension/issues/113

https://github.com/async-graphql/async-graphql/issues/1031 https://github.com/async-graphql/async-graphql/issues/873

[sunli829]

Thanks!😄

[onx2]

@sunli829 Happy to help 😄

Btw, is there a way to hide introspection queries from the tracing data? If not, is it possible to add the to extension? I wouldn't mind taking a stab at that b/c the tracing data for introspections is very noisy

[xoac]

@onx2 Thanks for example. I was able to connect it with jaeager but it's very noisy. It generates 1,5k+ spans in my case. Is this designed this way and should be used to debug async-graphgql?

Also I wasn't be able to make it works with opentelemetry 0.18 (but it worked with 0.17).

[onx2]

Yes I believe that is the design of tracing to create a span for each part of the query. It seems like you're looking at the introspection query though because of the __schema node. For now I think you can filter out those results in Jaeger UI, but I'd like to eventually open up a PR to add an option to ignore the introspection query.

BTW I recommend disabling introspection in production. See this test for an example of that: https://github.com/async-graphql/async-graphql/blob/036c165644b24b2fc6dd1ae81a96c1799dd7a361/tests/introspection.rs#L1232

As it relates to the version, I wasn't able to get 0.18 to work either. I think there was some API changes but haven't dug too deep into it. It probably will require an update to async_graphql.

[nickb937]

For me, my graphql-service was producing more than 50,000 spans per query of which 49990 of those spans were field spans taking 0.00ms. I ended up forking the Tracing extension to produce a version where the field spans were not included, effectively this meant changing the resolve function to:

    async fn resolve(
        &self,
        ctx: &ExtensionContext<'_>,
        info: ResolveInfo<'_>,
        next: NextResolve<'_>,
    ) -> ServerResult<Option<Value>> {
        let fut = next.run(ctx, info).inspect_err(|err| {
            error!(
                target: "async_graphql::graphql",
                error = tracing::field::display(&err.message),
                "error",
            );
        });
        fut.await
    }

Now I get around 5-10 spans per query and significantly lower processing requirements from the monitoring system.