Closed yuliamitttovahuspy closed 1 year ago
+1 The same issue is affecting me as well.
💯 Experiencing same thing with coroutines.
The same problem happens when CoroutineCrudRepository is used. For example, if the following code was used:
@Repository
interface SampleITemCoroutineCrudRepository : CoroutineCrudRepository<SampleItemEntity, UUID>
I found the solution how to fix spans, but it is not a good one: 1) Switch off annotations (it looks like they are not tracking things properly) and move to manual approach. For it, add new dependency:
implementation("io.opentelemetry:opentelemetry-extension-kotlin:1.27.0")
And env params listed below:
OTEL_INSTRUMENTATION_GRAPHQL_JAVA_ENABLED=false;
OTEL_INSTRUMENTATION_OPENTELEMETRY_API_ENABLED=true;
OTEL_INSTRUMENTATION_OPENTELEMETRY_INSTRUMENTATION_ANNOTATIONS_ENABLED=false;
OTEL_INSTRUMENTATION_SPRING_WEBFLUX_ENABLED=false;
suspend fun <Result> withFixedSpan(
name: String,
parameters: (SpanBuilder.() -> Unit)? = null,
parentSpan: Span? = null,
block: suspend (span: Span?) -> Result,
): Result {
val tracer = GlobalOpenTelemetry.getTracer("com.example", "0.0.0")
val parentSpanToSet = parentSpan ?: Span.current()
val childSpan: Span = tracer.spanBuilder(name).run {
if (parameters != null)
parameters()
coroutineContext[CoroutineName]?.let {
setAttribute("coroutine.name", it.name)
}
setParent(Context.current().with(parentSpanToSet))
startSpan()
}
return withContext(parentSpanToSet.asContextElement()) {
try {
childSpan.makeCurrent().use { scope -> block(childSpan) }
} catch (throwable: Throwable) {
childSpan.setStatus(StatusCode.ERROR)
childSpan.recordException(throwable)
throw throwable
} finally {
childSpan.end()
}
}
}
return withFixedSpan(
name = "SampleEntityRepositoryImpl.findAllBy",
) {
r2dbcEntityTemplate.select(SampleEntity::class.java)
.matching(filter.toQuery(sorting))
.flow().toList()
}
There are (at least) three places where things could be going wrong, only one of which would apply to this repositories codebase: 1) Our co-routine context propagation code has a bug in it [that lives here: https://github.com/open-telemetry/opentelemetry-java/tree/main/extensions/kotlin) 2) There are issues with the OTel-based instrumentation 3) There are issues with the Spring code that makes things break.
It would be helpful if you could narrow things down a bit on this, since only option 1) would apply to this codebase.
Hello @jkwatson I have created a sample project. Please find it here: https://github.com/YuliaMittova/opentelemetry_issue_5543 I think the problem is in how OpenTelemetry is processing @WithSpan spans when Flow emits more than one item with delay (which is a typical situation when working with a database). If you fetch a specific student (Flow will emit only one item), you will see the following picture:
But if you have Coroutine Flow which emits more than one item, you will see spans which don't have correct parents and are measured incorrectly:
That sounds like it's an instrumentation issue, then, and not an issue for the core libraries. Can you open an issue in the instrumentation repository for this?
@jkwatson thank you for a reply. https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues - is this a correct place to create an issue?
Closing now that issue has been moved over to opentelemetry-java-instrumentation
Description:
We have a project which uses GraphQL, Spring, Kotlin coroutines. For DB interactions org.springframework.data.r2dbc.core.R2dbcEntityTemplate is used. It can return or coroutine Flow or Spring Flux/Mono. If r2dbcEntityTemplate is used, the trace looks like this:
If we return List it will be different :
Steps to reproduce Gradle dependencies:
Configuration:
We will create a simple chain: SampleItemFetcher(GraphQL) -> SampleItemService(Spring) -> EntityRepository(Spring) -> EntityRepository (uses R2dbcEntityTemplate) Code is described below:
schema.graphql:
SampleItemFetcher:
SampleItemService:
SampleItemServiceImpl:
EntityRepository:
EntityRepositoryImpl:
DBRepository:
DBRepositoryImpl:
What did you expect to see? Spans should contain the correct ones.
What did you see instead? In broken trace RepoImpl.getAll is shown separately, but it should be a part of the parent span.
What version and what artefacts are you using?
Environment macOS, tested locally
Additional context Add any other context about the problem here.