Disable automatic creation of observation on some scheduled tasks

[nvervelle]

In our application, we have several scheduled tasks annotated with @Scheduled. We have different requirements regarding observability (tracing) depending on the scheduled task :

1. For most of them, we're happy that an observation is created automatically by spring boot each time the scheduled task is triggered.
2. For some of them, we do not want any observation, not even observations created by other methods called by the scheduled task.
3. For some of them, we first check if they are on pause or not (pause is triggered by an external event) and we only want an observation (and sub observations) if it's not on pause

Behavior 1 is the default behavior in spring boot, it works well !

For behavior 2, we implemented an ObservationPredicate that filters out observation generated by scheduled tasks we don't want to observe, which prevents generating the automatic observation for the scheduled task, and eventually sub-observations.

@Component
public class MyScheduledTaskObservationPredicate implements ObservationPredicate {

  @Override
  public boolean test(final String name, final Observation.Context context) {
    if (context instanceof ScheduledTaskObservationContext scheduledContext) {
      return scheduledContext.getTargetClass() != MyScheduledTask.class;
    }
    return true;
  }
}

For behavior 3, I didn't find a clean way to implement it. I tried a few things : they work, but not very clean.

First, I disabled the auto-configuration that makes observations created automatically, but then I have to manually observe each scheduled task for which I want behavior 1.

Then, I tried the following :

• Define an annotation DisableAutomaticObservation
• Implement my own class extending ScheduledAnnotationBeanPostProcessor and override createRunnable() :

  protected Runnable createRunnable(Object target, Method method, @Nullable String qualifier) {
      Runnable runnable = createRunnable(target, method);
      if (runnable != null) {
          return runnable;
      }
      Assert.isTrue(method.getParameterCount() == 0, "Only no-arg methods may be annotated with @Scheduled");
      Method invocableMethod = AopUtils.selectInvocableMethod(method, target.getClass());
              Supplier<ObservationRegistry> supplierObservationRegistry =
                      method.getAnnotation(DisableAutomaticObservation.class) != null ?
                              () -> ObservationRegistry.NOOP :
                              this.registrar::getObservationRegistry
      return new ScheduledMethodRunnable(target, invocableMethod, qualifier, supplierObservationRegistry);
  }

  or

  protected Runnable createRunnable(Object target, Method method, @Nullable String qualifier) {
      Runnable runnable = super.createRunnable(target, method, qualifier);
              if (method.getAnnotation(DisableAutomaticObservation.class) != null &&
                  runnable instanceof ScheduledMethodRunnable smr) {
                      return new ScheduledMethodRunnable(smr.getTarget(), smr.getMethod(), smr.getQualifier(), () -> ObservationRegistry.NOOP);
              }
  }

• Exclude SchedulingConfiguration and replace it with an implementation returning my implementation of ScheduledAnnotationBeanPostProcessor
• Manually observe the scheduled task when I want

This implementation works but it means tweaking spring boot configuration and requires copy/pasting spring boot code to properly override method.

Do you see a clean way to implement behavior 3 ? Otherwise, would it be possible to include such capability in spring boot (for example, with an annotation like DisableAutomaticObservation, it would be a simple modification of ScheduledAnnotationBeanPostProcessor)

[bclozel]

I'm not sure I understand the use case here.

3. For some of them, we first check if they are on pause or not (pause is triggered by an external event) and we only want an observation (and sub observations) if it's not on pause

How would you detect that a scheduled method would be on pause? What do you mean by that? Would it require the execution to be ongoing?

Then, I tried the following

All the solutions you've described could be implemented with an ObservationPredicate that looks at the observation context and checks the ScheduledTaskObservationContext#getMethod() for custom annotations. I don't think you would need to override any Spring Boot configuration nor implement your own ScheduledAnnotationBeanPostProcessor.

I'm probably missing something here.

[nvervelle]

Hi @bclozel

How would you detect that a scheduled method would be on pause? What do you mean by that? Would it require the execution to be ongoing?

I have some scheduled tasks that are triggered frequently, but can decide by themselves to be on pause until a given time depending on some conditions. For example, one scheduled task is used to purge data older than 30 days. Each time it's triggered, it retrieves the oldest data and purge it if it's older than 30 days. If it's not, then it decides to be on pause until this oldest data is 30 days old. That's just an example.

I'm probably missing something here.

Well, the problem is with children spans : if the ObservationPredicate says false, a NOOP observation is created and becomes the current observation. If the code called by the scheduled task creates also an observation, as its parent span is a NOOP, it will also be a NOOP. That's why it works for behavior 2, the scheduled task and children spans are NOOP. For behavior 3, I want children spans to be actual spans. With the ObservationPredicate, I can't have behavior 2 and behavior 3 depending on the use case.

[bclozel]

I see, thanks for the feedback.

So it sounds like you would like to change the active/noop status of an observation after it's been created.

Unfortunately, I don't think there is anything we can do in the Spring Framework @Scheduled instrumentation or in the Spring Boot auto-configuration. Please see this Micrometer issue (and especially the related ones) as I think this is really a Micrometer concern.

[bclozel]

Some additional thoughts.

The Observation API is one of the strengths of Micrometer. One can instrument apps and libraries for metrics, traces or both without introducing specific dependencies. Micrometer uses observation handlers that listen for the lifecycle of observations and turn them into metrics and traces. In this case, starting an observation has side-effects like starting to record a metric, adding headers to messages for tracing, and more. "Cancelling" an observation after it has started is challenging and would probably require specific checks that would hurt performance for instrumentations.

In your case, I understand that you probably don't want to record metrics for "paused jobs" as they're very short lived and are probably skewing numbers. For this, I think you could maybe let the observation be recorded, but add metadata to the observation context stating that it's been paused. This way, you can build your metrics dashboard using this dimension and avoid conflating paused/unpaused jobs.