Add execution metadata to scheduled tasks actuator endpoint

[enesify]

Hi,

It would be nice that Spring Boot Actuator exposes endpoints for extra information about scheduled tasks(crons) in addition to runnable, expression, fixedDelay, fixedRate and custom endpoints like these:

• Last Execution Time

• Last Execution Status

• Next Execution Time

• Ability to change expression

• Ability to run a scheduled task manually

I think these endpoints would be helpful for developers and for 3rd party monitoring tools (e.g. Spring Boot Admin)

Thanks.

[wilkinsona]

Thanks for the suggestions.

Last execution time, last execution status, and next execution time are not made available by Spring Framework's task infrastructure. I can see the value in them being available via the endpoint though. @jhoeller what do you think about exposing these in Framework for consumption by the Actuator?

I'm less sure about providing write operations for tasks. It isn't possible to set the expression on a cron task at the moment or to request immediate execution of a scheduled task so our hands are tied on both of those anyway without some changes to Spring Framework.

[scottylaw]

I'm not sure how to solve the issue with checking execution times or changing the cron expression but I also wanted to be able to run my scheduled tasks ad-hoc via the Spring Boot Admin dashboard and came up with the following. There is probably a better way to do this without having to pass in the fully qualified class name but since this is displayed on the Scheduled Tasks tab it has worked for us.

@Component
@EndpointJmxExtension(endpoint = ScheduledTasksEndpoint.class)
public class ScheduledTasksEndpointExtension {

    private final Logger logger = LoggerFactory.getLogger(ScheduledTasksEndpointExtension.class);
    private final AutowireCapableBeanFactory beanFactory;

    @Autowired
    public ScheduledTasksEndpointExtension(AutowireCapableBeanFactory beanFactory) {
        this.beanFactory = beanFactory;
    }

    @WriteOperation
    public ResponseEntity runTask(@Selector String taskName) {
        try {
            String clazzName = taskName.substring(0,taskName.lastIndexOf("."));
            Class<?> clazz = Class.forName(clazzName);
            Object object = clazz.newInstance();
            Method method = object.getClass().getDeclaredMethod(taskName.substring(taskName.lastIndexOf(".") + 1));
            beanFactory.autowireBean(object);
            method.invoke(object);
            return ResponseEntity.ok().build();
        } catch (InstantiationException | ClassNotFoundException | NoSuchMethodException | IllegalAccessException | InvocationTargetException ex) {
            logger.error("Error running {}",taskName,ex);
            return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body(ex.getMessage());
        }
    }

}

[wilkinsona]

Looking more closely, I don't think there's anything in Framework for last execution time, last execution status, and next execution time that could be exposed. If a fixed delay task is current running, the next execution time is unknown so it's not particularly surprising that it doesn't track it.

@enesify Can you please provide a bit more information about what properties of Spring Framework's scheduled tasks you wanted to be exposed?

[spring-projects-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[scottylaw]

@enesify you could expose this data with a custom actuator endpoint like what I showed above if you also set up a custom aspect with an Around and AfterThrowing advice. I'm thinking you would inject this ScheduleAspect bean into your custom actuator endpoint and then you'll need to add the read operations for each piece of data you wish to consume. Below is an example that only handles a single cron expression, logic would need to be added to handle the other types of schedules. Also note that the ScheduledTask object show below is a custom object that I created, it isn't the one from Spring Framework.

@Component
@Aspect
public class ScheduleAspect {

    private static final String ERROR_STATUS = "ERROR";
    private static final Map<String, ScheduledTask> taskMap = new HashMap<>();

    @Around("@annotation(schedule)")
    public Object around(ProceedingJoinPoint proceedingJoinPoint, Scheduled schedule) throws Throwable {
        ScheduledTask scheduledTask;
        String signature = proceedingJoinPoint.getSignature().toLongString();
        if (!taskMap.containsKey(signature)) {
            scheduledTask = new ScheduledTask();
            taskMap.put(signature,scheduledTask);
        } else {
            scheduledTask = taskMap.get(signature);
        }
        scheduledTask.setLastExecutionTime(LocalDateTime.now());
        Object proceed = proceedingJoinPoint.proceed();
        scheduledTask.setNextExecutionTime(getNextExecutionTime(schedule));
        return proceed;
    }

    @AfterThrowing(value = "@annotation(schedule)",throwing = "ex")
    public void afterThrowing(JoinPoint joinPoint, Exception ex, Scheduled schedule) {
        String signature = joinPoint.getSignature().toLongString();
        if (taskMap.containsKey(signature)) {
            ScheduledTask scheduledTask = taskMap.get(signature);
            scheduledTask.setLastExecutionStatus(ERROR_STATUS);
            scheduledTask.setLastExecutionException(ex);
            scheduledTask.setNextExecutionTime(getNextExecutionTime(schedule));
        }
    }

    private LocalDateTime getNextExecutionTime(Scheduled schedule) {
        CronSequenceGenerator cronSequenceGenerator = new CronSequenceGenerator(schedule.cron());
        java.util.Date nextRunDate = cronSequenceGenerator.next(Date.from(Instant.now()));
        return nextRunDate.toInstant().atZone(ZoneId.systemDefault()).toLocalDateTime();
    }
}

[bclozel]

Waiting for spring-projects/spring-framework#24560

[divyathaore]

@snicoll @bclozel : Any update on this enhancement ? I was exploring this similar requirement to monitor the schedulers using the spring boot actuator endpoints.. Can you suggest any way to achieve the same ?

[wilkinsona]

@divyathaore Unfortunately not. As shown above we are blocked on https://github.com/spring-projects/spring-framework/issues/24560.

[fhackenberger]

Here's an updated (Spring 6.x) version of the extension, which additionally checks if there's an instance of that bean already around.

import org.springframework.boot.actuate.endpoint.annotation.WriteOperation;
import org.springframework.boot.actuate.endpoint.web.annotation.EndpointWebExtension;
import org.springframework.boot.actuate.scheduling.ScheduledTasksEndpoint;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Component;

import lombok.extern.log4j.Log4j2;

/** Exposes a way to run a @Scheduled spring task through spring actuator
 * E.g.
 * <pre>curl -v -u 'httpUser' -i -X POST -H 'Content-Type: application/json' http://localhost:8080/REPLACEME/actuator/scheduledtasks/org.myscheduledbeans.ScheduledBean.springTimeout</pre>
 */
@Component
@EndpointWebExtension(endpoint = ScheduledTasksEndpoint.class)
@Log4j2
public class ScheduledTasksActuatorEndpointExtension {
    private final AutowireCapableBeanFactory beanFactory;

    @Autowired
    public ScheduledTasksActuatorEndpointExtension(AutowireCapableBeanFactory beanFactory) {
        this.beanFactory = beanFactory;
    }

    @WriteOperation
    public ResponseEntity runTask(@Selector String taskName) {
        try {
            String clazzName = taskName.substring(0, taskName.lastIndexOf("."));
            Class<?> clazz = Class.forName(clazzName);
            ObjectProvider<?> objProv = beanFactory.getBeanProvider(clazz);
            Object object = objProv.getIfUnique();
            if(object == null) {
                log.info(format("No unique bean found for type {0}, creating a new instance", clazzName));
                Optional<Constructor<?>> defCtor = Arrays.stream(clazz.getConstructors()).filter(c -> c.getGenericParameterTypes().length == 0).findFirst();
                object = defCtor.map(this::newInstance).orElseGet(null);
            }
            if(object == null)
                return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body("Failed to create instance of " + clazzName);
            Method method = object.getClass().getDeclaredMethod(taskName.substring(taskName.lastIndexOf(".") + 1));
            beanFactory.autowireBean(object);
            method.invoke(object);
            return ResponseEntity.ok().build();
        } catch (ClassNotFoundException | NoSuchMethodException | IllegalAccessException | InvocationTargetException | IllegalArgumentException ex) {
            log.error("Error running {}", taskName, ex);
            return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body(ex.getMessage());
        }
    }

    Object newInstance(Constructor<?> cTor) {
        try {
            return cTor.newInstance();
        } catch (InstantiationException | IllegalAccessException | IllegalArgumentException | InvocationTargetException e) {
            log.error(format("Failed to create new instance of {0}", cTor.getDeclaringClass().getName()));
            return null;
        }
    }
}

[ahoehma]

Here's an updated (Spring 6.x) version of the extension, which additionally checks if there's an instance of that bean already around.

Nice solution @fhackenberger.

May this could be improved a little bit by

1. checking that only "@scheduled" tasks can be executed, currently this endpoint looks like a little backdoor to all beans
2. may role based execution only
3. may more properties to include/exclude only specific tasks for this manual triggering

Ciao Andreas

[fhackenberger]

Thanks for the feedback @ahoehma. Here's an improved version that checks if the scheduled task is registered with the parent endpoint and provides include/exclude properties to configure it. I'm not sure what you mean with role based execution? The actuator endpoints can simply be augmented with spring security configuration to get authorisation features.

package org.acoveo.infostars.tapestryweb.spring;

import static java.text.MessageFormat.format;
import static java.util.Arrays.stream;
import static java.util.Optional.ofNullable;
import static java.util.stream.Stream.of;

import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.util.Arrays;
import java.util.HashSet;
import java.util.List;
import java.util.Optional;
import java.util.Set;
import java.util.stream.Stream;

import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.ObjectProvider;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.beans.factory.config.AutowireCapableBeanFactory;
import org.springframework.boot.actuate.endpoint.annotation.Selector;
import org.springframework.boot.actuate.endpoint.annotation.WriteOperation;
import org.springframework.boot.actuate.endpoint.web.annotation.EndpointWebExtension;
import org.springframework.boot.actuate.scheduling.ScheduledTasksEndpoint;
import org.springframework.boot.actuate.scheduling.ScheduledTasksEndpoint.ScheduledTasksDescriptor;
import org.springframework.boot.actuate.scheduling.ScheduledTasksEndpoint.TaskDescriptor;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Component;

import lombok.extern.log4j.Log4j2;

/** Exposes a way to run a @Scheduled spring task through spring actuator
 * E.g.
 * <pre>curl -v -u 'httpUser' -i -X POST -H 'Content-Type: application/json' http://localhost:8080/REPLACEME/actuator/scheduledtasks/org.myscheduledbeans.ScheduledBean.springTimeout</pre>
 * You can use the properties
 * <pre>
 * management.endpoints.scheduledtasks.runtaskextension.exclude=org.example.MyClass.schedule, org.example2.MyClass2.schedule
 * </pre>
 * to exclude specific taskNames or
 * <pre>
 * management.endpoints.scheduledtasks.runtaskextension.includes=org.example.MyClass.schedule, org.example2.MyClass2.schedule
 * </pre>
 * to set a whitelist of task names that are allowed to be run by this extension.
 */
@Component
@EndpointWebExtension(endpoint = ScheduledTasksEndpoint.class)
@Log4j2
public class ScheduledTasksActuatorEndpointRunTaskExtension {
    private final AutowireCapableBeanFactory beanFactory;
    private final ScheduledTasksEndpoint ep;
    private final Set<String> excludes = new HashSet<>();
    private final Set<String> includes = new HashSet<>();

    @Autowired
    public ScheduledTasksActuatorEndpointRunTaskExtension(AutowireCapableBeanFactory beanFactory, ScheduledTasksEndpoint ep,
            @Value("${management.endpoints.scheduledtasks.runtaskextension.exclude:#{null}}") String excludes,
            @Value("${management.endpoints.scheduledtasks.runtaskextension.includes:#{null}}") String includes) {
        this.beanFactory = beanFactory;
        this.ep = ep;
        ofNullable(excludes).map(ex -> stream(ex.split(", ?"))).orElseGet(Stream::empty).forEach(this.excludes::add);
        ofNullable(includes).map(ex -> stream(ex.split(", ?"))).orElseGet(Stream::empty).forEach(this.includes::add);
    }

    @WriteOperation
    public ResponseEntity<String> runTask(@Selector String taskName) {
        try {
            if(StringUtils.isEmpty(taskName))
                return ResponseEntity.status(HttpStatus.BAD_REQUEST).body("You need to provide a taskName");
            if(!includes.isEmpty() && !includes.contains(taskName))
                return ResponseEntity.status(HttpStatus.BAD_REQUEST).body(taskName + " is not on the includes list");
            if(excludes.contains(taskName))
                return ResponseEntity.status(HttpStatus.BAD_REQUEST).body(taskName + " is excluded from being run");
            // Check that the task to run actually belongs to a scheduled bean
            ScheduledTasksDescriptor schedTasks = ep.scheduledTasks();
            Optional<TaskDescriptor> taskDesc = of(schedTasks.getCron(), schedTasks.getFixedDelay(), schedTasks.getFixedRate(), schedTasks.getCustom()).flatMap(List::stream)
                    .filter(td -> taskName.equals(td.getRunnable().getTarget())).findFirst();
            if(!taskDesc.isPresent())
                return ResponseEntity.status(HttpStatus.BAD_REQUEST).body("No scheduled task found: " + taskName);
            // Get or create the scheduled bean to run the task on
            String clazzName = taskName.substring(0, taskName.lastIndexOf("."));
            Class<?> clazz = Class.forName(clazzName);
            ObjectProvider<?> objProv = beanFactory.getBeanProvider(clazz);
            Object object = objProv.getIfUnique();
            if(object == null) {
                log.info(format("No unique bean found for type {0}, creating a new instance", clazzName));
                Optional<Constructor<?>> defCtor = Arrays.stream(clazz.getConstructors()).filter(c -> c.getGenericParameterTypes().length == 0).findFirst();
                object = defCtor.map(this::newInstance).orElseGet(null);
            }
            if(object == null)
                return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body("Failed to create instance of " + clazzName);
            // Run the task
            Method method = object.getClass().getDeclaredMethod(taskName.substring(taskName.lastIndexOf(".") + 1));
            beanFactory.autowireBean(object);
            method.invoke(object);
            return ResponseEntity.ok().build();
        } catch (ClassNotFoundException | NoSuchMethodException | IllegalAccessException | InvocationTargetException | IllegalArgumentException ex) {
            log.error("Error running {}", taskName, ex);
            return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body(ex.getMessage());
        }
    }

    Object newInstance(Constructor<?> cTor) {
        try {
            return cTor.newInstance();
        } catch (InstantiationException | IllegalAccessException | IllegalArgumentException | InvocationTargetException e) {
            log.error(format("Failed to create new instance of {0}", cTor.getDeclaringClass().getName()));
            return null;
        }
    }
}

[ahoehma]

Pretty awesome! 👍 @fhackenberger

[fhackenberger]

@bclozel Would you be able to create a PR out of this?

[bclozel]

@fhackenberger I'm not sure what this code snippet does but I don't think this is related to the current issue. We already have a scheduled tasks endpoint and we'll use the work that's been done in https://github.com/spring-projects/spring-framework/issues/24560 to enrich the metadata.

I'll do that in a future 3.4.x milestone.

[ahoehma]

@fhackenberger your extension code is pretty cool .. but nothing for a framework ... I agree with @bclozel And your extension enabled acutator-based triggering of tasks ... which is very nice .. I would like to see a "springboot admin" integration for that :-)

[bclozel]

To clarify things, the current scope of this issue is described in this comment. We will add more metadata (Last Execution Time, Last Execution Status, Next Execution Time and Last Execution Status) to the existing endpoint. We will not be adding ways to trigger or change scheduled tasks at this time.

[fhackenberger]

Ah, I just thought that adding a way to trigger a task was part of this issue, as it's in the original issue request and also mentioned in #24560. But sure, if it's out of scope then people can simply add it by copying my extension into their codebase.

[bclozel]

New metadata will be shown shortly on the scheduledtasks actuator endpoint docs page.