nhl / link-move

A model-driven dynamically-configurable framework to acquire data from external sources and save it to your database.
Apache License 2.0
35 stars 15 forks source link

execution stats may mismatch real changes caused by custom stage listener code #195

Closed vitalz closed 1 year ago

vitalz commented 2 years ago

Execution stats may mismatch real changes caused by custom stage listener code.

For example, there is dry delete stage listener code:

public class DryDeleteStageListener<T extends BaseDataObject> {
    private static final Logger log = LoggerFactory.getLogger(DryDeleteStageListener.class);
    private final Consumer<T> consumer;

    public DryDeleteStageListener(Consumer<T> consumer) {
        this.consumer = consumer;
    }

    @AfterMissingTargetsFiltered
    public void afterMissingTargetsFiltered(Execution exec, DeleteSegment<T> segment) {
        segment.getMissingTargets().forEach(rowProxy -> consumer.accept((T) rowProxy.get(DeleteSegment.TARGET_COLUMN)));
        log.debug("Setting an empty list for the missing targets.");
        segment.setMissingTargets(DataFrameBuilder.builder().empty());
    }

}

And a LinkMove task like:

        LmTask inactivateTask = lmService.getTaskService()
                .delete(Record.class)
                .sourceMatchExtractor("record.xml")
                .matchById()
                .stageListener(new DryDeleteStageListener<Subject>(r -> r.setActive(false)))
                .batchSize(10000)
                .task();

Then its execution report will look like:

Deleted=N

Though really there are no records have been deleted.

For example, the cause of the issue for delete task stats is current implementation in com.nhl.link.move.runtime.task.delete.DeleteStatsListener for @AfterMissingTargetsFiltered event. It increments delete stats value as:

e.getStats().incrementDeleted(segment.getMissingTargets().height());
andrus commented 1 year ago

Fixed in 3.0.M1 as we switched to collecting statistics using @AfterTargetsCommitted event