search

nhl / link-move

A model-driven dynamically-configurable framework to acquire data from external sources and save it to your database.
Apache License 2.0
35 stars 15 forks source link

DeleteTask to load source keys outside the first segment #209

Closed andrus closed 1 year ago

andrus commented 1 year ago

Source keys within a delete task are loaded via a subtasks inside segment:0 scope, which is a bit confusing:

[3/delete] start
[3/delete] segment:0
[3/delete/source-keys] start
[3/delete/source-keys] segment:0
[3/delete/source-keys] segment:0 done in:1 out_keys:1
[3/delete/source-keys] done time:PT0.000582S segments:1 in:1 out_created:0 out_updated:0 out_deleted:0
[3/delete] segment:0 done in:2 out_deleted:1
[3/delete] segment:1
[3/delete] segment:1 done in:2 out_deleted:1
[3/delete] done time:PT0.001768S segments:2 in:2 out_created:0 out_updated:0 out_deleted:1

Let's call the "source-keys" subtask before we start running the delete processor. Which will have essentially the same effect, but with less nesting:

[3/delete] start
[3/delete/source-keys] start
[3/delete/source-keys] segment:0
[3/delete/source-keys] segment:0 done in:1 out_keys:1
[3/delete/source-keys] done time:PT0.001749S segments:0 in:0 out_keys:0
[3/delete] segment:0 start
[3/delete] segment:0 done in:2 out_deleted:2
[3/delete] segment:1
[3/delete] segment:1 done in:2 out_deleted:1
[3/delete] done time:PT0.003732S segments:2 in:2 out_deleted:2

Upgrade Notes

There's no more delete segment step that generates @AfterSourceKeysExtracted event. If you still want to do some processing at the beginning of the delete segment (and when the source keys are already set), listen to @AfterTargetsExtracted instead.