elastic / elasticsearch

Free and Open, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
68.7k stars 24.39k forks source link

[Transform] Updating frequency should apply after next checkpoint #86898

Open sophiec20 opened 2 years ago

sophiec20 commented 2 years ago

Found in 8.2.0-SNAPSHOT

Documentation says All updated properties except description do not take effect until after the transform starts the next checkpoint. This is so there is data consistency in each checkpoint.

However in practice, updating frequency does not "seem" to apply until the transform is stopped and started.

image

To reproduce:

  1. Created and started a continuous transform on a dataset (timestamps ranging between now-10d to now+10d), with frequency: 30s
  2. Closed the source index POST source-index/_close
  3. Observed timestamps of warning notifications, 30s apart
  4. Opened source index POST source-index/_open
  5. Updated transform to set frequency: 10s
  6. Inserted current timestamp docs into source index (to force the checkpoint)
  7. Observed notification message confirming checkpoint occurred
  8. Closed the source index
  9. Observed timestamps of warning notifications -- 30s apart, instead of expected 10s apart
  10. Open the source index
  11. Stop and restart the transform
  12. Close the source index
  13. Observed timestamps of warning notifications -- now at 10s apart

Despite updating frequency: 10s beforehand, the notification messages still show that the action is occurring every 30s after a checkpoint. After stopping and restarting the transform, then the notification messages occur every 10s.

Either 1/ the docs require a clarification, or 2/ the test isn't valid, or 3/ there is a bug in the way frequency is being applied

elasticmachine commented 2 years ago

Pinging @elastic/ml-core (Team:ML)