UI bug: Timeline process time indicator

jkppr commented 1 year ago

When ingesting a large timeline (>10M events) the processing can take quite some time. When we recently ingested a 52M event timeline the processing took around 10 hrs with 1600 events/s ingestions speed. With large timelines like this, there is a bug where approaching the end the status will show values >100% and/or jump randomly between 90-110% values. Because of this bug, an analyst cannot tell if everything is still going according to plan with the processing of the timeline.

Since we know the total number of events before starting the processing and we also know the average ingestion speed after a few minutes, it would be great if the process dialog could show some estimated import time for the timeline.

Example: With 52M events and an average import speed of 1500 events/s we can estimate an import time of ~9,6 hrs. If this calculation can be updated every 15 minutes this would improve the UX for the analysts.

berggren commented 1 year ago

We already do this calculation:

    secondsToComplete() {
      return this.totalEventsToIndex / this.avarageEventsPerSecond()

https://github.com/google/timesketch/blob/cb0eb03f5b9523288e05ede7388b131d83289804/timesketch/frontend-ng/src/components/Explore/TimelineChip.vue#L396C1-L397C69

There is some other bug that causes this. Needs more investigation. I'll change this to a "bug" instead of a "FR" so we can work on it.

jkppr commented 1 year ago

Some example screenshot of where this happened to me:

google / timesketch

UI bug: Timeline process time indicator #2899