search

sourcegraph / sourcegraph-public-snapshot

Code AI platform with Code Search & Cody
https://sourcegraph.com
Other
10.12k stars 1.29k forks source link

insights: improve queue priority system #39900

Open coury-clark opened 2 years ago

coury-clark commented 2 years ago

Currently the Code Insights queue system works based on time. Data points for recent time periods will populate before older time periods, without regard for the series they come from. This makes all insights backfill backwards through time at the same rate. This was a reasonable solution in the past, but the product has outgrown it in the following ways:

  1. Any error causes every insight to stop processing for as long as one series is erroring
  2. New series that are created cause other insights to stop backfilling until that series catches up

In the future it will probably be useful for series to queue up discretely, and follow some simple rules:

  1. Deprioritize a series if it is constantly erroring
  2. Smaller series take priority over global series
  3. Snapshots take priority over backfills
sourcegraph-bot-2 commented 2 years ago

Heads up @joelkw @felixfbecker @vovakulikov @unclejustin - the "team/code-insights" label was applied to this issue.

chwarwick commented 2 years ago

Avenue to explore possibility of having multiple queues, ie a priority queue