Add jitter to website refresh workflows

dust-tt / dust

Amplify your team's potential with customizable and secure AI assistants.

https://dust.tt

MIT License

877 stars 93 forks source link

Closed spolu closed 1 week ago

spolu commented 3 weeks ago

Updates to websites come as spikes which starve the aggregation of new websites

Easy solution is to smooth websites update on 8 attempt (1 per hour) More complex solution is to create a new queue for new websites

We may want to also increase concurrency and maybe decrease the upsert queue concurrency

spolu commented 2 weeks ago

Assigning to @JulesBelveze

spolu commented 2 weeks ago

(can't assign just yet but will)

spolu commented 2 weeks ago

Assigning @JulesBelveze as onboading task

philipperolet commented 2 weeks ago

Moving to team board 'in progress'