[proposal] Databend dependency management

[PsiACE]

Currently, between 800 and 1000 dependencies need to be compiled per full build/test of Databend.

dependent-bot is not applicable

• update one dependency at a time
• take up a lot of ci resources
• also require developer attention

Dependency Management

1. a batch update of dependencies at the beginning of each month
   • cargo upgrade
   • cargo udeps
   • manually update dependencies that break changes
2. updates to specific dependencies as required
   • cargo audit
   • development needs

As there is currently no suitable tool to update dependencies in batch, we will perform this process manually. (The main problem is the lack of a simple way to roll back specific packages based on reported errors.)

[BohuTANG]

The fisrt task is to remove the dependsbot from databend :)

[sundy-li]

If we update dependencies in batch, how could we know which new dependency is not valid?

[PsiACE]

If we update dependencies in batch, how could we know which new dependency is not valid?

We had to deal with these potential problems manually.

[Xuanwo]

Please take https://docs.renovatebot.com/rust/ into consideration.

renovatebot will update all crates in batch and have more behavior config than dependabot (which means more complex).

Some features I like:

• Dependency Dashboard: https://github.com/renovatebot/renovate/issues/2958
  • We can have a overview about current state of our dependence tree
• Batch update
  • We can update dependences in batch
  • More flexible, we can define groups. For example, we can upgrade tokio-* together in the same PR.
• Pre/Post upgrade jobs
  • We can define pre and post upgrade jobs
  • For example, run cargo udeps after upgrade

Anyway, I think we should not update crates by hand expect there are breaking changes.

[BohuTANG]

We should find/make a action that can do the update in batch. Dependabot has a disscusstion on it: https://github.com/dependabot/dependabot-core/issues/2174