chore: make local setup idempotent

[Spazzy757]

Changelog

• upgrades cockroachdb to v21.1.9
• upgrades kafka for integration tests to 3.4 (chart 22.0.1)
• removes gbv- prefix where possible for integration tests in favour of fly- prefix
• cleans up dev setup to be idempotent and not have crashloopback pods
• moves cockroachdb for integration tests and local dev to be a service
• moves kafka for integration tests and local dev to be a service
• updates sql scripts to be compatible with cocroachdb v21.1.9

Notes

• this is just a bit of a cleanup to make it slightly smoother to setup dev env and integration tests, if you dont like it let me know what I should change
• the cockroachdb changes I ran the testrunner and all the tests passed, im assuming that means the changes are valid

[netlify[bot]]

✅ Deploy Preview for vlab-research canceled.

Name	Link
🔨 Latest commit	fc2a8b957d0a0974e4fd4b1b1d23b0da666aa366
🔍 Latest deploy log	https://app.netlify.com/sites/vlab-research/deploys/648ad0648fd207000833ee3e

[netlify[bot]]

✅ Deploy Preview for virtuallab-videos canceled.

Name	Link
🔨 Latest commit	fc2a8b957d0a0974e4fd4b1b1d23b0da666aa366
🔍 Latest deploy log	https://app.netlify.com/sites/virtuallab-videos/deploys/648ad0644d5ef900087ca00d

[nandanrao]

I like the cleanup a lot!

However, I'm a bit confused about the file structure for /values. There is values/integrations/fly and values/integrations/kafka. But then there is still values/staging.yaml and values/test.yaml etc...

That seems a bit confusing to me, why not have a consistent structure?

[nandanrao]

Hm. Also, trying to run make dev it on a new machine, it fails because some of the apps are missing secrets? Such as "bot-envs"?

[Spazzy757]

Hm. Also, trying to run make dev it on a new machine, it fails because some of the apps are missing secrets? Such as "bot-envs"?

Yes, this is due to this line: https://github.com/vlab-research/fly/pull/124/files#diff-7a622d6368356a8d43646cae2132855752983918b924ca7326be4fd128c37570R33-R35

This is legacy as you need the env file in the dev folder (Its not saved in the repo and I haven't changed this)

[Spazzy757]

I like the cleanup a lot!

However, I'm a bit confused about the file structure for /values. There is values/integrations/fly and values/integrations/kafka. But then there is still values/staging.yaml and values/test.yaml etc...

That seems a bit confusing to me, why not have a consistent structure?

This was used to separate the "services" from "fly". Currenly staging cockroachdb is deployed as part of the "chart of charts" so there is not need to seperate it until we are ready top remove this dependency