search

nycmeshnet / meshdb

A convenient, stable, and sane database for tracking Members and Nodes for use with robots and humans
https://db.mesh.nycmesh.net
MIT License
8 stars 13 forks source link

Break celery worker and beat into their own pod #465

Closed WillNilges closed 1 month ago

WillNilges commented 1 month ago

Moves celery out of the main meshdb pod and into its own. The beat and worker each get their own container within that pod, and everyone communicates through Redis.

Inspired by: https://medium.com/ambient-innovation/health-checks-for-celery-in-kubernetes-cf3274a3e106

codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 85.71429% with 4 lines in your changes missing coverage. Please review.

Project coverage is 95.02%. Comparing base (4e72638) to head (6277ec8).

Files Patch % Lines
src/meshdb/celery.py 85.71% 4 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #465 +/- ## ========================================== - Coverage 95.07% 95.02% -0.05% ========================================== Files 110 110 Lines 5380 5407 +27 ========================================== + Hits 5115 5138 +23 - Misses 265 269 +4 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

WillNilges commented 1 month ago

https://medium.com/ambient-innovation/health-checks-for-celery-in-kubernetes-cf3274a3e106

Andrew-Dickinson commented 1 month ago

Happy to override codecov here. Testing celery statically is hard. With Datadog will we be able to monitor to ensure the tasks are running and alert if they stop? (and can we document that requirement, in #88 perhaps)

WillNilges commented 1 month ago

Can we move these celery related scripts into sub-directories? It's a lot of clutter for the root of the repo

Perhaps:

  • bin/

    • celery/

    • All the celery stuff

    • All the other scripts we're collecting

@Andrew-Dickinson I agree. I was going to make a scripts/ directory in a follow-up PR, but I'm happy to do it in this one.

WillNilges commented 1 month ago

Happy to override codecov here. Testing celery statically is hard. With Datadog will we be able to monitor to ensure the tasks are running and alert if they stop? (and can we document that requirement, in #88 perhaps)

Certainly. Feel free to add that to the reqs.

WillNilges commented 1 month ago

~maybe we can get the redefinitions to stop if we break out the checks into other files, like in the blog post I attached in the description.~

Update: Tried it, no dice.