instructlab / instructlab-bot

GitHub bot to assist with the taxonomy contribution workflow
Apache License 2.0
16 stars 17 forks source link

Add redis pooling and worker refactor #177

Closed nerdalert closed 6 months ago

nerdalert commented 6 months ago
nerdalert commented 6 months ago

@russellb Validated a few concurrent jobs over two PRs (the error intended on job 10 in the screencap). The worker redis client no longer bombs when running more than one job at once.

image
russellb commented 6 months ago

The worker redis client no longer bombs when running more than one job at once.

Each worker should only be running one job at a time, not doing them in parallel. Or do you just mean more than one job in the queue at the same time?

nerdalert commented 6 months ago

@russellb if you queued up two jobs concurrently, the redis connection bombed because the connection is closed by the previous job so it had a conn that was closed it was using to try and query redis. My guess is somewhere in the go routines it was getting triggered even though it wasn't running the job. Jobs are still sequential. You can see in the following a two minute gap between results. A screenshot was cleaner then logs since we can't use pub gists 😬

image
nerdalert commented 6 months ago

Also validated across two workers with two jobs. They each ran one of the jobs. 🎉

image