Question - stress testing and workers

[sheldondz]

Hi,

We are writing some stress tests to batch writes to an external data source, we have a worker that processes a job after x seconds and batches the data.

We plan to spawn a large number jobs to do this, is there a way to wait for all workers to be done and track for failures.

We currently using copied from the river tests itself, is this the best way to test this?

subscribeChan, subscribeCancel := client.Subscribe(river.EventKindJobCompleted, river.EventKindJobFailed)
defer subscribeCancel()
events := waitForNJobs(subscribeChan, N)

[brandur]

@sheldondz Yeah, subscriptions are one way to do this. However, note that subscription events are only emitted for jobs worked within the specific client instance that's subscribed to. i.e. If you have two separate nodes both running their own River client, each one must be subscribed to separately, and assuming each client is working roughly equal jobs, each subscription will only receive half the total jobs going through the queue.

The other strategy to consider would just be a classic poll loop. I know it sounds icky, but as long as you pick a reasonable period duration (not too fast to hammer the database, but not too slow as to be unresponsive), it'd probably work fine in practice.

[sheldondz]

@brandur thanks for the suggestions.

So for this case will it be possible for me to have 2 queues, with 2 clients?

Client 1 processes jobs using LISTEN/NOTIFY Client 2 processes jobs using polling. The queue for this client will only create jobs for the above scenario.