Weird issue when kicking off multiple workers

[jwoertink]

I have this worker that kicks off 4 other workers when it runs. I noticed that when it would kick off, it was like it was picking and choosing which of those 4 it wanted to start running. I decided to move all 4 of those up a level, and still the same issue.

Basically the flow is

• User logs in
• If successful, kick off 4 separate workers
• Redirect user to member's area.

Tailing the logs, I see that there's always at least 1 that runs, but it's literally any combination of the 4.

It basically looks like this:

if user
  PasswordWorker.new(user_id: user.id).enqueue
  EmailWorker.new(user_id: user.id).enqueue
  LoginWorker.new(user_id: user.id).enqueue
  PostbackWorker.new(user_id: user.id).enqueue
  redirect to: "/path"
end

Please include some details:

Crystal version: 0.31.1 Mosquito Shard version: commit: 0212a74fdb87b32c983c573fa93e853bc0fa036f

[robacarp]

@jwoertink In the example you gave, are you saying that it only runs the PasswordWorker, but not EmailWorker, LoginWorker, etc?

[jwoertink]

In that example, I'm saying that it may run all 4, or it may run just the LoginWorker, or it may run only the PasswordWorker and the PostbackWorker.... Like every permutation of those. It doesn't really make sense, but my guess is all of them get shoved in to redis, but only some get pulled back out.

[robacarp]

@jwoertink wow, that's significant. Thanks for reporting. I have a feeling it's going to be difficult to reproduce.

[jwoertink]

I'm sure. We just launched this in to production yesterday, so we're just now seeing it. I'm gonna do more digging in, but you know how it goes with the lack of debugging tools in crystal 😅 I'll let you know if I come up with any additional info.

[robacarp]

@jwoertink can you go into more detail about your deployment? I'm curious how many task runners you have, for example.

[jwoertink]

oh, I didn't even think about that.

require "./shards"
require "./workers/*"

Mosquito::Runner.start

I'm just running

crystal build src/mosquito.cr --release -o worker
./worker

So I never actually specified the number of workers or anything. What does that config look like?

[robacarp]

By default it's just the one worker. I meant spinning up ./worker more than once, possibly on multiple boxes.

[jwoertink]

We only have the one server, so only 1 process for that running.

[robacarp]

@jwoertink you mentioned you might do some digging, did anything get dug up?

[jwoertink]

I did indeed, and I did a horrible job with that 😢 sorry about that. I tried a few different things, but I was never able to recreate it outside of the app. My guess is we must have been doing something weird.

You can close this out for now if you'd like, and if I end up getting a better code example, I can always just reopen it.

[robacarp]

No worries! Thanks for checking back in. Please re-open if needed.