search

samsondav / rihanna

Rihanna is a high performance postgres-backed job queue for Elixir
MIT License
439 stars 47 forks source link

Rihanna Jobs sporadically not appearing in logs #97

Closed LannyBose closed 4 years ago

LannyBose commented 4 years ago

Hello!

On my local machine, I'm running into issues where Rihanna jobs are sporadically not appearing in logs but are being executed.

The flow is this:

External Twilio API to Phoenix Endpoint ⬇️ Phoenix Controller enqueues a Rihanna job ⬇️ Rihanna job performs DB work, sends a notification email, and updates GraphQL subscriptions

As I'm running my localhost Phoenix server, I'm (??) seemingly-randomly getting two different outputs in my logs:

Sometimes I see [related DB logging omitted]...

[debug] Processing with TwilioController.message_create/2
  Pipelines: [:logging, :api]
[info] Received POST /twilio/message
[info] Sent 201 response in 2.00ms
[debug] [Rihanna] Starting job 4179
[debug] Processed begin in 0ms
[debug] [Rihanna] Finished job 4179

...and other times I see...

[debug] Processing with TwilioController.message_create/2
  Pipelines: [:logging, :api]
[info] Received POST /twilio/message
[info] Sent 201 response in 2.26ms

...with no corresponding Rihanna logging. However, I know the job is executing because the emails are being delivered as expected and the database records are being updated as expected by the job.

The punchline for me is that in the no-Rihanna-logging scenarios, my GraphQL subscriptions aren't being pushed through, which is a big bad in the case of my app.

This is where my lack of OTP knowledge is going to lead me to ask an ignorantly-phrased question: Where are the un-logged job executions happening in these seemingly-random instances? How do I... get them to execute in the same place?

(Gosh this feels like an inarticulate question. And for that, I'm sorry)

LannyBose commented 4 years ago

Wow. Ok, solved.

I had IEX open in another terminal tab and the workers there were racing to pick up background jobs separate from my running Phoenix server.

User error. Closing. 😀

tpitale commented 4 years ago

That's really interesting. Thanks for sharing. I didn't know that could happen. 🤔