film42 / sidekiq-rs

A port of sidekiq to rust using tokio
MIT License
95 stars 10 forks source link

Generates ERRO Error leaked out the bottom: TimedOut when using demo code #6

Closed justmark closed 1 year ago

justmark commented 1 year ago

Hi,

I've used some Ruby code to inject some jobs into our Sidekiq implementation. I can pull these jobs through Ruby without issue.

I built a version of your demo that uses my Worker, but when I run things I just get a timeout error. I've checked the manager configuration to confirm the IP and auth password are correct. I get the following:

Jul 12 19:43:29.492 ERRO Error leaked out the bottom: TimedOut Jul 12 19:43:29.493 ERRO Error leaked out the bottom: TimedOut Jul 12 19:43:29.493 ERRO Error leaked out the bottom: TimedOut Jul 12 19:43:29.493 ERRO Error leaked out the bottom: TimedOut Jul 12 19:43:34.494 ERRO Error in scheduled poller routine: TimedOut Jul 12 19:43:59.493 ERRO Error leaked out the bottom: TimedOut Jul 12 19:43:59.493 ERRO Error leaked out the bottom: TimedOut Jul 12 19:43:59.494 ERRO Error leaked out the bottom: TimedOut Jul 12 19:43:59.494 ERRO Error in periodic job poller routine: TimedOut Jul 12 19:43:59.494 ERRO Error leaked out the bottom: TimedOut

Any thoughts on what this error is referring to?

Thanks.

film42 commented 1 year ago

Hey @justmark . I just ran into this while running WSL2 with ubuntu. I have a PR open (#5 ) that switches BRPOP to a 2 second timeout. I'm cleaning up that test suite and then I'll merge it.

film42 commented 1 year ago

I just released #5 as version 0.5.0 . Would you mind seeing if that resolved the issue?

justmark commented 1 year ago

Hi,

Ok, I updated to 0.5.0. The log spits out the following:

Jul 13 11:28:38.515 DEBG Enqueueing job, queue: queue:default, class: Sample::CrystalTestWorker Jul 13 11:28:38.516 DEBG Enqueueing job, queue: queue:default, class: Sample::CrystalTestWorker Jul 13 11:28:38.517 DEBG Enqueueing job, queue: queue:default, class: Sample::MyWorker Jul 13 11:28:38.517 DEBG Enqueueing job, queue: queue:default, class: CrystalTestWorker

I didn't define the queue:default, or the workers listed above.

I only registered a single queue:

let mut p = Processor::new(
        redis.clone(),
        logger.clone(),
        vec!["v1_statuses".to_string()],
    );
justmark commented 1 year ago

I think I have an idea as to my issue - how do I set the namespace? I looked through the docs and don't see any mention of setting the namespace.

film42 commented 1 year ago

The processor will correctly listen on the v1_statuses queue, but when you enqueue you need to be sure to specify the queue or define the default options. Example:

https://github.com/film42/sidekiq-rs/blob/8b2057b7fd79e6a61741187dd03639bde4ae2bfc/examples/demo.rs#L155-L163

Or override default options:

#[derive(Clone)]
struct HelloWorker;

#[async_trait]
impl Worker<()> for HelloWorker {
    async fn perform(&self, _args: ()) -> Result<(), Box<dyn std::error::Error>> {
        // I don't use any args. I do my own work.
        Ok(())
    }

    fn opts() -> WorkerOpts<(), Self>
    where
        Self: Sized,
    {
        WorkerOpts::new().queue("v1_statuses")
    }
}

So that HelloWork::perform_async(...) will always enqueue to the statuses_v1 queue by default.

film42 commented 1 year ago

I will open a second issue for namespaces. It's something that I would like to add support for but it's not clear if that should be owned by this lib or owned by the redis connection library. In ruby redis-namespace money patches its way into the redis. But in the rust world, it might make more sense for this lib to do it.

film42 commented 1 year ago

Opened a feature request for namespaces here https://github.com/film42/sidekiq-rs/issues/7

justmark commented 1 year ago

Hi,

My data is generated through Ruby, entirely. All my workers are currently in Ruby, and Elixir. I'm hoping to replace my Ruby workers with Rust (and your library), but I'm still not able to get it to work. I am explicitly defining the queue. I'm thinking that the issue is the namespace entirely, as I can consume fine when I define the namespace on Ruby (or Elixir).

film42 commented 1 year ago

Namespace will certainly be an issue if you're using that. One debug tool that's helpful is redis-cli monitor which will show all the commands streaming into redis. If some are using namesapces and others are not, then namespaces are the problem for sure. Which elixir lib are you using?

justmark commented 1 year ago

Yes, everything is namespaced. On Elixir we're using Exq.