ThreadsOnFork not taking any job with a thread_number of 1

[Haerezis]

Hello,

I'm trying to run my worker in ThreadsOnFork mode, with a number of thread of 1 for now (will increase in the future), with multiple queue, but only the first of the list is running it seems.

Here's the configuration I'm using :

Backburner.configure do |config|
  config.beanstalk_url       = ENV['BEANSTALK_URL']
  config.tube_namespace      = Rails.env
  config.max_job_retries     = 0 # default 0 retries
  config.retry_delay         = 5 # default 5 seconds
  config.respond_timeout     = 120
  config.default_worker      = Backburner::Workers::ThreadsOnFork
  config.logger              = Logger.new(STDOUT)
  config.primary_queue       = "jobs"
  config.reserve_timeout     = nil
  config.default_priority    = 65536
  config.priority_labels     = {
    immediate: 128,
    urgent:    512,
    high:      4096,
    normal:    65536,
    low:       131072,
    optional:  524288,
  }
end

and I'm running the worker like this (simplified a bit) : Backburner.work ["queue1" "queue2"]. All the queues start empty.

I've tried to get a full picture of what is happening, and this is what I can tell you :

• the fork for queue1 seems to finish "starting", because it ends up blocking on a "reserve" command in beaneater (Beaneater::Connection#transmit, on the connection.write), I'm guessing that it's waiting for a job.
• the fork for queue2 blocks on the same line, on the connection.write of Beaneater::Connection#transmit BUT not for a reserve, but "list-tubes-watched" command triggered by Backburner::Worker#all_existing_queues called in Backburner::ThreadsOnFork#prepare. To be clear, it is NOT a problem with the mutex, because both forks enter the synchronize block. The problem seems to be with the TCPSocket.

The fork for queue2 always run AFTER the fork for queue1 is blocking on the "reserve" command, at least according to the various puts I've added.

The queue2 fork blocking on the connection.write is normal (because it is waiting for a job), but not queue2 for the "list-tubes-watched" command. When the thread number is 1, Backburner::ThreadsOnFork don't use new_connection for each fork, thus all the fork use technically the same Backburner::Connection, thus the same Beaneater::Connection, and thus the same TCPSocket used by Beaneater::Connection. From my understanding, tcp socket are backed by a file descriptor in Linux, and forking a process also copy the file descriptors of the process, making the parent and the child using the same socket/file descriptor. Is it possible that because the queue1 fork put the socket in a waiting state, the queue2 fork can't send/receive, because it's using the same socket ?

I don't know, I'm just trying to find a "possible" explanation for this bug, but maybe you will have another idea/solution?

By the way if I'm forcing a self.connection = new_connection after the fork, the problem disappears, and both queue ends up blocking on a "reserve" command.

Thanks a lot in advance for any answer !

[contentfree]

Interesting find. Does it also go away if you use a thread_number larger than 1?

[Haerezis]

Yes, I've tried Backburner.work ["queue1" "queue2:2"], Backburner.work ["queue1:2" "queue2"] and Backburner.work ["queue1:2" "queue2:2"], and all of them lead to all the fork to finish, and block on the "reserve" command.

I think that for N queues, if N-1 have more than 1 threads, then they have all have a new connection (because of the new_connection used), and the last (if there is one) would use the original connection created in the constructor, but would be the only one using the TCPSocket associated with it (the other having new ones).