search

seomoz / qless

Queue / Pipeline Management
MIT License
294 stars 76 forks source link

Can't delete/timeout jobs via Web UI #281

Closed stephenreay closed 5 years ago

stephenreay commented 5 years ago

When trying to use the 'delete', 'timeout', 'untrack' or 'move' buttons in the Web UI for a job, the HTTP request it makes returns a HTTP 500 error, and both the html response, and the console for the server show the error

Note that 'track' seems to work as expected.

TypeError - can't dup NilClass

The stack trace for the error is below, as is the content of the 'boot' script we're using (called from systemd) in place of the shipping qless-web.rb.

Error output:

TypeError - can't dup NilClass:
/var/lib/gems/2.3.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1092:in `dup'
/var/lib/gems/2.3.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1092:in `block in dispatch!'
/var/lib/gems/2.3.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1092:in `each'
/var/lib/gems/2.3.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1092:in `dispatch!'
/var/lib/gems/2.3.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:924:in `block in call!'
/var/lib/gems/2.3.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1076:in `block in invoke'
/var/lib/gems/2.3.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1076:in `catch'
/var/lib/gems/2.3.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1076:in `invoke'
/var/lib/gems/2.3.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:924:in `call!'
/var/lib/gems/2.3.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:913:in `call'
/var/lib/gems/2.3.0/gems/rack-protection-2.0.5/lib/rack/protection/xss_header.rb:18:in `call'
/var/lib/gems/2.3.0/gems/rack-protection-2.0.5/lib/rack/protection/path_traversal.rb:16:in `call'
/var/lib/gems/2.3.0/gems/rack-protection-2.0.5/lib/rack/protection/json_csrf.rb:26:in `call'
/var/lib/gems/2.3.0/gems/rack-protection-2.0.5/lib/rack/protection/base.rb:50:in `call'
/var/lib/gems/2.3.0/gems/rack-protection-2.0.5/lib/rack/protection/base.rb:50:in `call'
/var/lib/gems/2.3.0/gems/rack-protection-2.0.5/lib/rack/protection/frame_options.rb:31:in `call'
/var/lib/gems/2.3.0/gems/rack-2.0.7/lib/rack/null_logger.rb:9:in `call'
/var/lib/gems/2.3.0/gems/rack-2.0.7/lib/rack/head.rb:12:in `call'
/var/lib/gems/2.3.0/gems/sinatra-2.0.5/lib/sinatra/show_exceptions.rb:22:in `call'
/var/lib/gems/2.3.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:194:in `call'
/var/lib/gems/2.3.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1957:in `call'
/var/lib/gems/2.3.0/gems/thin-1.7.2/lib/thin/connection.rb:86:in `block in pre_process'
/var/lib/gems/2.3.0/gems/thin-1.7.2/lib/thin/connection.rb:84:in `catch'
/var/lib/gems/2.3.0/gems/thin-1.7.2/lib/thin/connection.rb:84:in `pre_process'
/var/lib/gems/2.3.0/gems/thin-1.7.2/lib/thin/connection.rb:53:in `process'
/var/lib/gems/2.3.0/gems/thin-1.7.2/lib/thin/connection.rb:39:in `receive_data'
/var/lib/gems/2.3.0/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in `run_machine'
/var/lib/gems/2.3.0/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in `run'
/var/lib/gems/2.3.0/gems/thin-1.7.2/lib/thin/backends/base.rb:73:in `start'
/var/lib/gems/2.3.0/gems/thin-1.7.2/lib/thin/server.rb:162:in `start'
/var/lib/gems/2.3.0/gems/rack-2.0.7/lib/rack/handler/thin.rb:22:in `run'
/var/lib/gems/2.3.0/gems/vegas-0.1.11/lib/vegas/runner.rb:171:in `run!'
/var/lib/gems/2.3.0/gems/vegas-0.1.11/lib/vegas/runner.rb:113:in `start'
/var/lib/gems/2.3.0/gems/vegas-0.1.11/lib/vegas/runner.rb:77:in `initialize'
/usr/local/lib/bundlesites/qless-web.rb:15:in `new'
/usr/local/lib/bundlesites/qless-web.rb:15:in `<main>'

Boot script:

require 'vegas'
require 'qless'
require 'qless/server'

client = Qless::Client.new(
    :scheme => ((ENV['REDIS_SSL'] == 'true') ? 'rediss' : 'redis'),
    :host => ENV['REDIS_HOST'],
    :port  => ENV['REDIS_PORT'],
    :db => ENV['REDIS_DB'],
    :ssl_params => ((ENV['REDIS_SSL'] == 'true') ? {:ca_path => '/etc/ssl/certs/'} : nil)
)

puts client.inspect

Vegas::Runner.new(Qless::Server.new(client), "qless-web-#{ENV['APP_ENV']}")

Ruby isn't my bread and butter, we're using a PHP port of Qless within our application, but the Web UI (barring this issue) gives us a simple way to view/control the queues.

dlecocq commented 5 years ago

Interesting. Usually in the various language bindings the qless-core repo is included as a submodule. If the same with PHP bindings, what SHA is it at? Alternatively, link to the PHP bindings?

stephenreay commented 5 years ago

Hey @dlecocq

The PHP binding is (https://github.com/pdffiller/qless-php) which doesn't use a submodule, but we're currently using v3.8.1 which includes this version of the Qless Lua: https://github.com/pdffiller/qless-php/blob/v3.8.1/src/qless-core/qless.lua. Just to clarify: the php binding is working fine, both to queue jobs, pull them and then either complete/cancel etc.

Edit: sorry, forgot the URL. Edit2: clarify more specifically the version of the binding (and thus Lua) we're using

dlecocq commented 5 years ago

Thanks! Even though the PHP bindings are working fine, something that can cause problems is having too-far-mismatched SHAs for qless-core between the language bindings and the qless UI. However, that doesn't appear to be the case.

Is that error trace what shows up in the UI? Or is that the contents of the response of the 500-ing API call? What's particularly confusing is that the stack trace doesn't even reference qless after the essentially the launch of the server.

stephenreay commented 5 years ago

So I think a version of that stack trace may have been included in the XHR response to the call from clicking the buttons, but I actually got it from the console output of - so either direct output if it's run directly, or e.g. using journalctl when it runs under systemd.

From some other searching this exact error (and often with traces that reference Sinatra) come up all over the place, GH, SO, etc. It seems unlikely it's related to Qless directly, so I guess I'll keep searching for the solution.

dlecocq commented 5 years ago

Yeah, though it seemed odd that some buttons worked and some didn't. My first inclination when seeing an error with dup in a server was that it was related to dup(2), but apparently dup is a different thing in Ruby.

stephenreay commented 5 years ago

At a guess it seems like it's related to https://github.com/sinatra/sinatra/pull/1517, and because I was assigning the :ssl_params key to a hash when required, and nil when not.

stephenreay commented 5 years ago

Ok, followup: its not related to our custom boot script, it affects the default qless-web script too

stephenreay commented 5 years ago

Ok, yup, I 'solved' the issue by installing Ruby 2.6 from the Brightbox repo. This definitely seems to be caused by https://github.com/sinatra/sinatra/issues/1514, but despite merging the fix 5 months ago, they haven't made a release since then, so not much that can be done from your point of view, but I guess at least now this issue will show up if anyone else gets the same behaviour.

I'm gonna close this now, because I don't see any practical steps you can take to resolve it anyway.