rubycdp / ferrum

Headless Chrome Ruby API
https://ferrum.rubycdp.com
MIT License
1.68k stars 119 forks source link

thread.rb: terminated with exception #461

Open georgils opened 1 month ago

georgils commented 1 month ago

**Describe the bug** My script crashes after a Ferrum thread throws an exception. First I get this: #<Thread:0x0000027cd26b3900 C:/Users/User/.gem/ruby/3.2.0/gems/ferrum-0.15/lib/ferrum/utils/thread.rb:9 run> terminated with exception (report_on_exception is true): C:/Users/User/.gem/ruby/3.2.0/gems/ferrum-0.15/lib/ferrum/network.rb:376:inblock in subscribe_request_will_be_sent': undefined method `response=' for nil:NilClass (NoMethodError)

      previous_exchange.response = response
                       ^^^^^^^^^^^
    from C:/Users/User/.gem/ruby/3.2.0/gems/ferrum-0.15/lib/ferrum/client/subscriber.rb:71:in `block in call'
    from C:/Users/User/.gem/ruby/3.2.0/gems/ferrum-0.15/lib/ferrum/client/subscriber.rb:69:in `each'
    from C:/Users/User/.gem/ruby/3.2.0/gems/ferrum-0.15/lib/ferrum/client/subscriber.rb:69:in `each_with_index'
    from C:/Users/User/.gem/ruby/3.2.0/gems/ferrum-0.15/lib/ferrum/client/subscriber.rb:69:in `call'
    from C:/Users/User/.gem/ruby/3.2.0/gems/ferrum-0.15/lib/ferrum/client/subscriber.rb:50:in `block (2 levels) in start'
    from C:/Users/User/.gem/ruby/3.2.0/gems/ferrum-0.15/lib/ferrum/client/subscriber.rb:46:in `loop'
    from C:/Users/User/.gem/ruby/3.2.0/gems/ferrum-0.15/lib/ferrum/client/subscriber.rb:46:in `block in start'
    from C:/Users/User/.gem/ruby/3.2.0/gems/ferrum-0.15/lib/ferrum/utils/thread.rb:13:in `block in spawn'`

followed by

undefined methodinclude?' for nil:NilClass (NoMethodError)

                            if traffic.url.include?(CGI.escape(x))
                                          ^^^^^^^^^

from my code where traffic is from page.network.traffic.each { |traffic| .....}

Obviously my traffic.url is nil but somehow I don't believe this should be possible, I guess Chrome would not even create a request if the url is missing.

To Reproduce Sorry but can't imagine how to provide a meaningful context... I'll try to describe: 1. sending Chrome to a url. 2. filling a form which triggers a xhr 3. analyzing network traffic to obtain the specific xhr in question by filtering the array of Exchanges page.network.traffic on their url property.

Expected behavior

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

Additional context I don't know if this has something to do with the above (another nil object), but look at what I get in irb:

irb(main):062:0> browser = Ferrum::Browser.new(url: 'https://github.com/rubycdp/ferrum#frames') C:/Users/User/.gem/ruby/3.2.0/gems/ferrum-0.15/lib/ferrum/browser/process.rb:177:inws_url=': undefined method `host' for nil:NilClass (NoMethodError)

    @host = @ws_url.host
                   ^^^^^
    from C:/Users/User/.gem/ruby/3.2.0/gems/ferrum-0.15/lib/ferrum/browser/process.rb:73:in `initialize'
    from C:/Users/User/.gem/ruby/3.2.0/gems/ferrum-0.15/lib/ferrum/browser.rb:244:in `new'
    from C:/Users/User/.gem/ruby/3.2.0/gems/ferrum-0.15/lib/ferrum/browser.rb:244:in `start'
    from C:/Users/User/.gem/ruby/3.2.0/gems/ferrum-0.15/lib/ferrum/browser.rb:132:in `initialize'
    from (irb):62:in `new'
    from (irb):62:in `<main>'
    from C:/Ruby32-x64/lib/ruby/gems/3.2.0/gems/irb-1.6.2/exe/irb:11:in `<top (required)>'
    from C:/Ruby32-x64/bin/irb:33:in `load'
    from C:/Ruby32-x64/bin/irb:33:in `<main>'`
georgils commented 1 month ago

I took a look at the network.rb source code and did some debugging which led me to the conclusion that the problem is me periodically calling page.network.clear(:traffic). I was doing it instinctively as if cleaning traffic in Developer Tools - I believe what DevTools/Network shows is just a log and deleting it won't affect the browser's behaviour. But maybe this is not the case with Ferrum - I seem to be deleting an Exchange object and that's why select(request.id)[-2] returns nil...