Connection error is fatal (while 404 is not)

[mgorny]

It seems that connection errors are treated more fatally than 404 errors:

$ ./bin/pluto b candrews.ini 
activerecord-utils/0.4.0 (activerecord/6.0.2) on Ruby 2.7.0 (2019-12-25) [x86_64-linux]
activityutils/0.1.1 on Ruby 2.7.0 (2019-12-25) [x86_64-linux]
pluto/1.3.2 on Ruby 2.7.0 (2019-12-25) [x86_64-linux]
[info] db settings:
[info] {:adapter=>"sqlite3", :database=>"./candrews.db"}
-- create_table(:logs)
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/sqlite3/schema_statements.rb:91: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_definitions.rb:260: warning: The called method `initialize' is defined here
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:305: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_definitions.rb:201: warning: The called method `primary_key' is defined here
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_definitions.rb:202: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_definitions.rb:363: warning: The called method `column' is defined here
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_definitions.rb:378: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_definitions.rb:429: warning: The called method `new_column_definition' is defined here
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_definitions.rb:229: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_definitions.rb:363: warning: The called method `column' is defined here
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_definitions.rb:229: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_definitions.rb:363: warning: The called method `column' is defined here
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_definitions.rb:411: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_definitions.rb:363: warning: The called method `column' is defined here
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_definitions.rb:412: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_definitions.rb:363: warning: The called method `column' is defined here
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_creation.rb:17: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:1099: warning: The called method `type_to_sql' is defined here
   -> 0.0158s
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/internal_metadata.rb:41: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_definitions.rb:227: warning: The called method `string' is defined here
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/transactions.rb:212: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:274: warning: The called method `transaction' is defined here
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/persistence.rb:503: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/timestamp.rb:127: warning: The called method `create_or_update' is defined here
-- create_table(:props)
   -> 0.0108s
-- create_table(:activities)
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_definitions.rb:229: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_definitions.rb:363: warning: The called method `column' is defined here
   -> 0.0123s
-- create_table(:sites)
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_definitions.rb:229: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_definitions.rb:363: warning: The called method `column' is defined here
   -> 0.0114s
-- create_table(:subscriptions)
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_definitions.rb:424: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_definitions.rb:120: warning: The called method `initialize' is defined here
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_definitions.rb:142: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_definitions.rb:363: warning: The called method `column' is defined here
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:786: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:1172: warning: The called method `add_index_options' is defined here
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:1199: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:1262: warning: The called method `quoted_columns_for_index' is defined here
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:1266: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:1254: warning: The called method `add_options_for_index_columns' is defined here
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:1256: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:1237: warning: The called method `add_index_sort_order' is defined here
   -> 0.0376s
-- create_table(:feeds)
   -> 0.0135s
-- create_table(:items)
   -> 0.0231s
/home/mgorny/.gem/ruby/2.7.0/gems/activemodel-6.0.2.1/lib/active_model/type/integer.rb:13: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/home/mgorny/.gem/ruby/2.7.0/gems/activemodel-6.0.2.1/lib/active_model/type/value.rb:8: warning: The called method `initialize' is defined here
[info] Updating feed subscription >candrews< - >https://candrews.integralblue.com/tag/gentoo/feed/atom<...
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/relation/delegation.rb:115: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/home/mgorny/.gem/ruby/2.7.0/gems/activerecord-6.0.2.1/lib/active_record/relation.rb:27: warning: The called method `initialize' is defined here
[info] found cache entry for >https://candrews.integralblue.com/tag/gentoo/feed/atom<

*** error: Failed to open TCP connection to candrews.integralblue.com:443 (Connection refused - connect(2) for "candrews.integralblue.com" port 443)

(it stops processing larger file here, while 404s are ignored)

The .ini file:

title = Planet Gentoo (test)

[candrews]
title = Craig Andrews
link = https://candrews.integralblue.com/
feed = https://candrews.integralblue.com/tag/gentoo/feed/atom

[geraldb]

Thanks for reporting. The feed fetcher code is in https://github.com/feedreader/pluto/blob/master/pluto-feedfetcher/lib/pluto/feedfetcher/cond_get_with_cache.rb The code should catch SocketErrors for unknown domains. Maybe your case is somewhat different? I will double-check later this week. Cheers. Prost.

PS: Update: Just tried to access https://candrews.integralblue.com/tag/gentoo/feed/atom it works with a "vanilla" browser - I guess the problem is actually in HTTPS, that is, somehow the secure connection fails (in ruby).

[mgorny]

Actually, I think I just happened to hit a short downtime ;-). However, it can be easily reproduced by changing the port number:

title = Planet Gentoo (test)

[candrews]
title = Craig Andrews
link = https://candrews.integralblue.com/
feed = https://candrews.integralblue.com:81/tag/gentoo/feed/atom

FWICS the exception class is Errno::ECONNREFUSED. I'm sorry but my Ruby knowledge is too poor to know if there's a superclass that catches it all.

[geraldb]

Thanks for the detailed ruby exception Errno::ECONNREFUSED. I currently have no direct access with ruby to the internet (the wonders of a secure enterprise setup) but I will try it later today in my coffee break. No worries. Should be an easy fix.

[mgorny]

Actually, I'll probably have a pull request by then ;-).

[geraldb]

Thanks for the fix! I will keep the issue / ticket open until a new pluto-feedfetcher gem is published / pushed out. Thanks.

[geraldb]

Update: I tried the original https port and in debug mode I get this error:

[debug] GET /tag/gentoo/feed/atom uri=https://candrews.integralblue.com/tag/gentoo/feed/atom, redirect_limit=5
[info] found cache entry for >https://candrews.integralblue.com/tag/gentoo/feed/atom<

*** error: SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: unknown protocol

net/http.rb:933:in `connect_nonblock': SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: unknown protocol (OpenSSL::SSL::SSLError)

Thus, it might actually be a HTTPS error (and not a cannot connect).

[mgorny]

Well, in my case it was error: Failed to open TCP connection to candrews.integralblue.com:443 (Connection refused - connect(2) for "candrews.integralblue.com" port 443) but in general yes, I suppose there might be more problematic cases.

[geraldb]

Thanks for the update - I will push out new gems with your connect rescue fix and the new empty feed case tomorrow (Thursday) and than close this issue here. If you still run into the SSL error (1) with the update, please open a new issue. Cheers. (1) - maybe I'm using an too old ruby version e.g. 2.3.3 with too old openssl lib

[geraldb]

I pushed / published the new pluto gems. In the latest pluto version this reports / logs now an error something like:

[error] *** error: fetching feed 'candrews' - [Errno::ECONNREFUSED] 
               Failed to open TCP connection to candrews.integralblue.com:81 
               (No connection could be made because the target machine actively refused it. -
               connect(2) for "candrews.integralblue.com" port 81)