search

PNixx / clickhouse-activerecord

A Ruby database ActiveRecord driver for ClickHouse
MIT License
198 stars 100 forks source link

Getting rbuf_fill - end of file reached (EOFError) on production but works fine in development #160

Closed BilalBudhani closed 2 months ago

BilalBudhani commented 2 months ago

Here is my setup 


# frozen_string_literal: true

module ClickHouse
  class Record < ActiveRecord::Base
    establish_connection(
      adapter: "clickhouse",
      database: "default",
      username: ENV["CLICKHOUSE_USER"],
      password: ENV["CLICKHOUSE_PASSWORD"],
      ssl: URI(ENV["CLICKHOUSE_URL"]).scheme == "https",
      debug: Rails.env.development?,
      connection: Net::HTTP.start(URI(ENV["CLICKHOUSE_URL"]).host, URI(ENV["CLICKHOUSE_URL"]).port)
    )
    self.abstract_class = true

    # Set the session timezone to the team's time zone if the team is present
    default_scope { settings(session_timezone: ActiveSupport::TimeZone[(Current.team&.time_zone || "UTC")].tzinfo.name) }
  end
end
# frozen_string_literal: true

module ClickHouse
  class Visit < Record
  end
end

with this setup, it works fine in the development 

irb(main):013:0> ClickHouse::Visit.where(campaign_id: 2)
  Clickhouse visits (system) (65.9ms)  DESCRIBE TABLE `visits`
  Clickhouse ClickHouse::Visit Load (18.1ms)  SELECT visits.* FROM visits WHERE visits.campaign_id = 2 SETTINGS session_timezone = 'Etc/UTC'                                                           
=> []

but when I run the same query in production I get 

 production  irb(main):009:0> ClickHouse::Visit.where(campaign_id: 2)
/app/vendor/bundle/ruby/3.2.0/gems/net-protocol-0.2.2/lib/net/protocol.rb:237:in `rbuf_fill': end of file reached (EOFError)
 production  irb(main):010:0> ClickHouse::Visit.where(campaign_id: 2).to_sql
/app/vendor/bundle/ruby/3.2.0/gems/net-protocol-0.2.2/lib/net/protocol.rb:237:in `rbuf_fill': end of file reached (EOFError)
 production  irb(main):011:0>

I don't know where the problem is or if I am missing something in my production setup. I have other models backed by ClickHouse where I'm facing the same issue on production.

BilalBudhani commented 2 months ago

Here is the full stack trace for the issue 

/usr/local/bundle/gems/net-protocol-0.2.2/lib/net/protocol.rb:237:in `rbuf_fill'                              
/usr/local/bundle/gems/net-protocol-0.2.2/lib/net/protocol.rb:199:in `readuntil'                              
/usr/local/bundle/gems/net-protocol-0.2.2/lib/net/protocol.rb:209:in `readline'                               
/usr/local/bundle/gems/net-http-0.4.1/lib/net/http/response.rb:158:in `read_status_line'                      
/usr/local/bundle/gems/net-http-0.4.1/lib/net/http/response.rb:147:in `read_new'
/usr/local/bundle/gems/net-http-0.4.1/lib/net/http.rb:2342:in `block in transport_request'
/usr/local/bundle/gems/net-http-0.4.1/lib/net/http.rb:2333:in `catch'
/usr/local/bundle/gems/net-http-0.4.1/lib/net/http.rb:2333:in `transport_request'
/usr/local/bundle/gems/net-http-0.4.1/lib/net/http.rb:2306:in `request'
/usr/local/bundle/gems/sentry-ruby-5.17.3/lib/sentry/net/http.rb:40:in `block in request'
/usr/local/bundle/gems/sentry-ruby-5.17.3/lib/sentry/hub.rb:102:in `with_child_span'
/usr/local/bundle/gems/sentry-ruby-5.17.3/lib/sentry-ruby.rb:490:in `with_child_span'
/usr/local/bundle/gems/sentry-ruby-5.17.3/lib/sentry/net/http.rb:33:in `request'
/usr/local/bundle/gems/net-http-0.4.1/lib/net/http.rb:2320:in `send_entity'
/usr/local/bundle/gems/net-http-0.4.1/lib/net/http.rb:1973:in `post'
/usr/local/bundle/gems/clickhouse-activerecord-0.6.2/lib/active_record/connection_adapters/clickhouse/schema_statements.rb:63:in `block in do_system_execute'
/usr/local/bundle/gems/clickhouse-activerecord-0.6.2/lib/active_record/connection_adapters/clickhouse/schema_statements.rb:129:in `block in log_with_debug'
/usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract_adapter.rb:752:in `block in log'
/usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract_adapter.rb:743:in `log'
/usr/local/bundle/gems/clickhouse-activerecord-0.6.2/lib/active_record/connection_adapters/clickhouse/schema_statements.rb:129:in `log_with_debug'
/usr/local/bundle/gems/clickhouse-activerecord-0.6.2/lib/active_record/connection_adapters/clickhouse/schema_statements.rb:62:in `do_system_execute'
/usr/local/bundle/gems/clickhouse-activerecord-0.6.2/lib/active_record/connection_adapters/clickhouse/schema_statements.rb:156:in `table_structure'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:117:in `columns'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/schema_cache.rb:117:in `block in columns'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/schema_cache.rb:116:in `fetch'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/schema_cache.rb:116:in `columns'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/schema_cache.rb:125:in `block in columns_hash'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/schema_cache.rb:124:in `fetch'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/schema_cache.rb:124:in `columns_hash'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:580:in `load_schema!'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/attributes.rb:264:in `load_schema!'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/encryption/encryptable_record.rb:122:in `load_schema!'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:566:in `block in load_schema'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:563:in `synchronize'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:563:in `load_schema'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:429:in `attribute_types'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:455:in `type_for_attribute'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/type_caster/map.rb:16:in `type_for_attribute'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/arel/table.rb:107:in `type_for_attribute'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/table_metadata.rb:18:in `type'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/relation/predicate_builder.rb:59:in `build'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/relation/predicate_builder.rb:54:in `[]'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/relation/predicate_builder.rb:126:in `block in expand_from_hash'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/relation/predicate_builder.rb:79:in `each'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/relation/predicate_builder.rb:79:in `flat_map'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/relation/predicate_builder.rb:79:in `expand_from_hash'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/relation/predicate_builder.rb:25:in `build_from_hash'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/relation/query_methods.rb:1333:in `build_where_clause'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/relation/query_methods.rb:774:in `where!'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/relation/query_methods.rb:769:in `where'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/querying.rb:22:in `where'
(irb):9:in `<main>'
/usr/local/lib/ruby/3.2.0/irb/workspace.rb:119:in `eval'
/usr/local/lib/ruby/3.2.0/irb/workspace.rb:119:in `evaluate'
/usr/local/lib/ruby/3.2.0/irb/context.rb:502:in `evaluate'
/usr/local/lib/ruby/3.2.0/irb.rb:588:in `block (2 levels) in eval_input'
/usr/local/lib/ruby/3.2.0/irb.rb:777:in `signal_status'
/usr/local/lib/ruby/3.2.0/irb.rb:567:in `block in eval_input'
/usr/local/lib/ruby/3.2.0/irb/ruby-lex.rb:267:in `block (2 levels) in each_top_level_statement'
/usr/local/lib/ruby/3.2.0/irb/ruby-lex.rb:249:in `loop'
/usr/local/lib/ruby/3.2.0/irb/ruby-lex.rb:249:in `block in each_top_level_statement'
/usr/local/lib/ruby/3.2.0/irb/ruby-lex.rb:248:in `catch'
/usr/local/lib/ruby/3.2.0/irb/ruby-lex.rb:248:in `each_top_level_statement'
/usr/local/lib/ruby/3.2.0/irb.rb:566:in `eval_input'
/usr/local/lib/ruby/3.2.0/irb.rb:500:in `block in run'
/usr/local/lib/ruby/3.2.0/irb.rb:499:in `catch'
/usr/local/lib/ruby/3.2.0/irb.rb:499:in `run'
/usr/local/lib/ruby/3.2.0/irb.rb:421:in `start'
/usr/local/bundle/gems/railties-7.0.8.1/lib/rails/commands/console/console_command.rb:74:in `start'
/usr/local/bundle/gems/railties-7.0.8.1/lib/rails/commands/console/console_command.rb:19:in `start'
/usr/local/bundle/gems/railties-7.0.8.1/lib/rails/commands/console/console_command.rb:106:in `perform'
/usr/local/bundle/gems/thor-1.3.1/lib/thor/command.rb:28:in `run'
/usr/local/bundle/gems/thor-1.3.1/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/bundle/gems/thor-1.3.1/lib/thor.rb:527:in `dispatch'
/usr/local/bundle/gems/railties-7.0.8.1/lib/rails/command/base.rb:87:in `perform'
/usr/local/bundle/gems/railties-7.0.8.1/lib/rails/command.rb:48:in `invoke'
/usr/local/bundle/gems/railties-7.0.8.1/lib/rails/commands.rb:18:in `<main>'
<internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
<internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
/usr/local/bundle/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
bin/rails:4:in `<main>'
PNixx commented 2 months ago

Downgrade ruby version or upgrade rails version to 7.1+. Versions below 7.1 are no longer supported.

BilalBudhani commented 2 months ago

Thank you for your response. I'm just wondering what could be the reason behind the error in production and not in development.

woodhull commented 1 month ago

I am also experiencing this same issue on rails 7.1