search

3scale / apisonator

Red Hat 3scale API Management Apisonator backend
https://3scale.net
Apache License 2.0
35 stars 27 forks source link

Redshift import crashes when there is no data for a specific hour #196

Closed davidor closed 1 year ago

davidor commented 4 years ago

In S3, events exported via Kinesis are grouped by hour and it looks like the RedshiftAdapter crashes when it cannot find a directory (in this case '2017/02/18/11').

Here's an old stacktrace that shows the problem:

irb(main):006:0> ThreeScale::Backend::Stats::RedshiftAdapter.insert_pending_events
Loading events generated in hour: 2017-02-18 11:00:00 UTC
PG::InternalError: ERROR:  The specified S3 prefix '2017/02/18/11' does not exist
DETAIL:  
  -----------------------------------------------
  error:  The specified S3 prefix '2017/02/18/11' does not exist
  code:      8001
  context:   
  query:     634029
  location:  s3_utility.cpp:568
  process:   padbmaster [pid=29509]
  -----------------------------------------------

        from /var/lib/gems/2.2.0/gems/3scale_backend-2.69.0/lib/3scale/backend/stats/redshift_adapter.rb:304:in `exec'
        from /var/lib/gems/2.2.0/gems/3scale_backend-2.69.0/lib/3scale/backend/stats/redshift_adapter.rb:304:in `execute_command'
        from /var/lib/gems/2.2.0/gems/3scale_backend-2.69.0/lib/3scale/backend/stats/redshift_adapter.rb:349:in `import_s3_path'
        from /var/lib/gems/2.2.0/gems/3scale_backend-2.69.0/lib/3scale/backend/stats/redshift_adapter.rb:334:in `save_in_redshift'
        from /var/lib/gems/2.2.0/gems/3scale_backend-2.69.0/lib/3scale/backend/stats/redshift_adapter.rb:255:in `block in insert_pending_events'
        from /var/lib/gems/2.2.0/gems/3scale_backend-2.69.0/lib/3scale/backend/stats/redshift_adapter.rb:253:in `each'
        from /var/lib/gems/2.2.0/gems/3scale_backend-2.69.0/lib/3scale/backend/stats/redshift_adapter.rb:253:in `insert_pending_events'
        from (irb):6
        from /usr/bin/irb2.2:11:in `<main>'