search

ankane / blazer

Business intelligence made simple
MIT License
4.5k stars 471 forks source link

Incompatible character encodings error while rendering results #468

Closed tiagotex closed 6 months ago

tiagotex commented 6 months ago

We have noticed some encoding bugs when rendering Blazer queries using a MySQL 8 database.

The error is ActionView::Template::Error incompatible character encodings: UTF-8 and ASCII-8BIT (ActionView::Template::Error)

error

The error seemed similar to https://github.com/ankane/blazer/issues/459, which was addressed on https://github.com/ankane/blazer/commit/fd04dd05cbab55c611d9396fef933db3d9bb1d1b, since we are using the trilogy adaptor. However, after switching back to mysql2, the error is still present.

It seems like the issue is with rows returning ASCII-8BIT encoded data, and this is causing the crash in ActionView.

I tried to create a reproduction script but could not reproduce the problem except when querying the current data source.

The previous fixes for similar issues were to force UTF8 encoding, I tried to do a quick monkey patch by changing the base helper https://github.com/ankane/blazer/blob/master/app/helpers/blazer/base_helper.rb#L14

module Blazer
  module BaseHelper
    def blazer_format_value(key, value)
      if value.is_a?(Numeric) && !key.to_s.end_with?("id") && !key.to_s.start_with?("id")
        number_with_delimiter(value)
      elsif value.is_a?(String) && value =~ BLAZER_URL_REGEX
        # see if image or link
        if Blazer.images && (key.include?("image") || BLAZER_IMAGE_EXT.include?(value.split(".").last.split("?").first.try(:downcase)))
          link_to value, target: "_blank", rel: "noopener" do
            image_tag value, referrerpolicy: "no-referrer"
          end
        else
          link_to value, value, target: "_blank", rel: "noopener"
        end
      elsif value.is_a?(String) # This is the condition that we added to override encoding
        value.force_encoding("UTF-8")
      else
        value
      end
    end
  end
end

This fixed the issue, but not sure if this is the desired fix.

Please let me know if you would like me to submit a PR with the fix above.

Blazer: 3.0.3 Rails: 7.1.3.2 Ruby 3.3.0 Database: MySQL 8 with Trilogy adapter

Stacktrace ``` Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ASCII-8BIT (Encoding::CompatibilityError) from actionview (7.1.3.2) lib/action_view/buffers.rb:45:in `<<' from blazer (3.0.3) app/views/blazer/queries/run.html.erb:164:in `block (2 levels) in __var_www_tap_shared_bundle___ruby_______gems_blazer_______app_views_blazer_queries_run_html_erb___380738227109043872_767120' from blazer (3.0.3) app/views/blazer/queries/run.html.erb:151:in `each' from blazer (3.0.3) app/views/blazer/queries/run.html.erb:151:in `each_with_index' from blazer (3.0.3) app/views/blazer/queries/run.html.erb:151:in `block in __var_www_tap_shared_bundle___ruby_______gems_blazer_______app_views_blazer_queries_run_html_erb___380738227109043872_767120' from blazer (3.0.3) app/views/blazer/queries/run.html.erb:149:in `each' from blazer (3.0.3) app/views/blazer/queries/run.html.erb:149:in `__var_www_tap_shared_bundle___ruby_______gems_blazer_______app_views_blazer_queries_run_html_erb___380738227109043872_767120' from actionview (7.1.3.2) lib/action_view/base.rb:264:in `public_send' from actionview (7.1.3.2) lib/action_view/base.rb:264:in `_run' from actionview (7.1.3.2) lib/action_view/template.rb:261:in `block in render' from actionview (7.1.3.2) lib/action_view/template.rb:549:in `instrument_render_template' from actionview (7.1.3.2) lib/action_view/template.rb:249:in `render' from actionview (7.1.3.2) lib/action_view/renderer/template_renderer.rb:66:in `block (2 levels) in render_template' from actionview (7.1.3.2) lib/action_view/renderer/template_renderer.rb:60:in `block in render_template' from actionview (7.1.3.2) lib/action_view/renderer/template_renderer.rb:80:in `render_with_layout' from actionview (7.1.3.2) lib/action_view/renderer/template_renderer.rb:59:in `render_template' from actionview (7.1.3.2) lib/action_view/renderer/template_renderer.rb:11:in `render' from actionview (7.1.3.2) lib/action_view/renderer/renderer.rb:63:in `render_template_to_object' from actionview (7.1.3.2) lib/action_view/renderer/renderer.rb:31:in `render_to_object' from actionview (7.1.3.2) lib/action_view/rendering.rb:135:in `block in _render_template' from actionview (7.1.3.2) lib/action_view/base.rb:291:in `in_rendering_context' from actionview (7.1.3.2) lib/action_view/rendering.rb:134:in `_render_template' from actionpack (7.1.3.2) lib/action_controller/metal/streaming.rb:256:in `_render_template' from actionview (7.1.3.2) lib/action_view/rendering.rb:121:in `render_to_body' from actionpack (7.1.3.2) lib/action_controller/metal/rendering.rb:158:in `render_to_body' from actionpack (7.1.3.2) lib/action_controller/metal/renderers.rb:141:in `render_to_body' from actionpack (7.1.3.2) lib/abstract_controller/rendering.rb:27:in `render' from actionpack (7.1.3.2) lib/action_controller/metal/rendering.rb:139:in `render' from actionpack (7.1.3.2) lib/action_controller/metal/instrumentation.rb:29:in `block (2 levels) in render' from benchmark.rb:313:in `realtime' from activesupport (7.1.3.2) lib/active_support/core_ext/benchmark.rb:14:in `ms' from actionpack (7.1.3.2) lib/action_controller/metal/instrumentation.rb:29:in `block in render' from actionpack (7.1.3.2) lib/action_controller/metal/instrumentation.rb:98:in `cleanup_view_runtime' from activerecord (7.1.3.2) lib/active_record/railties/controller_runtime.rb:39:in `cleanup_view_runtime' from actionpack (7.1.3.2) lib/action_controller/metal/instrumentation.rb:28:in `render' from blazer (3.0.3) app/controllers/blazer/queries_controller.rb:268:in `block (2 levels) in render_run' from actionpack (7.1.3.2) lib/action_controller/metal/mime_responds.rb:214:in `respond_to' from blazer (3.0.3) app/controllers/blazer/queries_controller.rb:266:in `render_run' from blazer (3.0.3) app/controllers/blazer/queries_controller.rb:169:in `run' from actionpack (7.1.3.2) lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action' from actionpack (7.1.3.2) lib/abstract_controller/base.rb:224:in `process_action' from actionpack (7.1.3.2) lib/action_controller/metal/rendering.rb:165:in `process_action' from actionpack (7.1.3.2) lib/abstract_controller/callbacks.rb:259:in `block in process_action' from activesupport (7.1.3.2) lib/active_support/callbacks.rb:121:in `block in run_callbacks' from sentry-rails (5.16.1) lib/sentry/rails/controller_transaction.rb:17:in `block in sentry_around_action' from sentry-ruby (5.16.1) lib/sentry/hub.rb:109:in `block in with_child_span' from sentry-ruby (5.16.1) lib/sentry/span.rb:206:in `with_child_span' from sentry-ruby (5.16.1) lib/sentry/hub.rb:107:in `with_child_span' from sentry-ruby (5.16.1) lib/sentry-ruby.rb:478:in `with_child_span' from sentry-rails (5.16.1) lib/sentry/rails/controller_transaction.rb:14:in `sentry_around_action' from activesupport (7.1.3.2) lib/active_support/callbacks.rb:130:in `block in run_callbacks' from activesupport (7.1.3.2) lib/active_support/callbacks.rb:141:in `run_callbacks' from actionpack (7.1.3.2) lib/abstract_controller/callbacks.rb:258:in `process_action' from actionpack (7.1.3.2) lib/action_controller/metal/rescue.rb:25:in `process_action' from actionpack (7.1.3.2) lib/action_controller/metal/instrumentation.rb:74:in `block in process_action' from actionpack (7.1.3.2) lib/action_controller/metal/instrumentation.rb:73:in `process_action' from actionpack (7.1.3.2) lib/action_controller/metal/params_wrapper.rb:261:in `process_action' from activerecord (7.1.3.2) lib/active_record/railties/controller_runtime.rb:32:in `process_action' from actionpack (7.1.3.2) lib/abstract_controller/base.rb:160:in `process' from actionview (7.1.3.2) lib/action_view/rendering.rb:40:in `process' from actionpack (7.1.3.2) lib/action_controller/metal.rb:227:in `dispatch' from actionpack (7.1.3.2) lib/action_controller/metal.rb:309:in `dispatch' from actionpack (7.1.3.2) lib/action_dispatch/routing/route_set.rb:49:in `dispatch' from actionpack (7.1.3.2) lib/action_dispatch/routing/route_set.rb:32:in `serve' from actionpack (7.1.3.2) lib/action_dispatch/journey/router.rb:51:in `block in serve' from actionpack (7.1.3.2) lib/action_dispatch/journey/router.rb:131:in `block in find_routes' from actionpack (7.1.3.2) lib/action_dispatch/journey/router.rb:124:in `each' from actionpack (7.1.3.2) lib/action_dispatch/journey/router.rb:124:in `find_routes' from actionpack (7.1.3.2) lib/action_dispatch/journey/router.rb:32:in `serve' from actionpack (7.1.3.2) lib/action_dispatch/routing/route_set.rb:882:in `call' from skylight (6.0.4) lib/skylight/probes/action_dispatch/routing/route_set.rb:10:in `block in call' from skylight (6.0.4) lib/skylight/probes/action_dispatch/routing/route_set.rb:10:in `call' from railties (7.1.3.2) lib/rails/engine.rb:536:in `call' from railties (7.1.3.2) lib/rails/railtie.rb:226:in `public_send' from railties (7.1.3.2) lib/rails/railtie.rb:226:in `method_missing' from actionpack (7.1.3.2) lib/action_dispatch/routing/mapper.rb:22:in `block in ' from actionpack (7.1.3.2) lib/action_dispatch/routing/mapper.rb:51:in `serve' from actionpack (7.1.3.2) lib/action_dispatch/journey/router.rb:51:in `block in serve' from actionpack (7.1.3.2) lib/action_dispatch/journey/router.rb:131:in `block in find_routes' from actionpack (7.1.3.2) lib/action_dispatch/journey/router.rb:124:in `each' from actionpack (7.1.3.2) lib/action_dispatch/journey/router.rb:124:in `find_routes' from actionpack (7.1.3.2) lib/action_dispatch/journey/router.rb:32:in `serve' from actionpack (7.1.3.2) lib/action_dispatch/routing/route_set.rb:882:in `call' from skylight (6.0.4) lib/skylight/probes/action_dispatch/routing/route_set.rb:10:in `block in call' from skylight (6.0.4) lib/skylight/probes/action_dispatch/routing/route_set.rb:10:in `call' from singed (0.2.1) lib/singed/rack_middleware.rb:17:in `call' from warden (1.2.9) lib/warden/manager.rb:36:in `block in call' from warden (1.2.9) lib/warden/manager.rb:34:in `catch' from warden (1.2.9) lib/warden/manager.rb:34:in `call' from rack (3.0.9.1) lib/rack/tempfile_reaper.rb:20:in `call' from rack (3.0.9.1) lib/rack/etag.rb:29:in `call' from rack (3.0.9.1) lib/rack/conditional_get.rb:43:in `call' from rack (3.0.9.1) lib/rack/head.rb:15:in `call' from actionpack (7.1.3.2) lib/action_dispatch/http/permissions_policy.rb:36:in `call' from actionpack (7.1.3.2) lib/action_dispatch/http/content_security_policy.rb:33:in `call' from rack-session (2.0.0) lib/rack/session/abstract/id.rb:272:in `context' from rack-session (2.0.0) lib/rack/session/abstract/id.rb:266:in `call' from actionpack (7.1.3.2) lib/action_dispatch/middleware/cookies.rb:689:in `call' from actionpack (7.1.3.2) lib/action_dispatch/middleware/callbacks.rb:29:in `block in call' from activesupport (7.1.3.2) lib/active_support/callbacks.rb:101:in `run_callbacks' from actionpack (7.1.3.2) lib/action_dispatch/middleware/callbacks.rb:28:in `call' from sentry-rails (5.16.1) lib/sentry/rails/rescued_exception_interceptor.rb:12:in `call' from actionpack (7.1.3.2) lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call' from sentry-ruby (5.16.1) lib/sentry/rack/capture_exceptions.rb:28:in `block (2 levels) in call' from sentry-ruby (5.16.1) lib/sentry/hub.rb:251:in `with_session_tracking' from sentry-ruby (5.16.1) lib/sentry-ruby.rb:391:in `with_session_tracking' from sentry-ruby (5.16.1) lib/sentry/rack/capture_exceptions.rb:19:in `block in call' from sentry-ruby (5.16.1) lib/sentry/hub.rb:59:in `with_scope' from sentry-ruby (5.16.1) lib/sentry-ruby.rb:371:in `with_scope' from sentry-ruby (5.16.1) lib/sentry/rack/capture_exceptions.rb:18:in `call' from actionpack (7.1.3.2) lib/action_dispatch/middleware/show_exceptions.rb:31:in `call' from lograge (0.14.0) lib/lograge/rails_ext/rack/logger.rb:18:in `call_app' from railties (7.1.3.2) lib/rails/rack/logger.rb:24:in `block in call' from logstash-logger (0.26.1) lib/logstash-logger/tagged_logging.rb:5:in `block in tagged' from logstash-logger (0.26.1) lib/logstash-logger/tagged_logging.rb:16:in `tagged' from logstash-logger (0.26.1) lib/logstash-logger/tagged_logging.rb:5:in `tagged' from activesupport (7.1.3.2) lib/active_support/broadcast_logger.rb:240:in `method_missing' from railties (7.1.3.2) lib/rails/rack/logger.rb:24:in `call' from loggery-gem (87fb35a9b4a3) lib/loggery/metadata/middleware/rack.rb:16:in `block in call' from loggery-gem (87fb35a9b4a3) lib/loggery/metadata/store.rb:15:in `with_metadata' from loggery-gem (87fb35a9b4a3) lib/loggery/metadata/middleware/rack.rb:14:in `call' from actionpack (7.1.3.2) lib/action_dispatch/middleware/remote_ip.rb:92:in `call' from request_store (1.6.0) lib/request_store/middleware.rb:19:in `call' from actionpack (7.1.3.2) lib/action_dispatch/middleware/request_id.rb:28:in `call' from skylight (6.0.4) lib/skylight/probes/action_dispatch/request_id.rb:8:in `call' from rack (3.0.9.1) lib/rack/method_override.rb:28:in `call' from rack (3.0.9.1) lib/rack/runtime.rb:24:in `call' from actionpack (7.1.3.2) lib/action_dispatch/middleware/executor.rb:14:in `call' from rack (3.0.9.1) lib/rack/sendfile.rb:114:in `call' from actionpack (7.1.3.2) lib/action_dispatch/middleware/ssl.rb:79:in `call' from actionpack (7.1.3.2) lib/action_dispatch/middleware/host_authorization.rb:141:in `call' from rack-mini-profiler (3.3.1) lib/mini_profiler.rb:191:in `call' from skylight (6.0.4) lib/skylight/middleware.rb:99:in `call' from railties (7.1.3.2) lib/rails/engine.rb:536:in `call' from puma (6.4.2) lib/puma/configuration.rb:272:in `call' from puma (6.4.2) lib/puma/request.rb:100:in `block in handle_request' from puma (6.4.2) lib/puma/thread_pool.rb:378:in `with_force_shutdown' from puma (6.4.2) lib/puma/request.rb:99:in `handle_request' from puma (6.4.2) lib/puma/server.rb:464:in `process_client' from puma (6.4.2) lib/puma/server.rb:245:in `block in run' from puma (6.4.2) lib/puma/thread_pool.rb:155:in `block in spawn_thread' ```
ankane commented 6 months ago

Hi @tiagotex, thanks for reporting. Let's discuss in #466.