Encoding::UndefinedConversionError

beydogan commented 9 years ago

I'm getting this exception on markdown render requests for some repos.

Exception message is: ""\xC5" from ASCII-8BIT to UTF-8"

I'm trying to render readme files of repos; ruby/ruby is working fine but airbnb/ruby is not working.

Sample Code:

res = github_api.repos.contents.readme "airbnb", "ruby"
readme_base64 = res.content
decoded = Base64.decode64 readme_base64
readme = github_api.markdown.render text: decoded

Any solutions?

piotrmurach commented 9 years ago

Hi @beydogan, Could you please provide whole exception trace? The obvious question is why the response encoding is different from the character set contained by the markdown? Without spending time on it I cannot provide any answers. There are docs for extra parameters, maybe they will help.

beydogan commented 9 years ago

Hi @peter-murach

I think, Base64.decode64 cant decode some UTF-8 characters. Is there a way to get parsed readme? With custom media types as mentioned here: https://developer.github.com/v3/repos/contents/#custom-media-types

Here is full-trace:

activesupport (4.1.6) lib/active_support/core_ext/object/json.rb:34:in `encode'
activesupport (4.1.6) lib/active_support/core_ext/object/json.rb:34:in `to_json'
activesupport (4.1.6) lib/active_support/core_ext/object/json.rb:34:in `to_json_with_active_support_encoder'
activesupport (4.1.6) lib/active_support/json/encoding.rb:56:in `to_json'
/home/mehmet/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/json/common.rb:223:in `generate'
/home/mehmet/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/json/common.rb:223:in `generate'
activesupport (4.1.6) lib/active_support/json/encoding.rb:96:in `stringify'
activesupport (4.1.6) lib/active_support/json/encoding.rb:34:in `encode'
activesupport (4.1.6) lib/active_support/json/encoding.rb:21:in `encode'
activesupport (4.1.6) lib/active_support/core_ext/object/json.rb:37:in `to_json_with_active_support_encoder'
multi_json (1.10.1) lib/multi_json/adapters/json_common.rb:21:in `dump'
multi_json (1.10.1) lib/multi_json/adapter.rb:24:in `dump'
multi_json (1.10.1) lib/multi_json.rb:136:in `dump'
github_api (0.12.2) lib/github_api/request/jsonize.rb:28:in `encode_body'
github_api (0.12.2) lib/github_api/request/jsonize.rb:16:in `call'
faraday (0.9.0) lib/faraday/rack_builder.rb:139:in `build_response'
faraday (0.9.0) lib/faraday/connection.rb:377:in `run_request'
faraday (0.9.0) lib/faraday/connection.rb:177:in `post'
github_api (0.12.2) lib/github_api/request.rb:68:in `call'
github_api (0.12.2) lib/github_api/request/verbs.rb:35:in `post_request'
github_api (0.12.2) lib/github_api/client/markdown.rb:34:in `render'
github_api (0.12.2) lib/github_api/api.rb:224:in `block in execute'
github_api (0.12.2) lib/github_api/api.rb:211:in `run_callbacks'
github_api (0.12.2) lib/github_api/api.rb:223:in `execute'
github_api (0.12.2) lib/github_api/api.rb:183:in `block in method_added'
actionpack (4.1.6) lib/action_controller/metal/implicit_render.rb:4:in `send_action'
actionpack (4.1.6) lib/abstract_controller/base.rb:189:in `process_action'
actionpack (4.1.6) lib/action_controller/metal/rendering.rb:10:in `process_action'
actionpack (4.1.6) lib/abstract_controller/callbacks.rb:20:in `block in process_action'
activesupport (4.1.6) lib/active_support/callbacks.rb:113:in `call'
activesupport (4.1.6) lib/active_support/callbacks.rb:113:in `call'
activesupport (4.1.6) lib/active_support/callbacks.rb:149:in `block in halting_and_conditional'
activesupport (4.1.6) lib/active_support/callbacks.rb:229:in `call'
activesupport (4.1.6) lib/active_support/callbacks.rb:229:in `block in halting'
activesupport (4.1.6) lib/active_support/callbacks.rb:229:in `call'
activesupport (4.1.6) lib/active_support/callbacks.rb:229:in `block in halting'
activesupport (4.1.6) lib/active_support/callbacks.rb:166:in `call'
activesupport (4.1.6) lib/active_support/callbacks.rb:166:in `block in halting'
activesupport (4.1.6) lib/active_support/callbacks.rb:166:in `call'
activesupport (4.1.6) lib/active_support/callbacks.rb:166:in `block in halting'
activesupport (4.1.6) lib/active_support/callbacks.rb:166:in `call'
activesupport (4.1.6) lib/active_support/callbacks.rb:166:in `block in halting'
activesupport (4.1.6) lib/active_support/callbacks.rb:86:in `call'
activesupport (4.1.6) lib/active_support/callbacks.rb:86:in `run_callbacks'
actionpack (4.1.6) lib/abstract_controller/callbacks.rb:19:in `process_action'
actionpack (4.1.6) lib/action_controller/metal/rescue.rb:29:in `process_action'
actionpack (4.1.6) lib/action_controller/metal/instrumentation.rb:31:in `block in process_action'
activesupport (4.1.6) lib/active_support/notifications.rb:159:in `block in instrument'
activesupport (4.1.6) lib/active_support/notifications/instrumenter.rb:20:in `instrument'
activesupport (4.1.6) lib/active_support/notifications.rb:159:in `instrument'
actionpack (4.1.6) lib/action_controller/metal/instrumentation.rb:30:in `process_action'
actionpack (4.1.6) lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
activerecord (4.1.6) lib/active_record/railties/controller_runtime.rb:18:in `process_action'
actionpack (4.1.6) lib/abstract_controller/base.rb:136:in `process'
actionview (4.1.6) lib/action_view/rendering.rb:30:in `process'
actionpack (4.1.6) lib/action_controller/metal.rb:196:in `dispatch'
actionpack (4.1.6) lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
actionpack (4.1.6) lib/action_controller/metal.rb:232:in `block in action'
actionpack (4.1.6) lib/action_dispatch/routing/route_set.rb:82:in `call'
actionpack (4.1.6) lib/action_dispatch/routing/route_set.rb:82:in `dispatch'
actionpack (4.1.6) lib/action_dispatch/routing/route_set.rb:50:in `call'
actionpack (4.1.6) lib/action_dispatch/journey/router.rb:73:in `block in call'
actionpack (4.1.6) lib/action_dispatch/journey/router.rb:59:in `each'
actionpack (4.1.6) lib/action_dispatch/journey/router.rb:59:in `call'
actionpack (4.1.6) lib/action_dispatch/routing/route_set.rb:678:in `call'
omniauth (1.2.2) lib/omniauth/strategy.rb:186:in `call!'
omniauth (1.2.2) lib/omniauth/strategy.rb:164:in `call'
warden (1.2.3) lib/warden/manager.rb:35:in `block in call'
warden (1.2.3) lib/warden/manager.rb:34:in `catch'
warden (1.2.3) lib/warden/manager.rb:34:in `call'
rack (1.5.2) lib/rack/etag.rb:23:in `call'
rack (1.5.2) lib/rack/conditionalget.rb:25:in `call'
rack (1.5.2) lib/rack/head.rb:11:in `call'
actionpack (4.1.6) lib/action_dispatch/middleware/params_parser.rb:27:in `call'
actionpack (4.1.6) lib/action_dispatch/middleware/flash.rb:254:in `call'
rack (1.5.2) lib/rack/session/abstract/id.rb:225:in `context'
rack (1.5.2) lib/rack/session/abstract/id.rb:220:in `call'
actionpack (4.1.6) lib/action_dispatch/middleware/cookies.rb:560:in `call'
activerecord (4.1.6) lib/active_record/query_cache.rb:36:in `call'
activerecord (4.1.6) lib/active_record/connection_adapters/abstract/connection_pool.rb:621:in `call'
activerecord (4.1.6) lib/active_record/migration.rb:380:in `call'
actionpack (4.1.6) lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
activesupport (4.1.6) lib/active_support/callbacks.rb:82:in `run_callbacks'
actionpack (4.1.6) lib/action_dispatch/middleware/callbacks.rb:27:in `call'
actionpack (4.1.6) lib/action_dispatch/middleware/reloader.rb:73:in `call'
actionpack (4.1.6) lib/action_dispatch/middleware/remote_ip.rb:76:in `call'
actionpack (4.1.6) lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
actionpack (4.1.6) lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
railties (4.1.6) lib/rails/rack/logger.rb:38:in `call_app'
railties (4.1.6) lib/rails/rack/logger.rb:20:in `block in call'
activesupport (4.1.6) lib/active_support/tagged_logging.rb:68:in `block in tagged'
activesupport (4.1.6) lib/active_support/tagged_logging.rb:26:in `tagged'
activesupport (4.1.6) lib/active_support/tagged_logging.rb:68:in `tagged'
railties (4.1.6) lib/rails/rack/logger.rb:20:in `call'
actionpack (4.1.6) lib/action_dispatch/middleware/request_id.rb:21:in `call'
rack (1.5.2) lib/rack/methodoverride.rb:21:in `call'
rack (1.5.2) lib/rack/runtime.rb:17:in `call'
activesupport (4.1.6) lib/active_support/cache/strategy/local_cache_middleware.rb:26:in `call'
rack (1.5.2) lib/rack/lock.rb:17:in `call'
actionpack (4.1.6) lib/action_dispatch/middleware/static.rb:64:in `call'
rack (1.5.2) lib/rack/sendfile.rb:112:in `call'
railties (4.1.6) lib/rails/engine.rb:514:in `call'
railties (4.1.6) lib/rails/application.rb:144:in `call'
rack (1.5.2) lib/rack/lock.rb:17:in `call'
rack (1.5.2) lib/rack/content_length.rb:14:in `call'
rack (1.5.2) lib/rack/handler/webrick.rb:60:in `service'
/home/mehmet/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/webrick/httpserver.rb:138:in `service'
/home/mehmet/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/webrick/httpserver.rb:94:in `run'
/home/mehmet/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/webrick/server.rb:295:in `block in start_thread'

beydogan commented 9 years ago

I tried following request;

github_api.repos.contents.readme "airbnb", "ruby", media: 'html'

and got:

MultiJson::ParseError in ReposController#show

795: unexpected token at '<div .. LOTS OF HTML here'

Trace:

/home/mehmet/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/json/common.rb:155:in `parse'
/home/mehmet/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/json/common.rb:155:in `parse'
multi_json (1.10.1) lib/multi_json/adapters/json_common.rb:16:in `load'
multi_json (1.10.1) lib/multi_json/adapter.rb:20:in `load'
multi_json (1.10.1) lib/multi_json.rb:119:in `load'
github_api (0.12.2) lib/github_api/jsonable.rb:12:in `decode'
github_api (0.12.2) lib/github_api/response/jsonize.rb:13:in `block in <class:Jsonize>'
github_api (0.12.2) lib/github_api/response/jsonize.rb:25:in `call'
github_api (0.12.2) lib/github_api/response/jsonize.rb:25:in `parse'
faraday (0.9.0) lib/faraday/response.rb:16:in `on_complete'
faraday (0.9.0) lib/faraday/response.rb:9:in `block in call'
faraday (0.9.0) lib/faraday/response.rb:57:in `on_complete'
faraday (0.9.0) lib/faraday/response.rb:8:in `call'
faraday (0.9.0) lib/faraday/response.rb:8:in `call'
faraday (0.9.0) lib/faraday/request/url_encoded.rb:15:in `call'
faraday (0.9.0) lib/faraday/request/multipart.rb:14:in `call'
github_api (0.12.2) lib/github_api/request/jsonize.rb:23:in `call'
faraday (0.9.0) lib/faraday/rack_builder.rb:139:in `build_response'
faraday (0.9.0) lib/faraday/connection.rb:377:in `run_request'
faraday (0.9.0) lib/faraday/connection.rb:140:in `get'
github_api (0.12.2) lib/github_api/request.rb:68:in `call'
github_api (0.12.2) lib/github_api/request/verbs.rb:20:in `get_request'
github_api (0.12.2) lib/github_api/client/repos/contents.rb:30:in `readme'
github_api (0.12.2) lib/github_api/api.rb:224:in `block in execute'
github_api (0.12.2) lib/github_api/api.rb:211:in `run_callbacks'
github_api (0.12.2) lib/github_api/api.rb:223:in `execute'
github_api (0.12.2) lib/github_api/api.rb:183:in `block in method_added'
actionpack (4.1.6) lib/action_controller/metal/implicit_render.rb:4:in `send_action'
actionpack (4.1.6) lib/abstract_controller/base.rb:189:in `process_action'
actionpack (4.1.6) lib/action_controller/metal/rendering.rb:10:in `process_action'
actionpack (4.1.6) lib/abstract_controller/callbacks.rb:20:in `block in process_action'
activesupport (4.1.6) lib/active_support/callbacks.rb:113:in `call'
activesupport (4.1.6) lib/active_support/callbacks.rb:113:in `call'
activesupport (4.1.6) lib/active_support/callbacks.rb:149:in `block in halting_and_conditional'
activesupport (4.1.6) lib/active_support/callbacks.rb:229:in `call'
activesupport (4.1.6) lib/active_support/callbacks.rb:229:in `block in halting'
activesupport (4.1.6) lib/active_support/callbacks.rb:229:in `call'
activesupport (4.1.6) lib/active_support/callbacks.rb:229:in `block in halting'
activesupport (4.1.6) lib/active_support/callbacks.rb:166:in `call'
activesupport (4.1.6) lib/active_support/callbacks.rb:166:in `block in halting'
activesupport (4.1.6) lib/active_support/callbacks.rb:166:in `call'
activesupport (4.1.6) lib/active_support/callbacks.rb:166:in `block in halting'
activesupport (4.1.6) lib/active_support/callbacks.rb:166:in `call'
activesupport (4.1.6) lib/active_support/callbacks.rb:166:in `block in halting'
activesupport (4.1.6) lib/active_support/callbacks.rb:86:in `call'
activesupport (4.1.6) lib/active_support/callbacks.rb:86:in `run_callbacks'
actionpack (4.1.6) lib/abstract_controller/callbacks.rb:19:in `process_action'
actionpack (4.1.6) lib/action_controller/metal/rescue.rb:29:in `process_action'
actionpack (4.1.6) lib/action_controller/metal/instrumentation.rb:31:in `block in process_action'
activesupport (4.1.6) lib/active_support/notifications.rb:159:in `block in instrument'
activesupport (4.1.6) lib/active_support/notifications/instrumenter.rb:20:in `instrument'
activesupport (4.1.6) lib/active_support/notifications.rb:159:in `instrument'
actionpack (4.1.6) lib/action_controller/metal/instrumentation.rb:30:in `process_action'
actionpack (4.1.6) lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
activerecord (4.1.6) lib/active_record/railties/controller_runtime.rb:18:in `process_action'
actionpack (4.1.6) lib/abstract_controller/base.rb:136:in `process'
actionview (4.1.6) lib/action_view/rendering.rb:30:in `process'
actionpack (4.1.6) lib/action_controller/metal.rb:196:in `dispatch'
actionpack (4.1.6) lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
actionpack (4.1.6) lib/action_controller/metal.rb:232:in `block in action'
actionpack (4.1.6) lib/action_dispatch/routing/route_set.rb:82:in `call'
actionpack (4.1.6) lib/action_dispatch/routing/route_set.rb:82:in `dispatch'
actionpack (4.1.6) lib/action_dispatch/routing/route_set.rb:50:in `call'
actionpack (4.1.6) lib/action_dispatch/journey/router.rb:73:in `block in call'
actionpack (4.1.6) lib/action_dispatch/journey/router.rb:59:in `each'
actionpack (4.1.6) lib/action_dispatch/journey/router.rb:59:in `call'
actionpack (4.1.6) lib/action_dispatch/routing/route_set.rb:678:in `call'
omniauth (1.2.2) lib/omniauth/strategy.rb:186:in `call!'
omniauth (1.2.2) lib/omniauth/strategy.rb:164:in `call'
warden (1.2.3) lib/warden/manager.rb:35:in `block in call'
warden (1.2.3) lib/warden/manager.rb:34:in `catch'
warden (1.2.3) lib/warden/manager.rb:34:in `call'
rack (1.5.2) lib/rack/etag.rb:23:in `call'
rack (1.5.2) lib/rack/conditionalget.rb:25:in `call'
rack (1.5.2) lib/rack/head.rb:11:in `call'
actionpack (4.1.6) lib/action_dispatch/middleware/params_parser.rb:27:in `call'
actionpack (4.1.6) lib/action_dispatch/middleware/flash.rb:254:in `call'
rack (1.5.2) lib/rack/session/abstract/id.rb:225:in `context'
rack (1.5.2) lib/rack/session/abstract/id.rb:220:in `call'
actionpack (4.1.6) lib/action_dispatch/middleware/cookies.rb:560:in `call'
activerecord (4.1.6) lib/active_record/query_cache.rb:36:in `call'
activerecord (4.1.6) lib/active_record/connection_adapters/abstract/connection_pool.rb:621:in `call'
activerecord (4.1.6) lib/active_record/migration.rb:380:in `call'
actionpack (4.1.6) lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
activesupport (4.1.6) lib/active_support/callbacks.rb:82:in `run_callbacks'
actionpack (4.1.6) lib/action_dispatch/middleware/callbacks.rb:27:in `call'
actionpack (4.1.6) lib/action_dispatch/middleware/reloader.rb:73:in `call'
actionpack (4.1.6) lib/action_dispatch/middleware/remote_ip.rb:76:in `call'
actionpack (4.1.6) lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
actionpack (4.1.6) lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
railties (4.1.6) lib/rails/rack/logger.rb:38:in `call_app'
railties (4.1.6) lib/rails/rack/logger.rb:20:in `block in call'
activesupport (4.1.6) lib/active_support/tagged_logging.rb:68:in `block in tagged'
activesupport (4.1.6) lib/active_support/tagged_logging.rb:26:in `tagged'
activesupport (4.1.6) lib/active_support/tagged_logging.rb:68:in `tagged'
railties (4.1.6) lib/rails/rack/logger.rb:20:in `call'
actionpack (4.1.6) lib/action_dispatch/middleware/request_id.rb:21:in `call'
rack (1.5.2) lib/rack/methodoverride.rb:21:in `call'
rack (1.5.2) lib/rack/runtime.rb:17:in `call'
activesupport (4.1.6) lib/active_support/cache/strategy/local_cache_middleware.rb:26:in `call'
rack (1.5.2) lib/rack/lock.rb:17:in `call'
actionpack (4.1.6) lib/action_dispatch/middleware/static.rb:64:in `call'
rack (1.5.2) lib/rack/sendfile.rb:112:in `call'
railties (4.1.6) lib/rails/engine.rb:514:in `call'
railties (4.1.6) lib/rails/application.rb:144:in `call'
rack (1.5.2) lib/rack/lock.rb:17:in `call'
rack (1.5.2) lib/rack/content_length.rb:14:in `call'
rack (1.5.2) lib/rack/handler/webrick.rb:60:in `service'
/home/mehmet/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/webrick/httpserver.rb:138:in `service'
/home/mehmet/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/webrick/httpserver.rb:94:in `run'
/home/mehmet/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/webrick/server.rb:295:in `block in start_thread'

piotrmurach commented 9 years ago

How about before you Base64.decode64 check what is the res.content.encoding to make sure it's valid utf-8 and encode the string in utf-8 if not?

piotrmurach / github

Encoding::UndefinedConversionError #205