UTF-8 encoding issue when attempting to convert the response body of image URLs to JSON

[postmodern]

Trying to output a favicon.ico URL to JSON causes a UTF-8 encoding issue.

>>> Found new URL https://app.deriv.com/favicon.ico for website http://app.deriv.com
13.11s     warn: Async::Task [oid=0x1978] [ec=0x198c] [pid=23202] [2024-07-12 14:32:23 -0700]
               | Task may have ended with unhandled exception.
               |   JSON::GeneratorError: source sequence is illegal/malformed utf-8
               |   → lib/ronin/recon/value.rb:95 in `to_json'
               |     lib/ronin/recon/value.rb:95 in `to_json'
               |     /data/home/postmodern/code/ronin-rb/vendor/bundle/ruby/3.1.0/gems/ronin-core-0.2.0.rc1/lib/ronin/core/output_formats/ndjson.rb:47 in `<<'
               |     lib/ronin/recon/cli/commands/run.rb:285 in `block (2 levels) in run'
               |     lib/ronin/recon/engine.rb:395 in `block in on_value'
               |     lib/ronin/recon/engine.rb:393 in `each'
               |     lib/ronin/recon/engine.rb:393 in `on_value'
               |     lib/ronin/recon/engine.rb:550 in `process_value'
               |     lib/ronin/recon/engine.rb:443 in `process'
               |     lib/ronin/recon/engine.rb:231 in `block in run'
               |     /data/home/postmodern/code/ronin-rb/vendor/bundle/ruby/3.1.0/gems/async-2.12.1/lib/async/task.rb:164 in `block in run'
               |     /data/home/postmodern/code/ronin-rb/vendor/bundle/ruby/3.1.0/gems/async-2.12.1/lib/async/task.rb:377 in `block in schedule'

[postmodern]

We could either:

A. Base64 encode all response bodies. Downside would be that you could not easily grep plain-text response bodies. B. Only Base64 encode response bodies which are ended as binary Strings. We would need to add some kind of indicator that the response body was Base64 encoded.

{"binary":true,"body":"..."}

{"body":{"encoding":"ASCII-8BIT","value":"..."}}

{"body":{"encoding":"base64","value":"..."}}