Shopify / liquid

Liquid markup language. Safe, customer facing template language for flexible web apps.
https://shopify.github.io/liquid/
MIT License
11.05k stars 1.38k forks source link

[base64] Respect string encoding of input in base64_decode filters #1746

Closed gmalette closed 1 year ago

gmalette commented 1 year ago

Ruby's Base64.*decode64 functions return strings encoded in the Encoding::ASCII_8BIT scheme.

Ruby's string append has a special case if the string being appended is encoded over 7 bits, meaning that this works flawlessly:

decoded = @filters.base64_decode('Zm9v') # "foo" as base64
assert_equal("✅ foo", "✅ " + decoded)

However, for characters outside the 7 bit, such as most UTF-8 characters, this leads to a problem in which Ruby fails to concatenate the string, even if it's technically within a compatible domain.

decoded = @filters.base64_decode('4pyF') # "✅" as base64
assert_equal("✅ ✅", "✅ " + decoded)
#=> Minitest::UnexpectedError: Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ASCII-8BIT

To solve this problem and allow decoding emojis, I propose we attempt to change the encoding of the base64-decoded string, test if it's in the correct domain, and revert otherwise.