Encoding issues

[alexfalkowski]

Hi,

Thanks for the great library. We are using MWS::Reports::Client. When we

get_report('ID').body

I can see that text comes back as für

Though when we call

get_report('ID').parse.to_a

We get the text für

Seems to me something is using the wrong encoding. Maybe then CSV parsing is causing it?

[hakanensari]

Just to confirm, are you using the latest version?

[alexfalkowski]

From my Gemfile.lock peddler (1.6.1)

[alexfalkowski]

irb(main):020:0> body.encoding == 'UTF-8'
=> false
irb(main):021:0> body.encoding
=> #<Encoding:UTF-8>
irb(main):022:0> body.encoding == Encoding::UTF_8
=> true

[alexfalkowski]

So it looks like when I body.force_encoding(Encoding::UTF_8) the parsing works correctly.

[alexfalkowski]

So is it forcing the wrong encoding?

[alexfalkowski]

So this fixes is for me

module Peddler::FlatFileParserExtensions
  def scrub_body!(_)
    body.force_encoding(Encoding::UTF_8) unless body.encoding == Encoding::UTF_8
  end
end

class Peddler::FlatFileParser
  prepend Peddler::FlatFileParserExtensions
end

[hakanensari]

Thanks, @alexfalkowski. I pushed your fix to master.

[alexfalkowski]

Thank you!