getting invalid byte sequence UTF-8 when retreiving MWS Feed

[noorani786]

Mws::Feeds.client(m.amazon_account).get_feed_submission_result('84223017123') ArgumentError: invalid byte sequence in UTF-8 from /var/www/ncube_ecomm/shared/bundle/ruby/2.3.0/bundler/gems/peddler-b4f10425dbbe/lib/peddler/flat_file_parser.rb:35:in split' from /var/www/ncube_ecomm/shared/bundle/ruby/2.3.0/bundler/gems/peddler-b4f10425dbbe/lib/peddler/flat_file_parser.rb:35:inextract_content' from /var/www/ncube_ecomm/shared/bundle/ruby/2.3.0/bundler/gems/peddler-b4f10425dbbe/lib/peddler/flat_file_parser.rb:16:in initialize' from /var/www/ncube_ecomm/shared/bundle/ruby/2.3.0/bundler/gems/peddler-b4f10425dbbe/lib/peddler/parser.rb:20:innew' from /var/www/ncube_ecomm/shared/bundle/ruby/2.3.0/bundler/gems/peddler-b4f10425dbbe/lib/peddler/parser.rb:20:in new' from /var/www/ncube_ecomm/shared/bundle/ruby/2.3.0/bundler/gems/peddler-b4f10425dbbe/lib/peddler/client.rb:173:inrun' from /var/www/ncube_ecomm/shared/bundle/ruby/2.3.0/bundler/gems/peddler-b4f10425dbbe/lib/mws/feeds/client.rb:112:in `get_feed_submission_result'

[hakanensari]

Which marketplace are you running into this? What's the report type? It seems like we're not correctly setting the encoding type here.

[noorani786]

US market This is when I try to read the feed sumbission result report after submitting product listings.

[philsmy]

Yes, I have the same issue...do we have a solution for this?

[hakanensari]

@noorani786, @philsmy I just fixed another encoding issue, but this one seems distinct. Would you mind sharing with me the raw response body or the part you think is causing the issue?

[philsmy]

This is always happening in getting a report.

Here's the stack trace

ArgumentError: invalid byte sequence in UTF-8
    from /usr/local/rubies/2.2.2/lib/ruby/2.2.0/csv.rb:2007:in `=~'
    from /usr/local/rubies/2.2.2/lib/ruby/2.2.0/csv.rb:2007:in `init_separators'
    from /usr/local/rubies/2.2.2/lib/ruby/2.2.0/csv.rb:1529:in `initialize'
    from /usr/local/rubies/2.2.2/lib/ruby/2.2.0/csv.rb:1295:in `new'
    from /usr/local/rubies/2.2.2/lib/ruby/2.2.0/csv.rb:1295:in `parse'
    from /mnt/zonmaster-production/shared/bundle/ruby/2.2.0/bundler/gems/peddler-afa569715ef1/lib/peddler/flat_file_parser.rb:20:in `parse'

Here's what's coming down (I think I've obfuscated enough)

#<Excon::Response:0x007f2c6841be50 @data={:body=>"Date\tRating\tComments\tYour Response\tArrived on Time\tItem as Described\tCustomer Service\tOrder ID\tRater Email\tRater Role\t\n15/01/17\t2\tLa commande n'est pas encore livr\xE9e.\t\xA0Je suis tellement d\xE9sol\xE9e de conna\xEEtre que vous n'avez pas encore re\xE7u le colis,je viens de vous envoyer un message,on va faire tous les efforts pour vous aider et vous faire satisfaire!\tNo\tNo\t-\t407-xxxx-5887529\xxxx@marketplace.amazon.fr\tBuyer\n21/01/17\t1\tI never received this necklace.\t \tNo\tNo\t-\t026-9686605-xxxx\xxxx@marketplace.amazon.co.uk\tBuyer\n19/01/17\t3\tI'm a bit disappointed as the original picture stated that \"what u see is what u get\" this was regarding the picture with all jewellery showing. I wanted the whole thing Which reads, Jane Stone Bohemium style Foot jewellery with coin charms old silver alloy Anklet! I can't find the Foot jewellery that's advertised. I've received the Anklet. Please can u help?\t \tYes\t-\t-\t205-8174896-6743550\xxxx@marketplace.amazon.co.uk\tBuyer\n", :cookies=>[], :host=>"mws-eu.amazonservices.com", :headers=>{"Server"=>"Server", "Date"=>"Wed, 25 Jan 2017 01:46:40 GMT", "Content-Type"=>"text/plain;charset=Cp1252", "Content-Length"=>"997", "Connection"=>"keep-alive", "x-mws-quota-max"=>"60.0", "x-mws-quota-remaining"=>"59.0", "x-mws-quota-resetsOn"=>"2017-01-25T02:04:00.000Z", "Content-MD5"=>"xxx/yDg8brp9LdZbEhYw==", "x-mws-response-context"=>"xx+7EWbuMPFOfYKqaU4fCo=, il/xx/xx+RNlnXt+6HffFzaOlZX41YLafJP86VZ/yeFF0aQe TRK7/fiW0g==", "x-amz-request-id"=>"xx-c809-4ac3-bed2-xx", "x-mws-request-id"=>"xx-c809-4ac3-bed2-xx", "x-mws-timestamp"=>"2017-01-25T01:46:40.192Z", "Vary"=>"Accept-Encoding,User-Agent"}, :path=>"/", :port=>443, :status=>200, :status_line=>"HTTP/1.1 200 OK\r\n", :reason_phrase=>"OK", :remote_ip=>"54.239.39.200", :local_port=>49813, :local_address=>"xx.xx.xx.xx"}, @body="Date\tRating\tComments\tYour Response\tArrived on Time\tItem as Described\tCustomer Service\tOrder ID\tRater Email\tRater Role\t\n15/01/17\t2\tLa commande n'est pas encore livr\xE9e.\t\xA0Je suis tellement d\xE9sol\xE9e de conna\xEEtre que vous n'avez pas encore re\xE7u le colis,je viens de vous envoyer un message,on va faire tous les efforts pour vous aider et vous faire satisfaire!\tNo\tNo\t-\t407-8883641-5887529\xx@marketplace.amazon.fr\tBuyer\n21/01/17\t1\tI never received this necklace.\t \tNo\tNo\t-\t026-9686605-8193138\xx@marketplace.amazon.co.uk\tBuyer\n19/01/17\t3\tI'm a bit disappointed as the original picture stated that \"what u see is what u get\" this was regarding the picture with all jewellery showing. I wanted the whole thing Which reads, Jane Stone Bohemium style Foot jewellery with coin charms old silver alloy Anklet! I can't find the Foot jewellery that's advertised. I've received the Anklet. Please can u help?\t \tYes\t-\t-\t205-8174896-6743550\t4r6pydyzm65s5mr@marketplace.amazon.co.uk\tBuyer\n", @headers={"Server"=>"Server", "Date"=>"Wed, 25 Jan 2017 01:46:40 GMT", "Content-Type"=>"text/plain;charset=Cp1252", "Content-Length"=>"997", "Connection"=>"keep-alive", "x-mws-quota-max"=>"60.0", "x-mws-quota-remaining"=>"59.0", "x-mws-quota-resetsOn"=>"2017-01-25T02:04:00.000Z", "Content-MD5"=>"xx/xx==", "x-mws-response-context"=>"xx+7EWbuMPFOfYKqaU4fCo=, il/YgmkBAliuDu9FtWS4f/xx+xx+6HffFzaOlZX41YLafJP86VZ/yeFF0aQe TRK7/fiW0g==", "x-amz-request-id"=>"4052f460-c809-4ac3-bed2-dc8df67a6ef4", "x-mws-request-id"=>"xx-c809-4ac3-bed2-xx", "x-mws-timestamp"=>"2017-01-25T01:46:40.192Z", "Vary"=>"Accept-Encoding,User-Agent"}, @status=200, @remote_ip="54.239.39.200", @local_port=49813, @local_address="xx.xx.xx.xx">

[hakanensari]

A heisenbug — I can't reproduce this locally and am wondering if something about the environment is causing this.

I just pushed a small change to master to make the parser more lenient. Could you check if it's of any help? This will replace invalid characters with question marks.

[philsmy]

Sure. This happens for sure on our Ubuntu server, but I'll check if it happens locally. It only happens on a few feedback reports... I'll try to test this today.

On Thu, Jan 26, 2017 at 7:30 AM, Hakan Ensari notifications@github.com wrote:

A typical heisenbug — I can't reproduce this locally and am wondering if something about the environment is causing this.

I just pushed a small change to master to make the parser more lenient. Could you check if it's of any help? This will replace invalid characters with question marks.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hakanensari/peddler/issues/78#issuecomment-275253592, or mute the thread https://github.com/notifications/unsubscribe-auth/AABALyoY4uCE0aAewKmFrmYQxrO-2Zqjks5rV8ztgaJpZM4K3Jkd .

-- Phil Smy - "http://www.philsmy.com" filmsite: http://philsfilmlinks.com twitter: http://twitter.com/psmy facebook: http://facebook.com/philsmy

[noorani786]

@hakanensari The small change you made didn't fix the issue. Actually, the exception occurs on line 38 on the FlatFileParser.rb and not line 57. To fix the issue, I ultimately had to write the content to a temp-file and then read it back. This apparently fixes any encoding issues.

[hakanensari]

@noorani786, @philsmy, could you please check if ☝️ fixes the encoding bug?

[noorani786]

@hakanensari Yes, that fixed the issue for me. Thank you!