search

seomoz / qless

Queue / Pipeline Management
MIT License
294 stars 76 forks source link

InvalidByteSequenceError #208

Open pdelanauze opened 9 years ago

pdelanauze commented 9 years ago 
Encoding::InvalidByteSequenceError at /queues/do-it/running
"\xC3" on US-ASCII

file: common.rb location: encode line: 148

Just ran into this , i'm storing some html in those jobs, which the json parser doesn't seem to like ..

evanbattaglia commented 9 years ago

I was able to work around this in: https://github.com/seomoz/qless/tree/evans_encoding_error_workaround https://github.com/seomoz/qless/commit/762f7f68c7e9637a7cf7b36c02deea61f31d4032 It was at least good enough for me to run a script to reenqueue my failed jobs.

evanbattaglia commented 9 years ago

Update: for me this was caused by Ruby's default external encoding being "US-ASCII" instead "UTF-8", which was in turn caused by Ruby running in a docker container. The data jobs were valid UTF-8 (perhaps the strings had UTF-8 encoding when putting the job because the string originally came from a UTF-8 web page? not sure). The solution for me was to add the following to my Dockerfile:

ENV LANG C.UTF-8
ENV LANGUAGE en_US:C
ENV LC_ALL C.UTF-8

Then Ruby inside the container gives Encoding.default_external as UTF-8.