Prevent errors and give sensible results when people enter text with Unicode

[mialondon]

Something for first thing in the morning...

[rlskoeser]

I did some work on this - currently on masterfix branch (I'll get that resolved as soon as I can).

Easy way to test - go to a wikipedia page for other languages and cut & paste.

The site doesn't totally break on unicode/non-english input (even added some simple language detection to use language-appropriate stopwords when we can), but there is an issue if the code selects a unicode search term and pass it to bibs -- either we need to filter out unicode from search terms (hacky/quick fix) OR fork the bibs repo and correct it to support unicode search terms.

[moltude]

I think the quick fix is the way to go between now and noon. Are you thinking using something like unicodedata.normalize() to convert those characters?

[erose]

Relevant stackoverflow (?) http://stackoverflow.com/questions/2365411/python-convert-unicode-to-ascii-without-errors

On Fri, Aug 2, 2013 at 7:58 AM, Scott Williams notifications@github.comwrote:

I think the quick fix is the way to go between now and noon. Are you thinking using some thing unicodedata.normalize() to convert those characters?

— Reply to this email directly or view it on GitHubhttps://github.com/chnm/petulant-adventure/issues/50#issuecomment-22001362 .

[mialondon]

Check when deployed and close if ok.

[moltude]

In my testing this looks fixed so I'm closing this ticket.

Thanks guys!