Weird characters showing up in grocery names

wangyif2 / GroceryGo

A consolidated Android application that lets you view sales information on grocery items of major grocery items on the go, right where you need them

3 stars 1 forks source link

Weird characters showing up in grocery names #212

Open yancheva opened 11 years ago

yancheva commented 11 years ago

Only for Loblaws, No Frills and Sobeys

check how the crawler receives the raw string
check how it is stored in db
check if db stores it differently than before

yancheva commented 11 years ago

The character that is not being displayed properly is Unicode hex value \xae (registered trademark symbol: http://www.fileformat.info/info/unicode/char/ae/index.htm).

The crawler correctly stores "\xae" in the database, and uses a UTF-8 connection string.

The database table (Grocery) has a utf-8 character set.

The servlet already displays it incorrectly (although this could be just a UI issue): http://grocerygo.elasticbeanstalk.com/UpdateGroceryInfo

TODO: check JSON on the client

yancheva commented 11 years ago

Update: this is an issue with the server python setup. The database, the client, and the python crawler all have the correct encoding set up. When you run the crawler from local machine to write to the server db it works.

TODO: Re-configure the ec2 machine encoding with Robert (after Sep 12)