New and improved code

[minimalparts]

This is a completely revamped version of the linguistic-ey bits in the codebase. Specifically:

At the query stage

• No need for the tagger anymore. We just run on raw text.
• Streamlined version of the code. I have removed repeated uses of some functions and/or loading of the same data.
• Integrated code from the evaluation exercise with @languagerecipes. In particular, use of entropy over the query words.
• Updated all database-related scripts to reflect the new semantic space name (good bye wikiwoods...)

At the pears creation stage

• Added a 'demo' directory with some code to produce pears from Wikipedia categories.
• The demo directory contains actual pears in the users/ subfolder, which can be used for testing.
• Updated mkWordClouds.py so that the clouds reflect the topically salient words in the document.
• The profile of each user contains a measure of topical coherence as well as a word cloud reflecting his/her interests.

New semantic space

• I have used the first 50,000 entries of the *Don't coun't, predict' vectors of Baroni et al (2014), freely available at http://clic.cimec.unitn.it/composes/semantic-vectors.html. I have put the database version at http://clic.cimec.unitn.it/~aurelie.herbelot/openvectors.dump.bz2. (Note that the code to create pears still takes a .dm file. The .dm version of the space is at http://clic.cimec.unitn.it/~aurelie.herbelot/openvectors.dm.bz2.)

The demo pears are working well for me. But please test! Three topics are covered: cooking, video games and environmental issues. The app/ code is currently set to run locally over the pears in the demo folder. You will have to update your shared_pears_ids.txt in the app/ folder by doing something like

egrep "pear_id" $HOME/PeARS/demo/users/*/profile.txt |sed "s/profile.txt.pear.id:/ /" > shared_pears_ids.txt

(Sorry didn't get round to write a script for this. Perhaps someone feels inspired to do it :) )

[nandajavarma]

@minimalparts There is no issue with shared_pears_id.txt. I did something stupid there.
I am merging this PR to your development for easy merge to our organization repo.