No README in data-preparation

[avacariu]

I'm trying to figure out how to set this project up so I can work on it, but I can't figure out how to do the data-preparation step. There is the dataPrepration.sh script that seems like it's what I want to run to generate the data, but I'm not sure of everything I need to install before running this script.

I can figure out the Stanford dependencies (since they're easy to Google), but I can't figure out what 'UIUC SRL' is and where I can get it.

[anoopsarkar]

Running the data prep is too tedious. Better to use an existing json file that is the end result of the data prep module and start from there.

Then follow the instructions in:

https://github.com/sfu-natlang/lensingwikipedia/blob/master/website.md

See if you can login to cs-champ.cs.sfu.ca. You can get the json file from a subdirectory with the latest date in /var/www/html/data

[avacariu]

I've followed all the instructions in https://github.com/sfu-natlang/lensingwikipedia/blob/master/website.md and I have the backend kind of working[1], but I'm having trouble getting the frontend set up. The result of all the instructions is that the index.html file is served fine, but I get a random "-e" in the CSS and JS includes:

<link rel="stylesheet" type="text/css" href="-e lib.css">
<script src="-e lib.js"></script>

I've tried removing the -e manually but all I get is the top bar of the site; the rest is blank. I get the following console errors.

SyntaxError: illegal character lib.js:20
ReferenceError: $ is not defined 

Did I happen to miss a step somewhere?

[1] Error I'm having with backend is that it's throwing errors saying "KeyError: "No field named 'referencePoints'""

[theq629]

What environment are you building on? The makefile may currently assume GNU tools, so the -e in output looks like some tool (maybe just echo) is interpreting that as output instead of an option. The syntax error is probably something similar. You should try building the devel version to get more specific console errors.

[avacariu]

I'm building it on a Ubuntu 14.04 server so everything should be GNU.

I'll try the devel version.

[theq629]

Does Ubuntu run shell scripts with a strict sh interpreter (dash maybe) instead of bash? If it's using a built-in echo command that would also explain the difference.

[avacariu]

It uses bash unless otherwise specified. I just tested a simple script and it executes echo -e properly.

I built the devel version and it works fine. The only error I'm getting for that is the illegal character error but it says it's within d3.geo.projection.v0.min.js. It's probably related to the fact that the character encoding's not declared in the HTML file (since it contains non-ASCII chars).

[theq629]

Yes, the d3 files need to be served as utf-8 because they use non-ASCII variable names. I think setting encoding in the html file didn't fix that, although maybe I'm misremembering.

[avacariu]

Alright, I fixed the utf-8 issue, but I'm still getting a KeyError: "No field named 'referencePoints'" error on the backend. Any idea why that'd be?

Full log:

request at 2014-09-24 16:46:48.478455
handling view "7" of type "countbyreferencepoint": generating view
generating field counts for fields: referencePoints
whoosh search results: <Top 83521 Results for Every(None, boost=1.0) runtime=0.442426919937>
error while generating count views:
Traceback (most recent call last):
  File "/var/www/html/checkouts/20140920/domains/wikipediahistory/backend/queries.py", line 234, in generate_views
    self.generate_field_counts(response, field_count_views, whoosh_query)
  File "/var/www/html/checkouts/20140920/domains/wikipediahistory/backend/queries.py", line 142, in generate_field_counts
    if field in hit:
  File "/usr/local/lib/python2.7/dist-packages/whoosh/searching.py", line 1513, in __contains__
    or self.reader.has_column(key))
  File "/usr/local/lib/python2.7/dist-packages/whoosh/reading.py", line 901, in has_column
    coltype = self.schema[fieldname].column_type
  File "/usr/local/lib/python2.7/dist-packages/whoosh/fields.py", line 1202, in __getitem__
    raise KeyError("No field named %r" % (name,))
KeyError: "No field named 'referencePoints'"
not caching due to error

Thanks for your help, btw.

[theq629]

That looks like the index doesn't have reference points. If you built the index yourself, did you run the cluster program to add them?