Collecting Data Module - Githubissues

piatra / ssl-project

School assignment

MIT License

0 stars 0 forks source link

Collecting Data Module #3

Closed tiriplicamihai closed 8 years ago

tiriplicamihai commented 8 years ago

We need a way to build a training set.

The solution we agreed follows these steps:

come up with a list of search terms
search each term on google
For each link:
- get the age and gender using a service (like Alexa)
- open the page and get the text
- save the text in the database - I think we should have a different table for content with the URL as foreign key. We an consider limiting the size of the content.

piatra commented 8 years ago

Each demographic info (that from a different vendor) should be in it's own column. I gave it some thought and averaging the data doesn't make sense. What we can do is compare the accuracy using data obtained from vendor 1 vs vendor 2

piatra commented 8 years ago

Also todo:

remove the stop words when collecting the data from websites
remove stopwords when searching the terms