Write script to read in sources, order them based on primary & secondary keyword matches and count. - Githubissues

wmgeolab / scope

5 stars 3 forks source link

Write script to read in sources, order them based on primary & secondary keyword matches and count. #162

Open devsaxena974 opened 1 year ago

sigaloid commented 1 year ago

Finished the ranking: https://github.com/wmgeolab/scope/commit/31ac0159f910ae4e59f6aa0fe536fc2f3a82dc75

Todo:

Integrate ranking with GDELT results.
- Currently the results from GDELT are not ranked in a meaningful way to us
Re-rank the results based off of the keyword rankings
- Fetch each result's content (Readability-lxml, readability, newspaper3k (obsolete?), Prof. Nwala's scraper, etc?)
- Rank with new rank function from primary keyword, secondary keywords, article texts
Do we save the full text in the database?