User "word" matrix

[Pernilleww]

Use keywords from keywords, title, ingress

ADV: maybe named-entity-recognition

[jonasrodningen]

Kilder:

handbook.p119 (chap4)

Novel research works have introduced semantic techniques that shift from a keyword-based to a conceptbased representation of items and user profiles.

.... men vi starter nok med keywords, siden vi har keywords.

[jonasrodningen]

TODO-list

• [ ] ~make sure documentIDs exist for all articles...~

  • 32% of all non-homepage-events lacked docID.. many of these are category-page events

• [x] read in content-files and fields (see comment regarding fields)

• [x] make a keywords corpora (all keywords in dataset) - this is also a big weakness of our approach (can't handle unseen keywords, at this point, but one approach could be expanding the matrices when new ones appear in real life)

• [x] make a list of documentID <-> keywords

• [x] make user-keywords matrix

• [x] adjust weighting between categories and keywords. Categories should be lowered (e.g. x0.25), keywords multiplied up (x4)

• [x] at this point basic CF should be possible

Getting advanced:

• [ ] look for ideas in RS handbook chapter 4, may also find much good material for report
• [ ] may want to experiment getting tf-idf scores per user (not sure if that's the way, but my goal was to find which keywords are REALLY significant to them (?)) this could be added as extra feature. Alternatively tf-idf can be done on keywords per document to find how descriptive the keywords are for this document.. and this can help build a more defined user profile.
• [ ] if CF could use weights in the sim-function such that some columns have greater importance > we could incorporate info like isSubscriber, favoriteMainCategory

[jonasrodningen]

branch content_based

[jonasrodningen]

field=

id publishtime description teaser keyword #<[]> kw-classification kw-category kw-concept kw-company kw-entity #someway dirty: e.g. norge and norges are two different entities kw-location title score #some type of popularity for the article, read more about it