Remove duplicate articles due to multiple authors

PoonLab / tragula

Mapping networks of shared research expertise by word embedding analysis

MIT License

0 stars 0 forks source link

Open ArtPoon opened 6 days ago

ArtPoon commented 6 days ago

Some articles have two or more authors from the same faculty or department.
This is causing columns representing the same document to appear more than once in the term-document matrix
We should screen for this by removing duplicate columns (it is exceedingly unlikely that two documents will have exactly the same frequency distribution of words out of 10,000 words by chance)

ArtPoon commented 6 days ago

I don't think this is really going to skew topicspace because the word vectors are really long (total number of documents)...