ghost
commented
9 years ago How is the value 'r' in the formula for the sense-feature matrix calculated?
For example given
Java#NP 4 Windows#NP, Linux#NP,...and
Windows#NP 1 Linux#NP, OS#NP,
Windows#NP 2 Console#NP,then Java#NP#4 and Windows#NP#1 have Linux#NP in common, so the value should be > 0 (if i got it right). I would expect something like common_words_in_both_clusters/(number_of_words_in_cluster_1 + number_of_words_in_cluster_2).
alexanderpanchenko
Motivation
We need to develop a prototype of the system that builds structured topics (creates a model) and is able to label new texts according to these topics. This prototype is supposed to have all minimal functionality of the system (input/output) and implemented with the most straightforward set of algorithms. The goal is to make initial validation of the idea and then improve the prototype gradually. In this step we do no evaluation which will be done later as well (during the "official" 6 month period reserved for writing the thesis).
Implementation
The prototype will build structured topics out of sense clusters i.e. it will cluster word senses which are provided as input. The system will rely on existing developments of the LT team in order to efficiently implement computation of a similarity graph of senses.
The overall pipeline of the prototype (to be implemented in Java/Scala or a mix of both):
Create a sense-feature matrix out of these files in the following format:
for instance
here 0.75 is 1/(r+1)^0.33, where r is the rank of cluster word from 1 to n
To be more precise, the system should generate the same output as this package: https://github.com/tudarmstadt-lt/noun-sense-induction
This includes two additional files:
sense-id<TAB>freqandcluster-word<TAB>freq.For each sense retrieve top 200 most similar senses i.e. calculate a similarity graph of senses. You will need to adapt this project to do so efficiently using Apache Spark framework: https://github.com/tudarmstadt-lt/noun-sense-induction-scala. To understand how it works read
The framework outputs a file in the format:
for each sense, it would generate 200 nearest neighbours.
For start with Spark just look at examples: http://spark.apache.org/
Cluster the graph of sense similarities using Chinese Whispers or/and Markov Chain Clustering algorithm. Use this implementation: https://github.com/johannessimon/chinese-whispers. Alternatively you can use this implementation: http://maggie.lt.informatik.tu-darmstadt.de/jobimtext/documentation/sense-clustering/
In your case, instead of words you will cluster word senses.
http://wortschatz.uni-leipzig.de/~cbiemann/pub/2006/BiemannTextGraph06.pdf
Write a classification module that would use the structured topics, being clusters of senses, to annotate text documents. The module should
To implement this module you should use ElasticSearch index. One topic would be one document, and then use an input document as search query. The retrieval system will return a list of documents (topics) according to their TF-IDF score.
https://www.elastic.co/guide/en/elasticsearch/guide/current/scoring-theory.html
https://lucene.apache.org/core/4_0_0/core/org/apache/lucene/search/similarities/TFIDFSimilarity.html