task2-1_v2 apply BoW and TF-IDF on cord19 dtc

[Alaa-Ebshihy]

Use the document representation implementation to generated document representations for the dynamic test collections of the CORD 19 data set

put on mind also the calculation of the KD using these representations

[Alaa-Ebshihy]

First I need to apply on document collection and explore the effect using BoW This can be done by manually creating a correlation matrix between words using the vocab

[Alaa-Ebshihy]

After an experiment the correlation matrix is the same color

From noticing the vocab there are plenty of words which need to be cut out, there are two options to exploer:

• use Box plot to filter out for outliers
• use gensim library to filter most and least frequent words

[Alaa-Ebshihy]

Done on Friday 10.03.2023

• generated vocab vectors for whole set of test collections
• changed the tfidf and bow to gensim with a bit of filtering

Next to do

• trying to do for the correlation experiment but the kernel is dead --> probably we need to filter more for the words
• clean the representations since now the bow and tfidf are correlated with each other in the implementation

[Alaa-Ebshihy]

Done on Friday 17.03.2023

• generated KD as the norm of the cosine similarity of the vocab vector between two test collecion
• plotted the results but the value is very near
• I may try to check its correlation with results delta or find other way of KD

[Alaa-Ebshihy]

TODO as of 20.03.2023

• Use time series analysis to find significant point of change in the Result delta plot.

[Alaa-Ebshihy]

Update as of 21.03.2023

• used the ruptures to detect change points of time series for the RD
• Tried to collect the Robust collection using Pyterrier but the corpus is not there only the qrels and the topics
• The Robust collection need to be collected from the official web site after signing some forms (asked petra and probably she helpls)

TODO

• think of a design of file for result delta change points to save them so we can use to train a model using the features KD_norm, KD_MAX, KD_AVG to predict the result delta
• we have to construct training data for this one

[Alaa-Ebshihy]

Update as of 22.03.2023

• added functions to extract the result delta change points from a collection
• suggestion from tobias: can we use the whole cosine similarity vector as a feature vector instead of a single vector value ?! need to try out

TODO

• think how to make a training data set what are the splits
• verify detected RD is a good one

[Alaa-Ebshihy]

Reopen: some implementation need to be cleaned from the representation folder