EM Models -- Improvements / Model Testing / Evaluations

[evamaxfield]

Possible Improvements

Repo <-> Paper

• currently uses the short context all-mpnet-base-v2 model which allows for only 384 token sequences, waiting on https://github.com/UKPLab/sentence-transformers/issues/2400 to use the [https://huggingface.co/jinaai/jina-embeddings-v2-base-en](jina embedding models). In general though, different embedding models should be tested.
• currently only creates coefficients using paper_abstract_embedding * repo_readme_embedding but this just like the github user <-> author matching model, we should do some pairwise comparisons:
  • repo name <-> paper title
  • repo description <-> paper abstract
  • repo readme <-> paper abstract
  • repo description <-> paper TLDR
  • repo readme <-> paper TLDR
  • COMPUTED: use the github user <-> author matching model to get the "percentage_of_matched_devs_to_authors" to add as a single feature

Model Testing and Evaluations

• both of the EM matching models need to be trained as fine-tuned version of LLMs (hopefully with Isaac's help)
• all of the models should be investigated -- primarily this means that features should be added and removed to see what helps and what doesn't commonly this forms a table in the final paper where columns are the additions of different features, and rows are different model architectures with a single reported statistic for each row x column value (usually F1 in our classification case)
• ask nic: should we test prompted LLMs as part of this work?

Repo <-> Paper

• would be nice to add a dataset that is more focused on analysis work -- I already got the "biomed jupyter notebooks" dataset processed so theoretically we could annotate that for more varied examples.
• currently our dataset composition for model training is the following value counts of topics:

  Computer Science                  1522
  Engineering                        542
  Mathematics                        253
  Physics                            135
  Environmental Science              118
  Medicine                            92
  Biology                             85
  Linguistics                         57
  Materials Science                   32
  Geography                           23
  Chemistry                           21
  Geology                             20
  Psychology                          15
  Business                            15
  Education                           10
  Sociology                           10
  Economics                            9
  Art                                  8
  Law                                  7
  Political Science                    7
  History                              4
  Agricultural and Food Sciences       2

CS is heavily over-represented and so adding more social science and bio pairs would be incredibly useful.

[tomaarsen]

Hello!

The Sentence Transformers release has been published! You can now use

pip install -U sentence-transformers

to install v2.3.0.

P.s. I love the interactive graph!

• Tom Aarsen

[evamaxfield]

Thanks for the update and all the work you do!