Experiments on BERT-Large for baseline model creations

[HamedBabaei]

Initial tasks:

• [x] make wordnet finalized dataset
• [x] make geonames finalized dataset
• [x] make umls finalized dataset

We are about to test templates for datasets to come up with the best template for datasets. Templates are:

Wordnet templates:

• [x] 1. [sentence] . [A] POS is a [MASK].
• [x] 2. [sentence]. [A] part-of-speech is a [MASK].
• [x] 3. [sentence] . '[A]' POS is a [MASK].
• [x] 4. [sentence]. '[A]' part-of-speech is a [MASK].
• [x] 5. [A] POS is a [MASK].
• [x] 6. [A] part-of-speech is a [MASK].
• [x] 7. '[A]' POS is a [MASK].
• [x] 8. '[A]' part-of-speech is a [MASK].

UMLS templates: For all three datasets: [MEDICIN, USMODE, SNOMEDCT_US]

• [ ] 1. [sentence] . [A] in medicine is a [MASK].
• [ ] 2. [sentence]. [A] in biomedicine is a [MASK].
• [ ] 3. [sentence] . '[A]' in medicine is a [MASK].
• [ ] 4. [sentence]. '[A]' in biomedicine is a [MASK].
• [x] 5. [A] in medicine is a [MASK].
• [x] 6. [A] in biomedicine is a [MASK].
• [x] 7. '[A]' in medicine is a [MASK].
• [x] 8. '[A]' in biomedicine is a [MASK].

Geonames templates

• [x] 1. [sentence] . [A] is a [MASK].
• [x] 2. [sentence]. [A] geographically is a [MASK].
• [x] 3. [sentence] . '[A]' is a [MASK].
• [x] 4. [sentence]. '[A]' geographically is a [MASK].
• [x] 5. [A] is a [MASK].
• [x] 6. [A] geographically is a [MASK].
• [x] 7. '[A]' is a [MASK].
• [x] 8. '[A]' geographically is a [MASK].

For Geonames [sentence] we can use the more generic template that we designed. [NAME] is a place in [COUNTRY].

We are interested in this kind of template because of the following reasons:

• Template such as [A] is a [MASK] is a kind of general template that anyone can query from a search engine like google and here we want to use it as a query for knowledge graph to see whatsoever they have [A] knowledge or not.
• And we add sentences in the first place to inform the model that's the information we are talking and in this way, we are able o fine-tune our models for the better!
• These templates are appropriate for level 1 in UMLS and Geonames dataset because for level 2 we might need more tokens to be generated that that's the problem with BERT models and because of this we will move on with BART

Tasks are categorized into the following categories:

• [x] Define evaluation modules and structuring codes
• [x] Make datasets
• [x] Clean codes and make models in form of scrips
• [x] Model 1 is BERT-Large on 5 datasets, and 8 templates ~- [ ] Model 2 is Freq Model on 5 datasets, and 8 templates~

[HamedBabaei]

For the record (based on our discussion):

• NgramLM + BERT-Large is going to be our baseline
• The next model is T5
• I should consider the registration for ChatGPT API because it's going to be in our experiments as well!

[HamedBabaei]

Precision@k and Recal@k Evaluation Metrics:

Two quantities measure how well a search engine keeps this promise. Precision is the fraction of results that are relevant. Recall is the fraction of relevant results that are returned. In other words, a recall of 1 means the results include the whole truth, while a precision of 1 means the results include nothing but the truth.

[HamedBabaei]

Hi @jd-coderepos, The first results on WordNet Dataset are available here:

https://github.com/HamedBabaei/LLMs4OL/blob/main/results/WN18RR/01_BERT_Large_without_finetuning.log.txt

[HamedBabaei]

• Results are represented in the following directory: https://github.com/HamedBabaei/LLMs4OL/tree/main/results
• This issue has been referred and the only remaining section is generating sentences for UML sets which will be investigated in another issue
• Next step is testing T5 for modeling at all levels
• Also we decided to add conceptnet among our dataset after the finishing the project