Closed yonromai closed 11 months ago
Nicely organized @yonromai 👍. That notebook is a big help.
Do you have some examples you can share before scaling up?
See notebook section 2.1
Looking at a prompt and a corresponding completion doesn't turn up anything immediately concerning for me. Those classifications from GPT4 make sense for the most part.
Looking at some of the misclassifications didn't surface anything alarming either. I break them into 3 cases:
02-disease-root
not 03-disease-area
02-disease-root
(or maybe 03-disease-area
.. close call), not 01-disease-subtype
; I agree w/ GPT4 in calling this a "low" precision term given only the label/description02-disease-root
not 03-disease-area
high
precision labels from GPT4 are a great choice given only our definition of the task and the term details, yet the existence of descendants for this term in EFO would dictate it's probably better as a medium
term.Overall, I would say I agree with ~80% of the GPT4 classifications in these misclassified examples. To be clear, this does not suggest that 80% of those true labels are wrong -- making this classification using only labels + descriptions is a somewhat orthogonal task to doing it within the confines of EFO's structure (i.e. lots of these fall into case # 3 above).
I do think we can add these misclassified examples to the prompt though:
- id: Orphanet:75325
label: Osteosclerosis - ichthyosis - premature ovarian failure
definition: NA
precision: high
- id: EFO:0007237
label: dipetalonemiasis
definition: A filariasis that is a zoonotic infection caused by the nematode of the genus Dipetalonema.
precision: high
- id: MONDO:0013742
label: familial mesial temporal lobe epilepsy with febrile seizures
definition: NA
precision: high
- id: MONDO:0002321
label: sensory peripheral neuropathy
definition: Inflammation or degeneration of the sensory nerves.
precision: medium
- id: EFO:1000646
label: papillary carcinoma
definition: A malignant epithelial neoplasm characterized by a papillary growth pattern.
precision: medium
- id: EFO:0011057
label: neurotoxicity
definition: Toxicity that causes injury to the central or peripheral nervous system or damages its function.
precision: low
That would increase our few-shot example count from 9 to 15.
Ditto - awesome work @yonromai!
Most high distance from true label nodes are of class 01-disease-subtype
So that means, GPT would frequently assign low
precision to something that is actually high
precision? afaiu that would fall into the 2nd bucket in Eric's misclassification classification. Here's one example of that from the notebook:
{'dist': 6, 'efo_definition': 'A viral infectious disease that results_in infection in ' 'sheep and rarely humans, has_material_basis_in Louping ' 'ill virus, which is transmitted_by sheep tick, Ixodes ' 'ricinus. The infection has_symptom lethargy, has_symptom ' 'muscle pains, has_symptom fever, and has_symptom focal ' 'neurological signs.', 'efo_id': 'EFO:0007348', 'efo_label': 'louping ill', 'precisions': ['low', 'low', 'low'], 'true_label': '01-disease-subtype'},
I'm surprised GPT would assign low
precision here, just high level based on the description it, it does sound like high
precision disease, given the details. It would be interesting to have the reasoning/CoT for this case, for debugging purposes. I guess we do see that in the per class scores:
Thanks for those btw ^ 🙏
Please tell me what you think but it might be worth doing another pass at CoT prompts at a later time (i.e. after the workshop next week
Thanks for giving that a try, do you happen to have the explanations available somewhere? Like for the EFO:0007348
above. WRT further prompt tuning (after the workshop):
(I'd like to rebase this notebook on an upcoming change so I'll merge this PR and open a new one soon with updates)
( All "sections" references below belong to the "GPT-4 Experiments notebook", I wasn't able to hyperlink it :( )
This PR addresses the following comments:
@ravwojdyla
See notebook section 1.2
See notebook section 4
(Gist: Adding CoT didn't change the performance much. Note that I changed your prompt a bit - see
prompts/rav_cot_precision_v1.txt
) Please tell me what you think but it might be worth doing another pass at CoT prompts at a later time (i.e. after the workshop next week)I ended up cutting that corner and postponing this idea since it was causing significant changes to the way the prompts are currently injected/constructed.
@eric-czech
See notebook section 2.1
See notebook section 1.3 (and similarly 2.3)
Additionally, I wanted to see the impact of varying the number of completions ("choices") and see the impact on MAE.
(gist: Not surprisingly the more choices the better performance, although not drastic).