search

epic-open-source / seismometer

AI model evaluation with a focus on healthcare
https://epic-open-source.github.io/seismometer/
BSD 3-Clause "New" or "Revised" License
156 stars 15 forks source link

Add Content for Draft Text Model Evaluation #26

Open firste opened 3 weeks ago

firste commented 3 weeks ago

Is your feature request related to a problem? Please describe

A growing generative AI use case is related to drafting text for a clinician to review and edit prior to saving to the patient's record. Patient Medical Advice Requests is a well publicized workflow but there is also past research about other workflows including discharge summaries and letter generation.

Describe the solution you'd like

Develop a set of operational metrics like usage and edit distance. Also consider developing a set of semantic evaluation metrics like clinical similarity. We will also need to identify or develop a data set for testing and documentation purposes as well.

Describe alternatives you've considered

None, this is a feature that we would like to see built into seismometer.

Additional context

I am willing to own this development effort and am looking for collaborators to work with on its development. Please reach out to efirst@epic.com with your interest.

aurelienbono commented 2 weeks ago

Hello I want to work on this issue I would like to have the documentation @firste

firste commented 2 weeks ago

@aurelienbono Here is an RFC that I started for this change.