search

epic-open-source / seismometer

AI model evaluation with a focus on healthcare
https://epic-open-source.github.io/seismometer/
BSD 3-Clause "New" or "Revised" License
156 stars 15 forks source link

Enhance handling of model outputs #31

Open gbowlin opened 1 week ago

gbowlin commented 1 week ago

We do a check if the scores are all between some value and 100, and if so we divide by 100 so that we get percent values.

We should consider removing this code, or making this an explicit option in config.

_Originally posted by @gbowlin in https://github.com/epic-open-source/seismometer/pull/30#discussion_r1653325281_

        if 50 < dataframe[score].max() <= 100:  # Assume out of 100, readjust
            dataframe[score] /= 100
diehlbw commented 1 week ago

Adding part of my response, I think this should be reconsidered in combination with a notebook template extending to new output types. My first thought being, we'll need a flag or enum in configuration.

The next in line for support may mean that Draft Text does this, though regression support might be implemented first.