Developing eval_utils module

[jarumihooi]

Because

As a reformat task, and for future development of more evaluations, it should be considered if a common evaluation utilities module should be created.

• [ ] where should that module reside? Should it be a subsection of the clams_utils?

That module could contain commonly used code such as:

• goldretriever
• data extraction/preprocessing from golds and preds
• guid/file matching between golds and preds.
• common accuracy metrics
• etc

Note, not every evaluation will be similar, and thus some may use different metrics. As such, its likely that printing is unlikely to be universal enough for modulization.

The environment for CLAMS so far seems to require:

• ? clams-python
• mmif-python
• clams_utils for goldretriever, which is used in each eval, and in the annotation for NEL. The goldretriever code seems to be commonly used. Is it better to place certain codes in utilities? What is the organizational method?

Done when

No response

Additional context

No response

[keighrim]

Also related to #41, I think the most straightforward way of doing this will be providing a PyPI distro (clams-aapb-eval package hereinafter, working name for now) that includes all the utility modules and Evaluator superclass (ABC), so that evaluator developers can just pip-install and start writing. Then, conveniently, since I already set up clams-utils repo for daily PyPI release, I'm very tempted to use that to hold an additional clams-aapb-eval package . But there could be some issues with that approach, namely;

1. there's no inherit linkage between this repo and clams-utils repo, so we need a very clear documentation for this arbitrary use of separate repositories. As I stated also in https://github.com/clamsproject/clams-python/issues/147, I don't like this arbitrariness / segregation of clearly related components.
2. I'm not seeing a big chance that parts of clams-aapb-eval package is commonly used in other part of the whole clamsproject. Hence ,no need to "factorize" the eval package.
3. clams-utils is auto-released every day. If there's any problem in, for example, the clams.utils.aapb.eval package, even if it's a very minor bug, it will take time to do the fix and propagate the fix over PyPI.

Combining the above reasons, I firmly believe the package should reside in this repository. This will imply

1. we put a aapb_eval package in the project root
2. all other evaluator subdirectories should now be python packages (__init__.py)
3. all invocations of the evalute.pys should now through python -m subdir.evaluate <more> <flags> <as> <defined>
4. when evaluator developer finds an issue with the superclass, they can directly mend and push the fix

[jarumihooi]

Conversation of where this repo is to be placed had two different outcomes, can the comments above this be reviewed and updated plus any other details of this development?

On Mon, Jan 29, 2024 at 11:51 AM Keigh Rim @.***> wrote:

Assigned #42 https://github.com/clamsproject/aapb-evaluations/issues/42 to @jarumihooi https://github.com/jarumihooi.

— Reply to this email directly, view it on GitHub https://github.com/clamsproject/aapb-evaluations/issues/42#event-11630563773, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALHUBDVULEDP6DHK3GO6253YQ7HRPAVCNFSM6AAAAABA7L244CVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJRGYZTANJWGM3TOMY . You are receiving this because you were assigned.Message ID: @.*** com>

[keighrim]

Conversation of where this repo is to be placed had two different outcomes,

I don't think I follow this. This repo doesn't go anywhere else...?