search

explosion / spaCy

💫 Industrial-strength Natural Language Processing (NLP) in Python
https://spacy.io
MIT License
30.15k stars 4.4k forks source link

Allow evaluate CLI to take metrics from the command line arguments #8519

Open narayanacharya6 opened 3 years ago

narayanacharya6 commented 3 years ago

Currently the evaluate CLI has fixed list of metrics which are reported as part of the evaluation output. This is limiting when one wishes to make note of other metrics/scores that are part of custom pipeline components in their evaluation outputs. My suggestions is to allow users to pass a list of metrics that they wish to add to the existing set of metrics that are part of the evaluate CLI script.

Which page or section is this issue related to?

https://github.com/explosion/spaCy/blob/master/spacy/cli/evaluate.py#L79-L97

adrianeboyd commented 3 years ago

Hi, yes, we definitely need to think about how to make this more flexible for new/custom components.

In the meanwhile, the CLI scripts should be pretty easy to copy and adapt for your own needs. You can copy spacy/cli/evaluate.py, convert the relative imports into absolute imports with spacy.whatever and run it as a standalone script.

narayanacharya6 commented 3 years ago

Yes, that is my current approach to work around it :) If you guys have anything in mind of how you'd like to see this, we can discuss that and I'm happy to submit a PR if we decide that the change is good.

My naive approach would be to take in a python-style dict as a string from the CLI, evaluate that string as a dict (maybe some of the code from config parsing can be helpful here if we don't want use the ast.literal_eval, etc.?) and simply update the default dictionary of metrics that we already have.