openmpf / openmpf-evaluation

Framework for generating metrics and evaluating OpenMPF component algorithms
Other
1 stars 0 forks source link

Evaluate Components in Workflow Manager via REST API #8

Open hhuangMITRE opened 1 year ago

hhuangMITRE commented 1 year ago

Updated:

Jeff suggested an improvement on the current issue.

In addition to Java components, we would also want a way to automatically submit job runs via the REST API to the Workflow Manager.

This capability would resolve our need to create a basic CLI runner for Java components and would be beneficial for evaluation purposes in the long run (i.e. translation pipeline).

At minimum, we'd need a script or notebook (can be in any preferred language) that allows users to communicate with the WFM to:

Old Issues below: ~~We need a script or library that can run and process and output MPF JSONS for Java-based components. This would allow us to better assess language detection capabilities with Tika Text Detection.~~

At minimum this script would need to

1. Initialize Tika Text Detection component with input MPF Job JSON or dict. 2. Process jobs for a given file or directory containing media of interest. 3. Store MPF output JSONs (given an output directory, store with the same general file base name) and log general runtime.

Optional requirements are: 1. Ability to interface and submit jobs to active MPF component deployments (in Docker).