as6520
commented
2 years ago Yes, with a few caveats. We would have to answer a few questions about the deployment scenario like 1) How are the interactions with the API in deployment different from the current evaluation? This includes questions like how would the data be provided to the harness, how would we post results, would we get metadata in deployment too, and would the deployment still provide data in mini-batches. 2) Would the protocol have reduced/additional responsibilities, if so what would be those responsibilities?
Along with questions, the agent providing EVM/class scores has an internal dependency built into them for world change. This allows us to suppress FP novelty detection before we predict that the world has changed. In a deployment scenario where we don't estimate distribution change, this dependency would have to be removed
Would it be possible to have standalone agents, already trained in our core system, but can run on new data and give EVM/Class scores? The goal would hopefully be able to use the trained networks in a deployment system (without updating or world change).