search

dalab / dissolve-struct

Distributed solver library for large-scale structured output prediction, based on Spark. Project website:
http://dalab.github.io/dissolve-struct/
Apache License 2.0
17 stars 5 forks source link

allow approximate decoding oracles #2

Open martinjaggi opened 10 years ago

martinjaggi commented 10 years ago

allow the oracle to return crappy labels (test with intentionally messing up the answer e.g.)

possible criterion for a good enough answer (see also cache, issue #1) gamma >= 0.5 * gamma_fix

in the longer term, our interface for the oracles should allow us, i.e. the solver to call the oracle again with a flag requiring higher accuracy (maybe higher and higher etc). if the resulting gamma is never positive, then we stop optimizing and say that the oracle was not good enough.

some of the optimization based oracles in pystruct have this functionality as well: pystruct.github.io

tribhuvanesh commented 9 years ago

A warning is now thrown to stdout in case a decoding was weak.

In future, the oracle interface (model: StructSVMModel[X, Y], yi: Y, xi: X, decodingAccuracyRequest: Double) => Y needs to be supported. Wherein, decodingAccuracyRequest \in [0.0, 1.0]. 0.0 implies the decoded labels so far using the given oracle is completely crappy (since gammas are consistently negative). 1.0 implies that the oracle is producing gammas consistently above a certain threshold (say 0.8+).