search

VowpalWabbit / vowpal_wabbit

Vowpal Wabbit is a machine learning system which pushes the frontier of machine learning with techniques such as online, hashing, allreduce, reductions, learning2search, active, and interactive learning.
https://vowpalwabbit.org
Other
8.46k stars 1.93k forks source link

normal feedback is returned to sender when daemon receives a malformed example #1597

Open matanox opened 6 years ago

matanox commented 6 years ago

A malformed (Contextual Bandits) context sent to VW will only log to the log file, but the sender will receive a normal response and have no way to know they are providing a malformed context string to the daemon. To reproduce on the comman-dline:

$ echo "| a: 0:300" | netcat localhost 26542

Assume in the above case, the sender intends to obtain a distribution back from the daemon, but the context they provide is, as easily seen above, malformed.

An error will be logged, but a distribution will be returned as if it is a normal message. This will deeply hinder production integration workflows that expect "plain" software standards on the socket↔daemon channel, and can be a source of trouble for production code architectures interacting with the daemon. The solution should probably (or perhaps) be to introduce a special response replacing the normal one.

Confirmed on 8.5 and 8.6.1 as released (in both cases model used in the daemon trained on 8.5).

Please let me know if you can indeed reproduce...

JohnLangford commented 5 years ago

This seems very believable as a bug.

The core issue here is that there is no way for the daemon mode to report errors.

I'm not sure what to do about this. We aren't currently using the daemon mode, partly because more advanced communication protocols avoid this sort of failure.