it turns out we were already logging to papertrail, so now we get logs like
2017-09-13 00:38:37,273 - INFO - allennlp.service.server_sanic - prediction: {"model": "semantic-role-labeling", "inputs": {"sentence": "play the top song by billy joel released in nineties for his play in winter"}, "cached": false, "outputs": {"verbs": [{"verb": "play", "description": "[V: play] [ARG1: the top song by billy joel] released in nineties for his play in winter"}, {"verb": "released", "description": "play [ARG1: the top song by billy joel] [V: released] [ARGM-TMP: in nineties] [ARGM-PRP: for his play in winter]"}]}}
2017-09-13 00:38:45,282 - INFO - allennlp.service.server_sanic - prediction: {"model": "semantic-role-labeling", "inputs": {"sentence": "Did Uriah honestly think he could beat the game in under three hours?"}, "cached": false, "outputs": {"verbs": [{"verb": "Did", "description": "[V: Did] Uriah honestly think he could beat the game in under three hours ?"}, {"verb": "think", "description": "Did [ARG0: Uriah] [ARGM-ADV: honestly] [V: think] [ARG1: he could beat the game in under three hours] ?"}, {"verb": "could", "description": "Did Uriah honestly think [ARG0: he] [V: could] beat the game in under three hours ?"}, {"verb": "beat", "description": "Did Uriah honestly think [ARG0: he] [ARGM-MOD: could] [V: beat] [ARG1: the game] [ARGM-TMP: in under three hours] ?"}]}}
2017-09-13 00:38:51,672 - INFO - allennlp.service.server_sanic - prediction: {"model": "machine-comprehension", "inputs": {"passage": "The Matrix is a 1999 science fiction action film written and directed by The Wachowskis, starring Keanu Reeves, Laurence Fishburne, Carrie-Anne Moss, Hugo Weaving, and Joe Pantoliano. It depicts a dystopian future in which reality as perceived by most humans is actually a simulated reality called \"the Matrix\", created by sentient machines to subdue the human population, while their bodies' heat and electrical activity are used as an energy source. Computer programmer \"Neo\" learns this truth and is drawn into a rebellion against the machines, which involves other people who have been freed from the \"dream world.\"", "question": "Who stars in The Matrix?"}, "cached": false, "outputs": {"best_span_str": "Keanu Reeves, Laurence Fishburne, Carrie-Anne Moss, Hugo Weaving, and Joe Pantoliano"}}
2017-09-13 00:38:55,069 - INFO - allennlp.service.server_sanic - prediction: {"model": "textual-entailment", "inputs": {"premise": "If you help the needy, God will reward you.", "hypothesis": "Giving money to the poor has good consequences."}, "cached": false, "outputs": {"label_probs": [0.289307564496994, 0.1799432337284088, 0.5307492017745972]}}
if you search papertrail for e.g. "sanic - prediction" you should just get these lines. I'll create a script to collect and analyze these programatically.
ok, this is done in
https://github.com/allenai/allennlp/pull/297 and https://github.com/allenai/allennlp/pull/302
it turns out we were already logging to papertrail, so now we get logs like
if you search papertrail for e.g. "sanic - prediction" you should just get these lines. I'll create a script to collect and analyze these programatically.