CLEF-NewsREEL evaluation environment error - [datastream] Evaluator: no recommendations received for 60 secs, stopping.

[cgolian]

Hello,

I set up evaluation environment for CLEF-NewsREEL Task 2 according to this tutorial

However, when I run the last command from this tutorial: sudo ./idomaar.sh --comp-env-address http://192.168.56.1:8088 --data-source newsreel-test/2014-07-01.data.idomaar_1k.txt --newsreel I get this output from datastream: [datastream] Evaluator: no recommendations received for 60 secs, stopping.

Has anybody any idea what could cause this problem? My computing environment is definitely running. Here is the result of running vagrant up (computing environment)

And this is the output I get when I try to run the evaluation.

Thanks in advance.

[andreas-dai]

Hi,

if there are no recommendations received, there might be a problem with the recommender service. In order to check whether the recommender http endpoint is reachable, you could send a test message: wget http://192.168.56.1:8088/ The recommender should respond with Visit <h3><a href="http://www.clef-newsreel.org/">CLEF NewsREEL Challenge</a></h3>

http-get messages are only used for testing the recommender. In order to test whether the recommender also handles recommendation requests correctly, you could send a recommendation request using something like: curl -X POST -d "type=recommendation" -d "body=<AValidJsonLineFromTheLog>" http://192.168.56.1:8088

Kind Regards Andreas

[andreas-dai]

[cgolian]

Thank you for your quick response,

wget seems to be working. However, when running this curl I get: unknown MessageType: null from the recommender

[andreas-dai]

It seems that the recommender server is up and running. The response says that there is a problem with the data format of the test message. That is probably because I confused the names of the variables in the post request. I attach a debug message, that works for me (note, that the recommender will respond with an empty recommendation list [] if no items in the requested domain are known - the recommender collects the valid itemIDs in impression messages) Finally we now know, that the recommender server is running - but we do not know why no answers are received. example_command.txt

[cgolian]

Could please attach your post request? It seems like you attached mine by mistake.

[andreas-dai]

You are right. The json object must be provided as Parameter "properties". idomaar-test-message.txt

[cgolian]

So,

When I run the curl from 'idomaar-test-message.txt' the computing environment responds with empty recommendations list: {"recs": {"ints": {"3": []}}} (as it should).

However, when I run the idomaar script (with the same computing environment) I still get: INFO [datastream] Evaluator: no recommendations received for 60 secs, stopping. INFO [datastream] Exception in thread "Thread-6" java.lang.ArithmeticException: / by zero INFO [datastream] at eu.crowdrec.idomaar.evaluation.StreamingEvaluator.run(StreamingEvaluator.java:325)

Basically the computing environment responds to curl but not to the datastream. Any ideas @andreas-dai ?

Thanks in advance.

[andreas-dai]

A fix to the master branch has been pushed. (https://github.com/crowdrec/idomaar/commit/71b7f1372e744d2922ce91dc8245e11f9176463d). With the fix, we get back some meaningful evaluation results when running the CLEF tutorial.