Evaluation test

[davidemalagoli]

Hi @andras-sereny I was trying to integrate the spark-evaluation module using mahout example and zeromq interface. What I've seen running the idomaar-demo is the following error: INFO [bg:reco-manager] 2015-07-14 14:09:29,295 (PollableSourceRunner-KafkaSource-r1) [INFO - eu.crowdrec.flume.plugins.interceptor.IdomaarRecommendationInterceptor.intercept(IdomaarRecommendationInterceptor.java:111)] Received recommendation [{} [{"id":"1392170","rating":"10.0","rank":"1"},{"id":"1649419","rating":"10.0","rank":"2"},{"id":"975645","rating":"9.0","rank":"3"},{"id":"1623205","rating":"8.715197","rank":"4"},{"id":"2023587","rating":"8.6278","rank":"5"}]] INFO [bg:reco-manager] 2015-07-14 14:09:29,296 (PollableSourceRunner-KafkaSource-r1) [INFO - eu.crowdrec.flume.plugins.interceptor.IdomaarRecommendationInterceptor.intercept(IdomaarRecommendationInterceptor.java:121)] Sending event [recommendation 47 1362093742 {"reclen":5} {"subject":"user:28"} 19 {} [{"id":"1392170","rating":"10.0","rank":"1"},{"id":"1649419","rating":"10.0","rank":"2"},{"id":"975645","rating":"9.0","rank":"3"},{"id":"1623205","rating":"8.715197","rank":"4"},{"id":"2023587","rating":"8.6278","rank":"5"}]] INFO [bg:reco-manager] 2015-07-14 14:09:29,296 (PollableSourceRunner-KafkaSource-r1) [INFO - eu.crowdrec.flume.plugins.interceptor.IdomaarRecommendationInterceptor.intercept(IdomaarRecommendationInterceptor.java:75)] Received end of recommendation file INFO [bg:reco-manager] 2015-07-14 14:09:29,312 (PollableSourceRunner-KafkaSource-r1) [INFO - eu.crowdrec.flume.plugins.interceptor.IdomaarRecommendationInterceptor.intercept(IdomaarRecommendationInterceptor.java:78)] Sent 'FINISHED' to orchestrator, waiting for reply ... INFO [datastream] Evaluator: no recommendations received for 60 secs, stopping. INFO [datastream] Exception in thread "Thread-6" java.lang.ArithmeticException: / by zero INFO [datastream] at eu.crowdrec.idomaar.evaluation.StreamingEvaluator.run(StreamingEvaluator.java:325) INFO [datastream] at java.lang.Thread.run(Thread.java:745) INFO [datastream] INFO [datastream] Evaluation results INFO [datastream] ================== INFO [datastream] 69950 [ProducerSendThread-evaluator-producer] INFO kafka.client.ClientUtils$ - Fetching metadata from broker id:0,host:192.168.22.5,port:9092 with correlation id 0 for 1 topic(s) Set(output)

note that we have changed the output data adding info about duration (see https://github.com/crowdrec/idomaar/wiki/DATA-FORMAT) but it should not be the cause cause the problem was present before the modification.

Did you see this error behaviour before? It seams to me that the interceptor is sending the correct message perhaps there is some misconfig in topic names?

[davidemalagoli]

Hi @andras-sereny, I've made some minor modification (added new flume) and added the creation of the recommendation-result topic but it still doesn't work. Do you have time to have a look on that before september?

[andras-sereny]

I hope to have some time next week. Will take a look.

[davidemalagoli]

Hi @andras-sereny, we've modified the classpath for flume 1.6 and now the full process seams to work, I'm now getting a correct recommendation results in /tmp/recommendations, like the following:

recommendation 40 1362093742 {"reclen":5, "expected": {"ratings": [{"rating":1, "subject": {"type": "user", "id": 2}, "subject": {"object": "movie", "id": 3} } ]}} {"subject":"user:7"} 55 {} [{"id":"1702439","rating":"7.5554733","rank":"1"},{"id":"1623205","rating":"7.3246436","rank":"2"},{"id":"1024648","rating ":"7.1299243","rank":"3"},{"id":"1351685","rating":"6.6666665","rank":"4"},{"id":"1371111","rating":"6.464102","rank":"5"}] recommendation 41 1362093742 {"reclen":5} {"subject":"user:10"} 18 {} [{"id":"1840417","rating":"8.0","rank":"1"},{"id":"1024648","rating":"7.0" ,"rank":"2"},{"id":"1790885","rating":"7.0","rank":"3"},{"id":"1907668","rating":"7.0","rank":"4"},{"id":"1235522","rating":"7.0","rank":"5"}] recommendation 42 1362093742 {"reclen":5} {"subject":"user:11"} 23 {} [{"id":"1707386","rating":"10.0","rank":"1"},{"id":"1623205","rating":"4.9 629755","rank":"2"},{"id":"1024648","rating":"3.1594896","rank":"3"},{"id":"1428538","rating":"1.0","rank":"4"},{"id":"1045658","rating":"1.0","rank":"5"}] recommendation 43 1362093742 {"reclen":5} {"subject":"user:15"} 27 {} [{"id":"1547234","rating":"10.0","rank":"1"},{"id":"1840417","rating":"8.0 ","rank":"2"},{"id":"1276104","rating":"8.0","rank":"3"},{"id":"1235522","rating":"7.7758107","rank":"4"},{"id":"1790885","rating":"7.725913","rank":"5"}] recommendation 44 1362093742 {"reclen":5} {"subject":"user:16"} 5 {} [{"id":"1907668","rating":"7.0","rank":"1"}]

the evaluator is still getting the following error: INFO [datastream] Evaluator: no recommendations received for 60 secs, stopping. INFO [datastream] Exception in thread "Thread-6" java.lang.ArithmeticException: / by zero INFO [datastream] at eu.crowdrec.idomaar.evaluation.StreamingEvaluator.run(StreamingEvaluator.java:325) INFO [datastream] at java.lang.Thread.run(Thread.java:745) INFO [datastream] INFO [datastream] Evaluation results INFO [datastream] ==================

could you take a look on that?

[davidemalagoli]

Fixed with flume classpath update