search

artmoskvin / real-time-recommender

Real-time collaborative filtering recommender system
48 stars 28 forks source link

empty recommendations after sending learn events #3

Closed willli666 closed 4 years ago

willli666 commented 4 years ago

I sent about 30K clicks and 1k buys to the learn endpoint. The learn response is always "Message successfully sent to Kafka"

However, when I query for /bestsellers and /trendings, I only get empty responses like this

{
    "id": "ee1b85f6-6d24-448b-9a60-7387dd336b1f",
    "recommendation": []
}

Can you give me some suggestion on how to debug this? Where should I look at to find out why it's not working? Thanks

artmoskvin commented 4 years ago

@willli666 you can check whether there are any data in Cassandra. If it's not there, then check Kafka and Storm. In Storm UI you can usually see the number of processed events by topology.

willli666 commented 4 years ago

@moscowart My cassandra has no data. The Storm UI shows no data in the topology summary. So I look into Kafka. It has one topic recommender-clickstream. When I run this command, I'm expecting to see the same event I send over the learn endpoint but it does not print out anything.

./kafka-console-consumer.sh  --topic recommender-clickstream  --bootstrap-server localhost:2181

So it seems like the event was never sent to Kafka successfully. I then look into the source code and find this line inside src/main/resources/kafka-0.10.0.1-producer-defaults.properties

bootstrap.servers=172.17.0.1:9092

I then changed this to 127.0.0.1 as I'm directly running the jar without docker. But the problem remains. Is there any way for me to double check on wether the event is successfully relayed from the webserver to kafka?

artmoskvin commented 4 years ago

@willli666 are you sure about localhost:2181. Usually, Kafka listens on port 9092. You should also add --from-beginning flag to see older messages.

artmoskvin commented 4 years ago

@willli666 you can also improve sending messages to Kafka as described in org.apache.kafka.clients.producer.KafkaProducer#send. For debugging, I would simply add get() after sending it. But for production usage, you should process returned Future accordingly.

willli666 commented 4 years ago

Thanks for the help. It is working now with great results!