search

rinosukmandityo / maknews

Simple news creation and retrieval application with Hexagonal Architecture using Golang, Kafka, ElasticSearch, Redis and MySQL
30 stars 3 forks source link

feedpushr / kafka / data ingestion #1

Closed ghost closed 4 years ago

ghost commented 4 years ago

Hi,

Hope you are all well !

I was looking after some projects using kafka/golang and related to news. Then, I found your repository. To be clear, my aim is to learn more about kafka/zookeeper with a real world use case.

So, some questions popped from my brain while studying your repository :-)

Is it possible to plug this feed aggregator having a kafka plugin, https://github.com/ncarlier/feedpushr. The idea is to ingest all articles aggregated into a database.

Would it be possible to do so with maknews ? How complicated it would be ?

Thanks in advance for any insights or inputs on these questions.

Cheers, X

rinosukmandityo commented 4 years ago

Hi,

Hope you are all well !

I was looking after some projects using kafka/golang and related to news. Then, I found your repository. To be clear, my aim is to learn more about kafka/zookeeper with a real world use case.

So, some questions popped from my brain while studying your repository :-)

Is it possible to plug this feed aggregator having a kafka plugin, https://github.com/ncarlier/feedpushr. The idea is to ingest all articles aggregated into a database.

Would it be possible to do so with maknews ? How complicated it would be ?

Thanks in advance for any insights or inputs on these questions.

Cheers, X

Hi,

My repository basically send the news data into all these 4 places:

  1. Into elasticsearch for searching purpose
  2. Into Redis for caching purpose
  3. Into Kafka producer and there is also a consumer which run in different go routine. This consumer will read any new data from producer and write it into MySQL or MongoDB database.
  4. Into MySQL or MongoDB database.

So all news will be ingested into a database. I don't how feedpushr works, but I think it has same idea and it also using Kafka, but feedpushr is more complex than mine and has UI. My project is a way simpler than feedpushr in my opinion.