Feature - Real-time data streaming with Kafka - Githubissues

ayahusseini / spotify_success

Analysing Spotify song data to forecast success

MIT License

1 stars 0 forks source link

Feature - Real-time data streaming with Kafka #11

Open ayahusseini opened 2 weeks ago

ayahusseini commented 2 weeks ago

The Problem Currently, only a Kaggle training set from 2022 is being used. Eventually, I'd like to incrementally update the model with live data

Solution outline

Make decisions on how many tracks to sample, how frequently
Create a kafka topic for tracks
Build the producer (probably with the help of the spotify API)
Create the consumer

Acceptance Criteria

[ ] Replaces the extract script
[ ] Integrated into pipeline

Additional context

possible reference https://medium.com/data-engineering-indonesia/real-time-music-data-streaming-using-apache-kafka-leveraging-avenged-sevenfold-data-from-spotify-25304abcb9bb