Data engineering - Githubissues

masnun / php-lang

The repository for php-lang.net - the source for PHP content mashup

0 stars 1 forks source link

Open shahariaazam opened 11 years ago

shahariaazam commented 11 years ago

Actually we need to follow the 3 steps.

masnun commented 11 years ago

We need to identify the potential data sources
Decide on what properties of certain entity shall be stored (eg. for rss feeds only title, for tweets username + title)
Scrape the data sources at fixed intervals. Some data sources less often (eg. feeds), some in real time (tweets, SO)
Check for duplicates
Show the data sets to users in an unified interface

shahariaazam commented 11 years ago

Wow great thinking. It will help us to build this project from scratch. I like it.