search

masnun / php-lang

The repository for php-lang.net - the source for PHP content mashup
0 stars 1 forks source link

List of sources #2

Open shahariaazam opened 11 years ago

shahariaazam commented 11 years ago

We need to write down the list and figure out what should be greatest sources. So we can be prepared for them.

masnun commented 11 years ago

I have so far done -

We shall also need -

masnun commented 11 years ago

And there is PHP Weeklies - we need to think how we can integrate those.

masnun commented 11 years ago

How do we handle/store the data?

shahariaazam commented 11 years ago

We shouldn't be use any widget-type things. Cause if we use js widget then we can't handle those data as we want. We need raw data from each source then we will put them in database so we can provide search facility and other facility as we want in future.

2 types of source:

for RSS feed we can easily extracted them. And the hardest things is to scrap data from various types of site designed in different structure. We need to build different different scraper for each one. Cause many sources don't provide RSS or API. Actually project-wise complexity is to work with raw data (without RSS and API). The matter is how we will scrap them. Nothing to worry about. It will take time and we both have our main work. So we will discuss here about it when we will be free.

For twitter, yes it should be real time. And I wanted that people will get info from our site real-time. People will get real-time data from us. And I believe we can do that. Not only twitter. We will show data from all sources by real time. We will track all sources about 'when they are being update'. When the source will be 'modified' or 'updated' then automatically our scraper will run to grab the modified data.