search

bcampbell / journalisted

11 stars 7 forks source link

Streaming jhose #5

Closed donovanhide closed 11 years ago

donovanhide commented 11 years ago

Hi Ben, this does streaming replays using postgres cursors. Needs the new eventsource commit. Changed it so that an unspecified Last-Event-Id results in only new articles. To get all articles do:

curl localhost:9999/articles -H "Last-Event-ID: 0"

We should discuss trying notify support in your scrapers. Would make the polling much more efficient and would make it a lot easier to have a channel for each newspaper which would be very useful for testing SFM :-)

Cheers, Donny.

donovanhide commented 11 years ago

Don't forget to add -logtostderr=true to ./jhose to get the console output for debugging!

Glog is good!

bcampbell commented 11 years ago

OK all merged in and works fine here. I wouldn't worry too much about jlhose - I don't plan for it to be around too long. The idea is to have a separate, standalone scraping system which can feed JL, churnalism, whatever.