search

bbc / zeitgeist

Twitter Zeitgeist
http://zeitgeist.prototyping.bbc.co.uk/
Other
61 stars 4 forks source link

Tweets in queue but nothing in database #1

Open deplorableword opened 14 years ago

deplorableword commented 14 years ago

Hey,

Followed the readme and got my oAuth keys setup, after running: ruby smq/twitter_oauth.rb twitter_oauth_sample.yml I get the message '1000 tweets published to queue' which is awesome, I've got the second process running: ruby scripts/smq bridge --input twitter_sample is showing me tweets. However, none have made it into mysql? do all of these background jobs also have to be running?

seanohalpin commented 14 years ago

Hi,

Before answering your question, I'll explain what the existing command does.

The command ruby scripts/smq bridge --input twitter_sample runs the code in smq/bridge which simply copies its input to its output. The input is the queue called twitter_sample (defined in config/mq.yml). The output defaults to stdout. That's what you're seeing - the twitter stream bridged from the twitter_sample queue to stdout.

To save tweets to the database, you can run this command instead:

ruby scripts/smq save_tweets --input twitter_sample

You will get exception messages as this will try to save twitter 'delete' instructions which have not been filtered out of the input stream, but you should start to get tweets saved to the database.

I'll be adding some documentation on setting up your own filter pipelines in the next couple of days.

Thanks for trying out our code.

Regards, Sean

deplorableword commented 14 years ago

ah I see,

In terms of the background process to extract links, bit.ly, etc Do these also need to be running or are they just optional?

seanohalpin commented 14 years ago

They're optional if all you want to do is store tweets.

deplorableword commented 14 years ago

ok, just so I'm clear, to start tracking tweets and saving them into mysql without any link extraction jazz.

ruby smq/twitter_oauth.rb twitter_oauth_sample.yml and: ruby scripts/smq save_tweets --input twitter_sample

or have I got the wrong end of the stick? :/

seanohalpin commented 14 years ago

Yes. Have you tried it?

deplorableword commented 14 years ago

Yep and I'm afraid I'm not getting any tweets saved.

ruby smq/twitter_oauth.rb twitter_oauth_sample.yml is telling me that 1000 tweets have been pushed into the queue but I'm not seeing anything in MySQL. Where should I look to try to debug what's going wrong?