Unknown error in sentence tokenizer - perhaps blank tweet?

peteyreplies commented 10 years ago

When I run the ebook on @piscosour, I get: http://pastebin.com/atBRUjR5

/task/__gems__/gems/punkt-segmenter-0.9.1/lib/punkt-segmenter/punkt/sentence_tokenizer.rb:81:in `split_in_sentences': undefined method `size' for nil:NilClass (NoMethodError)
        from /task/__gems__/gems/punkt-segmenter-0.9.1/lib/punkt-segmenter/punkt/sentence_tokenizer.rb:21:in `sentences_from_text'
        from /task/ebook.rb:85:in `block in <top (required)>'
        from /task/ebook.rb:84:in `each'
        from /task/ebook.rb:84:in `<top (required)>'
        from /usr/local/lib/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
        from /usr/local/lib/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
        from __runner__.rb:213:in `<main>'
PARAMS: {"force"=>true}
MAX_ID 391278037205262336 TWEETS: 175
MAX_ID 366962806219743233 TWEETS: 186
MAX_ID 347425438747996160 TWEETS: 179
MAX_ID 335165100371632128 TWEETS: 177
MAX_ID 325160051251744768 TWEETS: 158
MAX_ID 319831699913584640 TWEETS: 180
MAX_ID 310696202129776640 TWEETS: 185
MAX_ID 296759621899677697 TWEETS: 178
MAX_ID 285813782536663041 TWEETS: 181
MAX_ID 274316315760664576 TWEETS: 177
MAX_ID 267032957871611905 TWEETS: 162
MAX_ID 262253443677306881 TWEETS: 191
MAX_ID 257254130383466496 TWEETS: 180
MAX_ID 253628382632349696 TWEETS: 180
MAX_ID 251560175029321728 TWEETS: 179
MAX_ID 247498124036280320 TWEETS: 12
MAX_ID 247361259316588545 TWEETS: 0
1559 tweets found

Iron.io throws an error and doesn't tweet.

peteyreplies commented 10 years ago

Update: tried again for a new, clean project, with different source accounts and apps, got the same error:

/task/__gems__/gems/punkt-segmenter-0.9.1/lib/punkt-segmenter/punkt/sentence_tokenizer.rb:81:in `split_in_sentences': undefined method `size' for nil:NilClass (NoMethodError)
    from /task/__gems__/gems/punkt-segmenter-0.9.1/lib/punkt-segmenter/punkt/sentence_tokenizer.rb:21:in `sentences_from_text'
    from /task/ebook.rb:100:in `block in <top (required)>'
    from /task/ebook.rb:99:in `each'
    from /task/ebook.rb:99:in `<top (required)>'
    from /usr/local/lib/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
    from /usr/local/lib/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
    from __runner__.rb:213:in `<main>'
PARAMS: {"force"=>true}
MAX_ID 404621683576680448 TWEETS: 200
MAX_ID 397630619305267201 TWEETS: 200
MAX_ID 385777514003439616 TWEETS: 200
MAX_ID 377940878809378816 TWEETS: 200
MAX_ID 367401518934130690 TWEETS: 200
MAX_ID 355986228354617345 TWEETS: 200
MAX_ID 340599796433821696 TWEETS: 200
MAX_ID 326859284195442688 TWEETS: 200
MAX_ID 315262834047086592 TWEETS: 200
MAX_ID 306973249160482816 TWEETS: 200
MAX_ID 289910849764728833 TWEETS: 200
MAX_ID 270009696939802625 TWEETS: 200
MAX_ID 249413980685234178 TWEETS: 199
MAX_ID 232985731734331392 TWEETS: 200
MAX_ID 223503252287799298 TWEETS: 200
MAX_ID 211937080635424768 TWEETS: 48
MAX_ID 210427096143626240 TWEETS: 0
3248 tweets found

So I guess that would suggest it's a local problem? Although I still don't understand why...

harrisj commented 10 years ago

Added a blank tweet check that might help in revision 31aa8425de4f7656b52ce6f359ad91967d28887e

peteyreplies commented 10 years ago

I already commented - but yes, this made it work! Thanks!

Nate's code was still broken, FYI, but after some poking around I managed to fix that too: https://github.com/peteyreplies/iron_ebooks/commit/97b6078622f5fc9ee9907b08ea5ce51db2f62c55

Thanks again Jacob! Happy Holidays!

On Dec 28, 2013, at 10:27 PM, Jacob Harris notifications@github.com<mailto:notifications@github.com> wrote:

Added a blank tweet check that might help in revision 31aa842https://github.com/harrisj/iron_ebooks/commit/31aa8425de4f7656b52ce6f359ad91967d28887e

— Reply to this email directly or view it on GitHubhttps://github.com/harrisj/iron_ebooks/issues/6#issuecomment-31310401.

harrisj / iron_ebooks

Unknown error in sentence tokenizer - perhaps blank tweet? #6