search

dannyvassallo / insta_scrape

The instagram swiss army knife. Restores all deprecated hashtag functionality and grants public api access from instagram's front end without any of the authorization.
https://rubygems.org/gems/insta_scrape
MIT License
60 stars 27 forks source link

long_scrape does not working #4

Closed sigaev-pro closed 7 years ago

sigaev-pro commented 8 years ago

Hello,

I'm tring to use long_scrape_hashtag method but it does not working.

Following code returns only 21 images.

require "insta_scrape"

scrape_result = InstaScrape.long_scrape_hashtag("test", 60) scrape_result.each do |post| puts post.image puts post.link end

dannyvassallo commented 8 years ago

@sigaev-pro Specs still pass -- results can be inconsistent because it's a scraper. Have you tried again? I pulled over 2k in specs.

sigaev-pro commented 8 years ago

I have file insta.rb with code above and I start it simply with ruby insta.rb

My be I'm doing something wrong?

dannyvassallo commented 8 years ago

Running your same code dropped to 6 seconds vs 60 returns 240 results for me:

require "insta_scrape"

scrape_result = InstaScrape.long_scrape_hashtag("test", 6)
scrape_result.each do |post|
  puts post.image
  puts post.link
end

puts scrape_result.length
dannyvassallo commented 8 years ago

@sigaev-pro are you running the newest version of the gem?

sigaev-pro commented 8 years ago

I have absolutely the same code and following Gemfile:

source 'https://rubygems.org'

gem "insta_scrape"
gem "capybara"
gem "phantomjs"
gem "poltergeist"

bundle command output:

Using addressable 2.4.0
Using mime-types-data 3.2016.0521
Using mini_portile2 2.1.0
Using pkg-config 1.1.7
Using rack 2.0.1
Using cliver 0.3.2
Using phantomjs 2.1.1.0
Using multi_json 1.12.1
Using websocket-extensions 0.1.2
Using bundler 1.12.5
Using mime-types 3.1
Using nokogiri 1.6.8
Using rack-test 0.6.3
Using websocket-driver 0.6.4
Using xpath 2.0.0
Using capybara 2.7.1
Using poltergeist 1.9.0
Using insta_scrape 1.1.0
Bundle complete! 4 Gemfile dependencies, 18 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.

And output of ruby insta.rb is:

POST COUNT: 21

Image: https://scontent.cdninstagram.com/t51.2885-15/s640x640/sh0.08/e35/c0.24.1080.1080/14052477_541535239364576_570187213_n.jpg?ig_cache_key=MTMyNTQ0NDY2OTk0NDk3NTY3OA%3D%3D.2.c
Link: https://www.instagram.com/p/BJk7OehgVE-/?tagged=test

Image: https://scontent.cdninstagram.com/t51.2885-15/e15/14052279_661295820694056_1028128664_n.jpg?ig_cache_key=MTMyNDkwNTc5MTA1MTgxNzM1Ng%3D%3D.2
Link: https://www.instagram.com/p/BJjAswcgGGM/?tagged=test

Image: https://scontent.cdninstagram.com/t51.2885-15/e15/c145.0.321.321/14052347_167943086966939_440358818_n.jpg?ig_cache_key=MTMyNTI3NzI2ODAyNTM5NDQwMg%3D%3D.2.c
Link: https://www.instagram.com/p/BJkVKdVjcTi/?tagged=test

Image: https://scontent.cdninstagram.com/t51.2885-15/e35/14033487_689258851226348_598776577_n.jpg?ig_cache_key=MTMyNTQxMjc2NDQxMTU0MTU5OA%3D%3D.2
Link: https://www.instagram.com/p/BJkz-MLhLhe/?tagged=test

Image: https://scontent.cdninstagram.com/t51.2885-15/e35/c93.0.534.534/13658743_1846999198861682_2109010898_n.jpg?ig_cache_key=MTMyNTQwMTkyMzMxMDE2OTc5Nw%3D%3D.2.c
Link: https://www.instagram.com/p/BJkxgbnha7F/?tagged=test

Image: https://scontent.cdninstagram.com/t51.2885-15/e15/c157.0.405.405/14052217_1774352182780325_322783727_n.jpg?ig_cache_key=MTMyNDE1MzExMjY0NjYwOTg4NA%3D%3D.2.c
Link: https://www.instagram.com/p/BJgVj2Dh2vc/?tagged=test

Image: https://scontent.cdninstagram.com/t51.2885-15/e15/c236.0.607.607/14099819_223564914712947_1128392518_n.jpg?ig_cache_key=MTMyNTQwNzA4OTU0MzE5ODQ1MQ%3D%3D.2.c
Link: https://www.instagram.com/p/BJkyrnDAbbz/?tagged=test

Image: https://scontent.cdninstagram.com/t51.2885-15/s640x640/sh0.08/e35/14031767_1747734938828279_1309090808_n.jpg?ig_cache_key=MTMyNTUxMDU0MzY3OTAxMTM3OQ%3D%3D.2
Link: https://www.instagram.com/p/BJlKNEOBboz/?tagged=test

Image: https://scontent.cdninstagram.com/t51.2885-15/e35/14052488_1315557765128607_626743908_n.jpg?ig_cache_key=MTMyNTA0MDYyNTYxMTM1Mzg2OQ%3D%3D.2
Link: https://www.instagram.com/p/BJjfW26hq8N/?tagged=test

Image: https://scontent.cdninstagram.com/t51.2885-15/e15/c157.0.406.406/14063389_641852615994934_455220390_n.jpg?ig_cache_key=MTMyNTY5NDEyODg1MjEyMjYzMA%3D%3D.2.c
Link: https://www.instagram.com/p/BJlz8lPDTwG/?tagged=test

Image: https://scontent.cdninstagram.com/t51.2885-15/e35/14052437_530807390454071_657554186_n.jpg?ig_cache_key=MTMyNTY5MzkzNjY3Njg4MzI3Mg%3D%3D.2
Link: https://www.instagram.com/p/BJlz5yQg8tI/?tagged=test

Image: https://scontent.cdninstagram.com/t51.2885-15/s640x640/sh0.08/e35/c0.134.1080.1080/14145347_1180681328658398_425496361_n.jpg?ig_cache_key=MTMyNTY5Mzg3MDA3OTc2MDQ5NA%3D%3D.2.c
Link: https://www.instagram.com/p/BJlz40PBFhu/?tagged=test

Image: https://scontent.cdninstagram.com/t51.2885-15/s640x640/sh0.08/e35/c0.134.1080.1080/14033424_1017465708367089_476263893_n.jpg?ig_cache_key=MTMyNTY5MDA1MjQyNDE2NTUwNg%3D%3D.2.c
Link: https://www.instagram.com/p/BJlzBQxAuiC/?tagged=test

Image: https://scontent.cdninstagram.com/t51.2885-15/s640x640/sh0.08/e35/14052768_297621503948141_1832770539_n.jpg?ig_cache_key=MTMyNTY4OTY5OTM3Mjk3ODM1Nw%3D%3D.2
Link: https://www.instagram.com/p/BJly8H9hVC1/?tagged=test

Image: https://scontent.cdninstagram.com/t51.2885-15/s640x640/sh0.08/e35/14134500_172332459839304_683539627_n.jpg?ig_cache_key=MTMyNTY4ODU4MTU1Nzg0NzI3MQ%3D%3D.2
Link: https://www.instagram.com/p/BJlyr26jATn/?tagged=test

Image: https://scontent.cdninstagram.com/t51.2885-15/e15/14027386_320838048263489_207337257_n.jpg?ig_cache_key=MTMyNTY4Nzg3NzE0OTY5NjIzNw%3D%3D.2
Link: https://www.instagram.com/p/BJlyhm4jKDt/?tagged=test

Image: https://scontent.cdninstagram.com/t51.2885-15/s640x640/sh0.08/e35/c0.135.1080.1080/14052259_549728015215663_1443368039_n.jpg?ig_cache_key=MTMyNTY4NjUyNDc4ODA0NzI4OA%3D%3D.2.c
Link: https://www.instagram.com/p/BJlyN7Zg324/?tagged=test

Image: https://scontent.cdninstagram.com/t51.2885-15/s640x640/sh0.08/e35/c0.135.1080.1080/14063185_1363488587014567_676698560_n.jpg?ig_cache_key=MTMyNTY4NjMzOTc0NDM3MjMxNg%3D%3D.2.c
Link: https://www.instagram.com/p/BJlyLPEDRZc/?tagged=test

Image: https://scontent.cdninstagram.com/t51.2885-15/e35/c64.0.351.351/14063120_175282819547924_2013249600_n.jpg?ig_cache_key=MTMyNTY4NTc2NDMwMjA3MDg0MQ%3D%3D.2.c
Link: https://www.instagram.com/p/BJlyC3JBGQ5/?tagged=test

Image: https://scontent.cdninstagram.com/t51.2885-15/e35/14052317_1769576356652670_1192656887_n.jpg?ig_cache_key=MTMyNTY4NTU1MjM5NzM3NTM0MQ%3D%3D.2
Link: https://www.instagram.com/p/BJlx_xyg1dt/?tagged=test

Image: https://scontent.cdninstagram.com/t51.2885-15/s640x640/sh0.08/e35/14134565_530073543859151_525753568_n.jpg?ig_cache_key=MTMyNTY4NTI5MDI4NzUyNDQ3Nw%3D%3D.2
Link: https://www.instagram.com/p/BJlx79rjGp9/?tagged=test

https://scontent.cdninstagram.com/t51.2885-15/s640x640/sh0.08/e35/c0.24.1080.1080/14052477_541535239364576_570187213_n.jpg?ig_cache_key=MTMyNTQ0NDY2OTk0NDk3NTY3OA%3D%3D.2.c
https://www.instagram.com/p/BJk7OehgVE-/?tagged=test
https://scontent.cdninstagram.com/t51.2885-15/e15/14052279_661295820694056_1028128664_n.jpg?ig_cache_key=MTMyNDkwNTc5MTA1MTgxNzM1Ng%3D%3D.2
https://www.instagram.com/p/BJjAswcgGGM/?tagged=test
https://scontent.cdninstagram.com/t51.2885-15/e15/c145.0.321.321/14052347_167943086966939_440358818_n.jpg?ig_cache_key=MTMyNTI3NzI2ODAyNTM5NDQwMg%3D%3D.2.c
https://www.instagram.com/p/BJkVKdVjcTi/?tagged=test
https://scontent.cdninstagram.com/t51.2885-15/e35/14033487_689258851226348_598776577_n.jpg?ig_cache_key=MTMyNTQxMjc2NDQxMTU0MTU5OA%3D%3D.2
https://www.instagram.com/p/BJkz-MLhLhe/?tagged=test
https://scontent.cdninstagram.com/t51.2885-15/e35/c93.0.534.534/13658743_1846999198861682_2109010898_n.jpg?ig_cache_key=MTMyNTQwMTkyMzMxMDE2OTc5Nw%3D%3D.2.c
https://www.instagram.com/p/BJkxgbnha7F/?tagged=test
https://scontent.cdninstagram.com/t51.2885-15/e15/c157.0.405.405/14052217_1774352182780325_322783727_n.jpg?ig_cache_key=MTMyNDE1MzExMjY0NjYwOTg4NA%3D%3D.2.c
https://www.instagram.com/p/BJgVj2Dh2vc/?tagged=test
https://scontent.cdninstagram.com/t51.2885-15/e15/c236.0.607.607/14099819_223564914712947_1128392518_n.jpg?ig_cache_key=MTMyNTQwNzA4OTU0MzE5ODQ1MQ%3D%3D.2.c
https://www.instagram.com/p/BJkyrnDAbbz/?tagged=test
https://scontent.cdninstagram.com/t51.2885-15/s640x640/sh0.08/e35/14031767_1747734938828279_1309090808_n.jpg?ig_cache_key=MTMyNTUxMDU0MzY3OTAxMTM3OQ%3D%3D.2
https://www.instagram.com/p/BJlKNEOBboz/?tagged=test
https://scontent.cdninstagram.com/t51.2885-15/e35/14052488_1315557765128607_626743908_n.jpg?ig_cache_key=MTMyNTA0MDYyNTYxMTM1Mzg2OQ%3D%3D.2
https://www.instagram.com/p/BJjfW26hq8N/?tagged=test
https://scontent.cdninstagram.com/t51.2885-15/e15/c157.0.406.406/14063389_641852615994934_455220390_n.jpg?ig_cache_key=MTMyNTY5NDEyODg1MjEyMjYzMA%3D%3D.2.c
https://www.instagram.com/p/BJlz8lPDTwG/?tagged=test
https://scontent.cdninstagram.com/t51.2885-15/e35/14052437_530807390454071_657554186_n.jpg?ig_cache_key=MTMyNTY5MzkzNjY3Njg4MzI3Mg%3D%3D.2
https://www.instagram.com/p/BJlz5yQg8tI/?tagged=test
https://scontent.cdninstagram.com/t51.2885-15/s640x640/sh0.08/e35/c0.134.1080.1080/14145347_1180681328658398_425496361_n.jpg?ig_cache_key=MTMyNTY5Mzg3MDA3OTc2MDQ5NA%3D%3D.2.c
https://www.instagram.com/p/BJlz40PBFhu/?tagged=test
https://scontent.cdninstagram.com/t51.2885-15/s640x640/sh0.08/e35/c0.134.1080.1080/14033424_1017465708367089_476263893_n.jpg?ig_cache_key=MTMyNTY5MDA1MjQyNDE2NTUwNg%3D%3D.2.c
https://www.instagram.com/p/BJlzBQxAuiC/?tagged=test
https://scontent.cdninstagram.com/t51.2885-15/s640x640/sh0.08/e35/14052768_297621503948141_1832770539_n.jpg?ig_cache_key=MTMyNTY4OTY5OTM3Mjk3ODM1Nw%3D%3D.2
https://www.instagram.com/p/BJly8H9hVC1/?tagged=test
https://scontent.cdninstagram.com/t51.2885-15/s640x640/sh0.08/e35/14134500_172332459839304_683539627_n.jpg?ig_cache_key=MTMyNTY4ODU4MTU1Nzg0NzI3MQ%3D%3D.2
https://www.instagram.com/p/BJlyr26jATn/?tagged=test
https://scontent.cdninstagram.com/t51.2885-15/e15/14027386_320838048263489_207337257_n.jpg?ig_cache_key=MTMyNTY4Nzg3NzE0OTY5NjIzNw%3D%3D.2
https://www.instagram.com/p/BJlyhm4jKDt/?tagged=test
https://scontent.cdninstagram.com/t51.2885-15/s640x640/sh0.08/e35/c0.135.1080.1080/14052259_549728015215663_1443368039_n.jpg?ig_cache_key=MTMyNTY4NjUyNDc4ODA0NzI4OA%3D%3D.2.c
https://www.instagram.com/p/BJlyN7Zg324/?tagged=test
https://scontent.cdninstagram.com/t51.2885-15/s640x640/sh0.08/e35/c0.135.1080.1080/14063185_1363488587014567_676698560_n.jpg?ig_cache_key=MTMyNTY4NjMzOTc0NDM3MjMxNg%3D%3D.2.c
https://www.instagram.com/p/BJlyLPEDRZc/?tagged=test
https://scontent.cdninstagram.com/t51.2885-15/e35/c64.0.351.351/14063120_175282819547924_2013249600_n.jpg?ig_cache_key=MTMyNTY4NTc2NDMwMjA3MDg0MQ%3D%3D.2.c
https://www.instagram.com/p/BJlyC3JBGQ5/?tagged=test
https://scontent.cdninstagram.com/t51.2885-15/e35/14052317_1769576356652670_1192656887_n.jpg?ig_cache_key=MTMyNTY4NTU1MjM5NzM3NTM0MQ%3D%3D.2
https://www.instagram.com/p/BJlx_xyg1dt/?tagged=test
https://scontent.cdninstagram.com/t51.2885-15/s640x640/sh0.08/e35/14134565_530073543859151_525753568_n.jpg?ig_cache_key=MTMyNTY4NTI5MDI4NzUyNDQ3Nw%3D%3D.2
https://www.instagram.com/p/BJlx79rjGp9/?tagged=test
21
dannyvassallo commented 8 years ago

@sigaev-pro I've re-run specs and again no failure. You shouldn't need to install any other dependencies from your Gemfile, InstaScrape has them already. You can also try to install it globally and see if you can produce any results.

These are the tests that are passing locally and on CI. I also keep the gem installed on my machine and have gotten more than 21 results from "test".

Try this:

#Gemfile
source 'https://rubygems.org'

gem 'insta_scrape'
#insta.rb
require "insta_scrape"

scrape_result = InstaScrape.long_scrape_hashtag("test", 6)
scrape_result.each do |post|
  puts post.image
  puts post.link
end

puts scrape_result.length

Run bundle install && ruby insta.rb

dannyvassallo commented 8 years ago

@sigaev-pro did this work?

dannyvassallo commented 8 years ago

@sigaev-pro I'm going to lock this conversation after today. If you can let me know whether or not this worked / helped that would be great, If not that's fine also -- I'll leave this here in case it can help someone else.

dannyvassallo commented 7 years ago

@sigaev-pro international bugfix applied -- may have been the root of your issue. Try the new version of the gem.