search

dannyvassallo / insta_scrape

The instagram swiss army knife. Restores all deprecated hashtag functionality and grants public api access from instagram's front end without any of the authorization.
https://rubygems.org/gems/insta_scrape
MIT License
60 stars 27 forks source link

Collect post date when scraping posts #7

Closed oguzcanhuner closed 7 years ago

oguzcanhuner commented 8 years ago

Add a switch to all public methods which collect posts. This switch instructs the scraper to go through each individual post page and scrape the post date from there. The switch makes the methods run much slower, but it does give scope to collect comments (which you couldn't otherwise do).

In terms of implementation, I considered creating a new public method which would collect extra information about posts. This seemed redundant since all of the other public methods were returning posts in slightly different ways, but each one could benefit from returning extra post data. I decided to add a switch to every public method which returned posts, called include_meta_data and set it to false by default, so that a user wouldn't have to use it if they didn't need it.

I've added three tests but just wanted to get your thoughts on this before adding more and updating the README.

Thanks!

dannyvassallo commented 8 years ago

@oguzcanhuner Thanks for this -- will review ASAP!

dannyvassallo commented 8 years ago

@oguzcanhuner I really like this! Do you think the switch is necessary though? Did you notice any lag in the scrape when you ran it with include_meta_data set to true?

We could just leave it in the object by default and the end user can opt out of using the info when rendering the posts. What do you think?

oguzcanhuner commented 8 years ago

@dannyvassallo hey, thanks for replying!

Unfortunately the lag is definitely significant (I haven't measured it properly but it feels at least three times slower than the normal methods), which is why I wanted to have it as an opt-in switch and make it clear in the README that it will be much slower.

dannyvassallo commented 8 years ago

@oguzcanhuner Perfect then. Looks good to me. If you want to update the README, I'll retest and then merge. Thanks!