search

eloquence / freeyourstuff.cc

freeyourstuff.cc - universal content liberation
Creative Commons Zero v1.0 Universal
79 stars 4 forks source link

Plugin only retrieves the first 15 posts before timing out. #130

Open bermudashawn opened 4 years ago

bermudashawn commented 4 years ago

For a few months now the extension has been timing out after the first retrieval. I have not received more that 15 post. Can you extend the time out, or is there another issue?

Screen Shot 2020-03-08 at 4 02 00 PM
eloquence commented 4 years ago

Hey Shawn, I've not been able to reproduce this, but you're not alone with this issue. https://freeyourstuff.cc/browse shows you the downloads from folks who've recently published their Quora answers using the extension -- there's an export of 2,000 answers as recently as February 6, but other folks seem to be getting stuck at 15.

I can adjust the timeout settings but I suspect there might be something else going on. In any case, I'll issue a release in the next few days with higher timeout settings and then maybe you can try again to see if that makes a difference?

bermudashawn commented 4 years ago

Dear Erik, I have installed chrome on a second mac and added the extension. It is working as normal, but using a wired connection. So it could be a WiFi issue, or machine specific. Best Wishes, Shawn

bermudashawn commented 4 years ago

Dear Erik, I guess I spoke too soon. The job ran fine up to 3076 of 3780 items when a grey banner came up saying "Something went wrong". I re-ran the extension and it stopped at 15. SO maybe it is a combination of things? Also, would there be a way to to download a certain time frame, like since the last download or this year or last 12 months? Best Wishes, Shawn

eloquence commented 4 years ago

I've just made some tweaks to the scroll logic which may help with the "15 answers" problem, and I've also increased the timeouts. These changes are in version 0.5.9 (you can check which version you have by typing chrome://extensions into the address bar, if it's still at 0.5.8 it should be updated automatically within the next day or two).

I'm going to try to let it run overnight to see if it gets all your answers. Generally with very large datasets like this, I've noticed that Quora also does seem to experience general slowness the further down the feed you go, so keeping my fingers crossed.

Regarding downloading increments, I don't see a way to do this by extracting answers from the Quora answers feed as we currently do. It doesn't support date ranges or anything like that, as far as I can tell. The https://www.quora.com/content page looks more promising. Unfortunately, I can't see anyone else's "Your Content" feed, only my own, and with only a dozen answers or so on my account, it doesn't make for very good testing. :(

eloquence commented 4 years ago

Bad news: freeyourstuff.cc is currently in Google review hell, so the new version with the fix is not available yet. Will post an update here when it is.

eloquence commented 4 years ago

OK, Google finally let me publish the update -- if you're running version 0.6.1, you have the fix. Let me know whenever you have a chance to try again. :)

bermudashawn commented 4 years ago

Dear Erik Thanks much. It is at 2145 of 3866 and running! Best wishes, Shawn

Shawn Murphy Bermuda (441) 305-6764 US Cell (323) 283-4792 US (310) 594-3626

On Mar 22, 2020, at 7:56 PM, Erik Moeller notifications@github.com wrote:

 OK, Google finally let me publish the update -- if you're running version 0.6.1, you have the fix. Let me know whenever you have a chance to try again. :)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

bermudashawn commented 4 years ago

Dear Erik, It stopped at 3,265 of 3,866. Best Regards, Shawn

eloquence commented 4 years ago

Thanks, I'll see if I can reproduce this by running it against your public answers page. Did it at least give you a set of answers at the end?

eloquence commented 4 years ago

Well, I was able to download most of them, it stopped around 3,640. Note that the count from Quora is not 100% reliable. The oldest answer I have is from December 11, 2017. Can you check under "Your Content" if you have any older ones than that?

bermudashawn commented 4 years ago

Dear Erik, My oldest answer is 2018-01-31, but that is when I started anyway. Best Regards,

eloquence commented 4 years ago

Hi Shawn, I can send you the answers I was able to download for you if that helps - just send me an email at eloquence AT gmail DOT com.