Collecting Private Bookmarks

[Luna-35813]

This is a great program and works really well with public bookmarks!

I know it's been a while (3 years) since this program was created and the likely hood of this issue being fixed is low but I just wanted to try.

So I've found that once you enter password it won't print any urls (private or public) it only works if you only enter your username.

When you enter your password it will always say, "Logging in Login error! fetching urls Finished! with 0 urls"

I assume there is an issue with the password portion of the code because my password is definitely correct but I might be wrong. I have tried looking through the code but my python knowledge isn't advanced enough to figure out anything.

Thank you for creating this program, if it comes down to it I will just make all my bookmarks public.

[Hamarel]

Hy! I found the Problem. The Login is no longer correct. Change this part

 #log in
  url = 'https://archiveofourown.org/user_sessions/'
  data = {
    'user_session[login]': username,
    'user_session[password]': password,
    'authenticity_token': authenticity_token #if you don't pass this along nothing works
  }

to this

#log in
  url = 'https://archiveofourown.org/users/login'
  data = {
    'user[login]': username,
    'user[password]': password,
    'authenticity_token': authenticity_token #if you don't pass this along nothing works
  }

and it will work!

[estherflails]

Hello! I'm having the same issue with the error message, but it doesn't work with public bookmarks either. I tried the solution suggested here, but it made no difference. Can anyone help? 😓

[syrtis-m]

I have a fix of the code implemented on this repo : https://github.com/shuos-rst/ao3-bookmark-getter try downloading that python file and running it.

[estherflails]

Thank you, I tried it. It works! 😊 Unfortunately, it doesn't output all of my bookmarks. When I ask for metadata, it gives me around 2300, without it, around 600. Could it be because of Cloudflare? I don't know if Ao3 uses it or not, but when I've been downloading the URLs for a bit and for some minutes afterwards, I couldn't access the website. It just gave me a completely white screen and a "Retry later" message. While the website was like that, it wouldn't let me start downloading: Logging in Logged in! fetching urls Finished! with 0 URLs

This probably wouldn't be a problem if I didn't have so many bookmarks 😅 Could a sleep time be implemented so it takes longer to fetch the URLs but it can finish without ao3 getting overwhelmed?

(By the way, it seems the original (with the edit) works too, I was just an idiot who doesn't know how to use the terminal properly. Oops 😅)

[syrtis-m]

huh, that's weird. I updated my version (link in the first comment from me) so that it pauses for 3 seconds every 10 works, and 15 seconds every 100 works -- hopefully that'll fix the problem you have. unfortunately, I only have ~400 bookmarks, so I can't really test this fix -- if it's still having problems, we can try bumping that pause time up even further.

let me know how it goes! 😄

[estherflails]

Thank you! I set the sleep times a bit higher and I managed to get all my URLs, so long as I'm logged out. (It's almost 200 short but I had the account for a long time. Those ones probably got deleted. 😢)

When I log in, though, it only goes up to 875, even if I set the sleep time higher. No idea why, but I guess I can grab my private bookmarks manually if I have to.

I ended up setting the sleep time to 30 and 100, but they might not need to be that high. For logged-in download, to get the max I could, it needed to be at least around 120 and 160.

FYI for people with a lot of bookmarks who might be reading this: it seems the terminal can only display a limited number of rows. I only got the last 3000 URLs when I downloaded with metadata because the first ones got eaten, so I guess the limit is 9000 (downloading with metadata causes one URL to basically "take up" three rows instead of one - a separate one for the title and an empty one between each title & URL.). So if you have more bookmarks than that, you might want to copy-paste the results at some point before the whole operation is finished and sort out the duplicates later if you need to.

[syrtis-m]

wonderful! one last update from my side -- i updated my code so that it prompts the user how long they want to set each sleep timer, so that they don't need to edit any code. no idea why i didn't just do that last time :)

[NylaTheWolf]

wonderful! one last update from my side -- i updated my code so that it prompts the user how long they want to set each sleep timer, so that they don't need to edit any code. no idea why i didn't just do that last time :)

Sorry for bringing this up again, but I tried using your fork but I only got ~580 out of my 905 bookmarks?

I installed it as a zip file and put it into a directory, cd'd into it, and ran the script. I put my username and password. I said no to getting any metadata. II put "y" when asked if I wanted the program to pause every 10 seconds + 100 seconds, but I put 0 when I was prompted if I wanted the program to delay every 10 works. I put 1 for every 100 works though.

I'm also now experiencing the "Retry later" error haha

Edit: Tried running the script again because right after posting that comment the bookmark pages started working again. This time I put 1 for both every 10 works and every 100 works. It still gave me 568 works 😔

[syrtis-m]

Edit: Tried running the script again because right after posting that comment the bookmark pages started working again. This time I put 1 for both every 10 works and every 100 works. It still gave me 568 works 😔

Try doing 10 every 10 works and 100 every 100 works. Best I can figure out, AO3 has some sort of limit for how quickly you can access it, and the script blows past that when doing big batches of url fetching -- that's why there's the "retry later" error.

Hope this helps! If not, LMK and I'll try to hunt down the error this weekend.

[justneryth]

Edit: Tried running the script again because right after posting that comment the bookmark pages started working again. This time I put 1 for both every 10 works and every 100 works. It still gave me 568 works 😔

Try doing 10 every 10 works and 100 every 100 works. Best I can figure out, AO3 has some sort of limit for how quickly you can access it, and the script blows past that when doing big batches of url fetching -- that's why there's the "retry later" error.

Hope this helps! If not, LMK and I'll try to hunt down the error this weekend.

Hi. I've tried setting 250 for every 10 works and 720 for every 100 works. However, I was only able to retrieve 520 out of 850 works.