nianeyna / ao3downloader

Utility for downloading fanfiction in bulk from the Archive of Our Own
GNU General Public License v3.0
201 stars 18 forks source link

Properly identify links to individual chapters of a work as "work" links #102

Closed 3iren closed 10 months ago

3iren commented 1 year ago

I'm not sure what exactly is wrong with the fic I put in, but it's stuck at getting pages. So after about 230 page pulls I tried the good old turning it off and on again, and that unfortunately did not work. Not quite sure which link it is as the downloads don't appear to be in order in my files, but I suspect it's an unfinished fic that isn't a link to a series nor is it in a series. Apologises if this is already a known issue.

logging in generating list of work links that are already in the downloads folder (will be skipped) downloading works 5%|████▏ | 1/20 [00:00<00:10, 1.88it/s]finished getting page 1. starting page 2 finished getting page 2. starting page 3 finished getting page 3. starting page 4 finished getting page 4. starting page 5 finished getting page 5. starting page 6 finished getting page 6. starting page 7 finished getting page 7. starting page 8 finished getting page 8. starting page 9 finished getting page 9. starting page 10 finished getting page 10. starting page 11 finished getting page 11. starting page 12 finished getting page 12. starting page 13 finished getting page 13. starting page 14 finished getting page 14. starting page 15 finished getting page 15. starting page 16 finished getting page 16. starting page 17 finished getting page 17. starting page 18 finished getting page 18. starting page 19 finished getting page 19. starting page 20 finished getting page 20. starting page 21 finished getting page 21. starting page 22 finished getting page 22. starting page 23 finished getting page 23. starting page 24 finished getting page 24. starting page 25 finished getting page 25. starting page 26 finished getting page 26. starting page 27 finished getting page 27. starting page 28 finished getting page 28. starting page 29 finished getting page 29. starting page 30 finished getting page 30. starting page 31 finished getting page 31. starting page 32 finished getting page 32. starting page 33 finished getting page 33. starting page 34 finished getting page 34. starting page 35 finished getting page 35. starting page 36 finished getting page 36. starting page 37 finished getting page 37. starting page 38 finished getting page 38. starting page 39 finished getting page 39. starting page 40 finished getting page 40. starting page 41 ao3 has requested a 300 second break paused at: 04:08:40 resuming at: 04:13:40

nianeyna commented 1 year ago

Interesting, would you mind emailing me your log.jsonl file? nianeyna at gmail

nianeyna commented 1 year ago

You appear to be entering links to individual fics instead of to collections of fics (such as series, author's works, search results, etc) which is not intended (see issue #92). If you want to download works one by one manually, there's no reason to use ao3downloader - just use the download buttons provided by ao3.

3iren commented 1 year ago

huh interesting, I've been choosing the download links from file option and directed it to a .txt file with links I've copied and pasted from my bookmarks to make downloading a lot of fics easier

nianeyna commented 1 year ago

Ah, I see. Well, if you want to do that, in the txt file make sure you link to the full work rather than the first chapter (you can check this by whether "chapters" appears in the url). That said... why not just give it the link to your bookmarks?

3iren commented 1 year ago

Maybe I'm just misunderstanding something really simple due to my tech illiteracy, I'm not using AO3 bookmarks because I was an idiot and for some reason did not make an account for ages :'). So instead I'm using my browser bookmarks which have been exported in seperate txt files based on the fandom for easy organising.

nianeyna commented 1 year ago

Ohhhh, I assumed you meant ao3 bookmarks, sorry! Yeah, makes sense. Hm. I may be able to fix it so that it can recognize chapter links - I'll reopen this issue to keep track of that idea. In the meantime, you would have to delete the "chapters" bit from all of your links to get it to work, unfortunately.

verotheelf commented 1 year ago

If you want, FanFicFare pulls the full work out of alt url formats for ao3 so there's probably some regex logic you can reuse

nianeyna commented 10 months ago

Should be fixed now - any url containing /works/ followed by a work id should now be recognized and properly handled, even if there is more stuff in the url after the work id