nianeyna / ao3downloader

Utility for downloading fanfiction in bulk from the Archive of Our Own
GNU General Public License v3.0
201 stars 18 forks source link

Creating a link list with metadata is throwing an error. #84

Closed instaloadererror closed 1 year ago

instaloadererror commented 1 year ago

I tried twice and got the same error at the same point. ao3downloader error

If you want the specific link I used without having to type it out: https://archiveofourown.org/works?commit=Sort+and+Filter&work_search%5Bsort_column%5D=kudos_count&work_search%5Bother_tag_names%5D=&work_search%5Bexcluded_tag_names%5D=&work_search%5Bcrossover%5D=&work_search%5Bcomplete%5D=&work_search%5Bwords_from%5D=&work_search%5Bwords_to%5D=&work_search%5Bdate_from%5D=&work_search%5Bdate_to%5D=&work_search%5Bquery%5D=&work_search%5Blanguage_id%5D=&tag_id=Wednesday+%28TV+2022%29

nianeyna commented 1 year ago

Is there an error message recorded in the log file?

instaloadererror commented 1 year ago

Is there an error message recorded in the log file?

Would I find that by using the "v" option? I don't know what to do with a jsonl file. I selected the "v" option but it hasn't done anything yet. Does it normally take a while to convert?

nianeyna commented 1 year ago

Yeah, it can take a hot minute. If it's been thinking the entire 30 minutes since you left this comment your log file might just be too powerful for it though, lol. If that's the case feel free to email me the raw .jsonl and I'll take a look.

instaloadererror commented 1 year ago

Yeah, it can take a hot minute. If it's been thinking the entire 30 minutes since you left this comment your log file might just be too powerful for it though, lol. If that's the case feel free to email me the raw .jsonl and I'll take a look.

I've emailed it to you. It did eventually finish, but I can't open the .html in Chrome or Firefox all the way. It's too big.

nianeyna commented 1 year ago

Do you have a vpn?

instaloadererror commented 1 year ago

Do you have a vpn?

I do not.

nianeyna commented 1 year ago

Well, that rules out that easy explanation. I might have to investigate this more later - it's getting late.

verotheelf commented 1 year ago

It's erroring out because it hit a fic that doesn't have a summary. Haven't fully looked into it, but the issue probably originates from get_work_metadata() in parse_soup.py

nianeyna commented 1 year ago

@verotheelf oh, fantastic, thank you! That is immeasurably helpful - you're a treasure

verotheelf commented 1 year ago

You're welcome! Happy to support the programs I love.

nianeyna commented 1 year ago

@instaloadererror I have tested the page you had problems on with the latest code, and this should now be fixed. @verotheelf thank you so much again for your help on this!