kemayo / leech

Turn a story on certain websites into an ebook for convenient reading
MIT License
154 stars 24 forks source link

Somthing crashes the prgram when removing spoilers on royal road on In Lokis Honor #74

Closed KeinNiemand closed 2 years ago

KeinNiemand commented 2 years ago

When I try and download In Lokis Honor (https://www.royalroad.com/fiction/34323/in-lokis-honor-season-ii) Leech crashes with this error: Traceback (most recent call last): File "/home/lukas/.local/bin/leech", line 8, in sys.exit(cli()) File "/home/lukas/.local/lib/python3.8/site-packages/click/core.py", line 829, in call return self.main(args, kwargs) File "/home/lukas/.local/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/home/lukas/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/lukas/.local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "/home/lukas/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(args, **kwargs) File "/home/lukas/.local/lib/python3.8/site-packages/leech.py", line 168, in download story = open_story(site, url, session, login, options) File "/home/lukas/.local/lib/python3.8/site-packages/leech.py", line 109, in open_story story = handler.extract(url) File "/home/lukas/.local/lib/python3.8/site-packages/sites/royalroad.py", line 55, in extract contents, updated = self._chapter(chapter_url, len(story) + 1) File "/home/lukas/.local/lib/python3.8/site-packages/sites/royalroad.py", line 72, in _chapter self._clean_spoilers(content, chapterid) File "/home/lukas/.local/lib/python3.8/site-packages/sites/royalroad.py", line 96, in _clean_spoilers spoiler_title = spoiler['data-caption'] File "/home/lukas/.local/lib/python3.8/site-packages/bs4/element.py", line 1486, in getitem return self.attrs[key] KeyError: 'data-caption'

commenting out clean spoiler fixes the issue, so theirs some kind of bug related to that.

kemayo commented 2 years ago

This is specifically because of this chapter: https://www.royalroad.com/fiction/34323/in-lokis-honor-season-ii/chapter/559347/life-26-intermission-map-status

I think the author might have manually added the HTML markup for the spoilers, omitting the normal attributes? Anyway, I can add fallbacks.