webcomics / dosage

dosage is a comic strip downloader and archiver
https://dosage.rocks/
MIT License
122 stars 59 forks source link

Oglaf stopped working. #115

Open kierun opened 6 years ago

kierun commented 6 years ago

The error is:

Oglaf> ERROR: XPath //a[div[@id="nx"]] not found at URL http://oglaf.com/.   

It might be because this one is a two parter. The previous one was fine.

kierun commented 6 years ago

The problem is because there are two (or more) parts. This weeks was fine.

TobiX commented 4 years ago

This is really hard to track down, since it only happens sometimes :man_shrugging: I still leave this up for now.

pckizer commented 3 years ago

This is happening again right now and I think I see what's going on but I haven't looked at the internals of dosage enough to know the full correct fix. For the trigger, it looks like it only happens when a day's comic has 2 pages/images triggering the '//link[@rel="next"]' match.

It appears that the custom fetchUrls in the Oglaf module will handle the first available '//link[@rel="next"]', but after going to that 2nd page and not finding any more image pages (the "next" button is greyed out with no '//link[@rel="next"]'), the urls.extend(super(Oglaf, self).fetchUrls(url, data, search)) has nothing to extend with and throws an error/exception.

I have a minimally working module now by doing a naive modification to the Oglaf fetchUrls changing:

def fetchUrls(self, url, data, search):
    urls = []
    urls.extend(super(Oglaf, self).fetchUrls(url, data, search))
    if search == self.imageSearch:

to:

def fetchUrls(self, url, data, search):
    urls = []
    try:
        urls.extend(super(Oglaf, self).fetchUrls(url, data, search))
    except Exception:
        # Cannot go further in extending URLs, failure means done
        pass
    if search == self.imageSearch:

I hope that helps enough that the maintainers can create a fully correct patch for it.