#1454 xenforo batch post dl

[Kiradien]

New Manual parser for xenforo forums; downloads all posts across multiple pages - resolves ticket #1454

[dteviot]

@Kiradien

You don't need to do this:

        let chapters = [...Array(pageCount).keys()].map(index => { 
            var link = dom.createElement("a");
            link.setAttribute("href", `${baseURI}${pagingUriComponent}${index}`); //ChangeMe!
            link.appendChild(dom.createTextNode(`Page ${index + 1}`));
            return link;
        });
        return chapters.map(a => util.hyperLinkToChapter(a));

Something like this should work

        return [...Array(pageCount).keys()].map(index => ({ 
            sourceUrl: `${baseURI}${pagingUriComponent}${index}`,
            title: `Page ${index + 1}`
        }));

Also, should the //ChangeMe be there?

Have a look at splitContentIntoEpubItems(). This is used to convert single web page from the source into multiple chapters in the Epub.

[Kiradien]

Also, should the //ChangeMe be there?

No, no it really shouldn't. That was supposed to be a reminder for me to go back to that section and basically dig around to figure out exactly what you suggested as a cleaner solution, and I completely forgot about it. I'll make a new pull request and leave it open for suggestions/improvements for a day or so. Thank you for the much cleaner code.

As for splitContentIntoEpubItems() - I'll look into it once I've got a bit more willpower to look into this again - though admittedly, the current version may need to stand as a superior option in the long term. One of the threads that I used to test this (and one of the main reasons I wanted to get this working) actually had over 700 pages of posts at 25 posts per page - so well over 10000 'chapters' when converted that way... As you can imagine, with that many super short 'chapters', things got somewhat weird.

[gamebeaker]

@dteviot Can't 10000 be changed to 99999? I think the limit only has something todo with the naming scheme.

[dteviot]

@gamebeaker 99999 is definitely too big. An epub is a zip file, and the (old) zip spec is maximum of 64k files in a zip. I think I've seen something in jszip that sets maximum to 16k. Also, have you ever tried to use the ToC for an epub with 10k chapters?

[gamebeaker]

@dteviot no i haven't tried i thought that 10k was completely arbitrary. I think the largest book i created had ~4k chapter.

[Kiradien]

Before playing around with this, I thought 10k was arbitrary as well; issues cropped up with time actually rendering the epub - it took over a minute to open the file - but the real nightmare was debugging. I'm pretty sure something went wrong in the javascript arrays somewhere along the line as the chapters came out in the wrong order with some repeats...

I admit it might've been the code, but the outcome was pretty random/screwy when the exact same code appeared to work fine for fewer pages. Normally debugging isn't an issue, but it's a bit of a nightmare to debug an object containing >10k chapters. Just gonna say that erasing everything and reverting to this version was extremely satisfying