Tweets split across multiple files

[readtedium]

Hi there, I was running into an issue where it was not importing tweets before 2016. (I have a lot of tweets.) After looking in the archive folder, I think the root cause is that Twitter split up the file into two parts, tweets.js, and tweets-part1.js.

I can probably work around this manually, but just flagging in case anyone has a similar issue.

[readtedium]

If it helps anyone, I made this tweak to manage it:

async function main() {
    console.log(`Import started at ${new Date().toISOString()}`);
    console.log(`SIMULATE is ${SIMULATE ? "ON" : "OFF"}`);

    const fTweetsPart1 = FS.readFileSync(process.env.ARCHIVE_FOLDER + "/data/tweets-part1.js");
    const fTweetsPart2 = FS.readFileSync(process.env.ARCHIVE_FOLDER + "/data/tweets.js");

    function cleanTweetFileContent(fileContent) {
        return fileContent
            .toString()
            .replace(/window\.YTD\.tweets\.part[0-9]+ = \[/, "[")
            .replace(/;$/, "");
    }

    const tweetsPart1 = JSON.parse(cleanTweetFileContent(fTweetsPart1));
    const tweetsPart2 = JSON.parse(cleanTweetFileContent(fTweetsPart2));
    const tweets = [...tweetsPart1, ...tweetsPart2];

I don’t know if it will be a perfect fix or if there’s a more efficient way to do it but for people with giant archives like mine, I hope it helps.

[marcomaroni-github]

@readtedium I applied your changes with some improvements to make it more flexible even for those who have more than two files in this branch, if it works I'll merge it into the main