LessWrong Portable

Download the current versions of:

About this

This is started as the latest in a long history of independent, disorganized projects to scrape collections of posts from LessWrong into ebooks. A few selected examples of others:

• jb55's lesswrong-print
• dato's lesswrong-bundle
• OneWhoFrog's lw2ebook
• srlee309's LessWrongEBookCreator
• natewind's lesswrong-fb2

...Not to mention the official version of the Sequences.

So, why on earth did I start another? LessWrong 2.0. Since LessWrong 2.0 was voted to replace LessWrong Classic (see point 4), All other the existing aggregators have broken. This isn't a big deal, since they really only need to run once (correctly) in order to create the ebook, but anyone who wants to modify them and scrape new ebooks won't be able to use them.

As separate rationale, The Codex by Scott Alexander is open for reading. Not that all this content wasn't available elsewhere before, but this is the most intentional linearly-organized collection of his best writings I've seen. I want to read it, and as I read most things, I want to do it on my ebook reader.

What about the non-LessWrong Content?

Oh! I realized that with a surprisingly tiny bit of refactoring, this is flexible enough to work on content outside of LW. So I did the refactoring and here we are.

Where are the Ebook files?

In the output directory.

I want to make my own version! What should I do?

First, Clone this Repository

git clone https://github.com/AABoyles/LessWrong-Portable.git
cd LessWrong-Portable/

Now set up your environment:

npm install

Finally, run npm start <book>, along with the name of the book you want to build. Currently, the options include:

• default - A dummy package that demonstrates the JSON schema by creating an ebook containing only this post.
• codex - The Codex of Scott Alexander
• rationalityabridged - Rationality Abridged by Quaerendo
• inadequate - Inadequate Equilibria by Eliezer Yudkowsky
• meditation - LessWrong on Meditation by LessWrong Authors
• intelligent - The Abridged Guide to Intelligent Characters by Eliezer Yudkowsky
• replacingguilt - The Replacing Guilt Series by Nate Soares
• parenting - Jeff Kaufman on Parenting

There are also meta files for some other content from outside the rationalist community:

• hedonistic - The Hedonistic Imperative by David Pearce
• wbwelonmusk - Wait but Why on Elon Musk by Tim Urban
• scip - The Structure and Interpretation of Computer Programs

So, for example:

npm start codex

That will download all of the content of the Codex into the cache/ directory, and then assemble them all into an EPUB file (outputs/TheCodex.epub).

I'm sure I'm forgetting stuff. Let me know.

I want to make a custom book/sequence! How do I do that?

First follow the directions to build your own version. Once you get to the build step (i.e. npm start <whatever>), instead of building one of the available options, copy the default build meta file to a version named for your own sequence/book.

For example, I wanted to create a book using some LessWrong posts on meditation. Here's what I did:

cp meta/default.json meta/meditation.json

Next, edit meta/meditation.json. Changing this is mostly optional, except for the URLs. That's really, really important. I used these posts as a starting point:

• Meditation, insight, and rationality. (Part 1 of 3)
• Meditation, insight, and rationality. (Part 2 of 3)

The contents of the urls array in my meta config file isn't the full url, but the path following the source. So, my meta config file should look like this:

{
    "img": "images/lw.png",
    "shorttitle": "LessWrongOnMeditation",
    "metadata": {...},
    "titleSelector": ".PostsPageTitle-root",
    "contentSelector": ".PostsPage-postContent",
    "source": "http://lesswrong.com/",
    "urls": [
        "posts/QqSNFcGSZdnARx56E/meditation-insight-and-rationality-part-1-of-3",
        "posts/QjoTFHzvrxQg9A6j3/meditation-insight-and-rationality-part-2-of-3"
    ]
}

If you want to make a book from content outside of LW, you're going to need to change a few more things. The fields in the metadata object should be more-or-less self-explanatory. The titleSelector and contentSelector fields, probably less so. If you're not familiar with CSS selectors, this is going to take a little bit of training. Feel free to email me for help.

Now you can build your new book.

npm start meditation

That should generate a new file entitled output/LessWrongOnMeditation.epub. Enjoy!

Best Practice: Commit your new meta config file to your repository and push it upstream. I'm very interested in aggregating other materials, so if you can manage it, submit a pull request!

How did you make the MOBI version?

It turns out that programatically generating Kindle Formats (e.g. AZW, MOBI) is weirdly difficult. Use Calibre or this Weird Script from Amazon. I may get around to scripting it, someday.

How do you make a PDF/Text/Markdown/[Whatever] Version?

I'm probably not going to. Feel free to fork this repo and figure it out yourself.

How do you make a Word Version?

Go away.

Why does the script call wget, instead of using an http library?

I went through four different libraries to try to make synchronous http requests, and they all did this super annoying thing where they would return a page that hadn't rendered the text content yet (No, it wasn't a promise. Cut me some slack, please). Weirdly, when I made (what I thought was) the same request in curl, it gave me the content I needed. So, instead of figuring out the right way to do it, I just did the thing that worked. I switched to wget when I needed to run a build on a Windows machine and wget was easier to get running. This confers the added bonus in that Ubuntu has wget out of the box but curl must be installed.

Why synchronous requests?

Because it doesn't need to be done fast, but it does need to be done in a precise sequence. Writing an asynchronous version might save a few seconds at runtime, but would take me at least another hour or two to code up. I strongly doubt the number of times this script will ever be run will add up to the development time cost.

I ran the build myself, but it failed! What gives?

If the server barfs for some reason, the script will continue. After all, why waste bandwidth and effort? Re-run it and it will only try to download the files it didn't get the first time. There may be a couple that aren't downloading for structural, rather than essentially random reasons. To fill these in for the canonical ebooks, I just manually saved copies of those pages in the cache/ directory.

If it's having trouble with a custom book you've cooked up, make sure that your the CSS selector for the title is exactly correct. It should be precise enough to identify the title and only the title. If the selector comes up empty, LWP assumes it failed and won't generate a book in the end, though it will continue caching content until it reaches the end of the available URLs.

Why are there some meta files for books that aren't in the output folder?

This is left to the reader as an exercise in reading between the lines.

What's the roadmap?

1. Whatever's in the issues queue.
2. Maybe organize some new sequences in a way that I find useful and add them.

If you want anything else, let me know and I'll tackle it when I've got some spare time. HAHAHAHA.