search

patrickkfkan / patreon-dl

Patreon Downloader
116 stars 8 forks source link
download patreon

readme

Buy Me a Coffee at ko-fi.com

patreon-dl

A Patreon downloader written in Node.js.

Features

You can run patreon-dl from the command-line or use it as a library for your project. Node.js v18 or higher required.

Limitations

FFmpeg dependency

FFmpeg is required when downloading:

Not all video downloads require FFmpeg, but you should have it installed on your system anyway.

Embedded YouTube videos - Premium access

patreon-dl supports downloading embedded YouTube videos. In addition, if you have a YouTube Premium subscription, you can connect patreon-dl to your account and download videos at qualities available only to Premium accounts (e.g. '1080p Premium'). For CLI users, you would configure patreon-dl as follows:

$ patreon-dl --configure-youtube

For library usage, see Configuring YouTube connection.

...or you may just refer to the next section on how to download enhanecd-quality videos without a Premium account.

Embedded videos - external downloader

You can specify external programs to download embedded videos. For YouTube videos, this will replace the built-in downloader. See the example config on how to do this. For library usage, see External downloaders.

The example config utilizes yt-dlp, a popular program capable of downloading YouTube and Vimeo content. As of current release, yt-dlp is also able to download Premium-quality YouTube videos without a Premium account.

Installation

  1. First, install Node.js.
  2. Then, install FFmpeg (if you are going to download videos).

  3. Then, in a terminal, run the following command:

    $ npm i -g patreon-dl

    The -g option is for installing patreon-dl globally and have the CLI executable added to the PATH. Depending on your usage, you might not need this.

CLI usage

$ patreon-dl [OPTION]... URL

OPTION

Option Alias Description
--help -h Display usage guide
--config-file <path> -C Load configuration file at <path> for setting full options
--cookie <string> -c Cookie for accessing patron-only content; how to obtain cookie.
--ffmpeg <path> -f Path to FFmpeg executable
--out-dir <path> -o Directory to save content
--log-level <level> -l Log level of the console logger: info, debug, warn or error; set to none to disable the logger.
--no-prompt -y Do not prompt for confirmation to proceed
--dry-run Run without writing files to disk (except logs, if any). Intended for testing / debugging.
--list-tiers <creator>

List tiers for the given creator(s). Separate multiple creators with a comma.

The purpose of this is to let you find out what tier IDs to set for posts.in.tier filtering option under include section of configuration file.
--list-tiers-uid <user ID> Same as --list-tiers, but takes user ID instead of vanity.
--configure-youtube

Configure YouTube connection.

patreon-dl supports downloading embedded YouTube videos. If you have a YouTube Premium account, you can connect patreon-dl to it for downloading Premium-quality streams.

URL

Supported URL formats

// Download a product
https://www.patreon.com/<creator>/shop/<slug>-<product_id>

// Download posts by creator
https://www.patreon.com/<creator>/posts
https://www.patreon.com/c/<creator>/posts
https://www.patreon.com/user/posts?u=<user_id>

// Dowload a single post
https://www.patreon.com/posts/<post_id>
https://www.patreon.com/posts/<slug>-<post_id>

// Download posts in a collection
https://www.patreon.com/collection/<collection_id>

Multiple URLs

You may specify multiple URLs by separating them with a comma. E.g.:

// First download posts by johndoe, followed by posts by janedoe.
$ patreon-dl "https://www.patreon.com/johndoe/posts,https://www.patreon.com/janedoe/posts"

Supplying URLs through file

You can also use a file to supply URLs to patreon-dl. For example, you can have a urls.txt that has the following content:

# Each URL is placed in its own line
# Comments (lines starting with '#') will be ignored

https://www.patreon.com/johndoe/posts
https://www.patreon.com/janedoe/posts

You can then pass urls.txt to patreon-dl:

$ patreon-dl urls.txt

In this file, you can also override include options provided in a configuration file passed to patreon-dl (through the -C option). include options allow you specify what to include in downloads. This overriding mechanism allows you to specify different content to download for each target URL. For example, you might have the following include option in your configuration file:

...

[include]

# Include posts that belong only to tier ID '-1' (public tier)
posts.in.tier = -1

...

Then, in your urls.txt, you can override as follows:

# URL 1
https://www.patreon.com/johndoe/posts

# Override 'posts.in.tier = -1' in [include] section of configuration file.
# This will cause downloader to download posts from URL 1 belonging to tier with
# ID '123456' or '789100'.

include.posts.in.tier = 123456, 789100

# Other include options - they basically have the same name as those
# in the configuation file, but prepended with 'include.':
#
# include.locked.content
# include.posts.with.media.type
# include.posts.published.after
# include.posts.published.before
# include.campaign.info
# include.content.info
# include.preview.media
# include.content.media
# include.all.media.variants
# include.images.by.filename
# include.audio.by.filename
# include.attachments.by.filename

# URL 2 
https://www.patreon.com/janedoe/posts

# If you don't place any 'include.*' statements here, the downloader will use
# options from configuration file or default values if none provided.

# URL 3
...

Directory structure

Content is saved with the following directory structure:

out-dir
    ├── campaign
        ├── campaign_info
        ├── posts
        │   ├── post 1
        │   │   ├── post_info
        │   │   ├── images
        │   │   ├── ...
        │   ├── post 2
        │       ├── post_info
        │       ├── images
        │       ├── ...
        ├──shop
            ├── product 1
                ├── product_info
                ├── content_media
                ├── ...

Configuration file

Command-line options are limited. To access the full range of options, create a configuration file and pass it to patreon-dl with the (capital) -C option.

Refer to the example config to see what options are offered. Also see How to obtain Cookie.

Note that you can override an option from a configuration file with one provided at the command-line, provided of course that a command-line equivalent is available.

Library usage

To use patreon-dl in your own project:

import PatreonDownloader from 'patreon-dl';

const url = '....';

const downloader = await PatreonDownloader.getInstance(url, [options]);

await downloader.start();

Here, we first obtain a downloader instance by calling PatreonDownloader.getInstance(), passing to it the URL we want to download from (one of the supported URL formats) and downloader options, if any.

Then, we call start() on the downloader instance to begin the download process. The start() method returns a Promise that resolves when the download process has ended.

Downloader options

An object with the following properties (all optional):

Option Description
cookie Cookie to include in requests; required for accessing patron-only content. See How to obtain Cookie.
useStatusCache Whether to use status cache to quickly determine whether a target that had been downloaded before has changed since the last download. Default: true
pathToFFmpeg Path to ffmpeg executable. If not specified, ffmpeg will be called directly when needed, so make sure it is in the PATH.
pathToYouTubeCredentials Path to file storing YouTube credentials for connecting to a YouTube account when downloading embedded YouTube videos. Its purpose is to allow YouTube Premium accounts to download videos at higher than normal qualities. For more information, see Configuring YouTube connection.
outDir Path to directory where content is saved. Default: current working directory
dirNameFormat How to name directories: (object)
filenameFormat Naming of files: (object)
include What to include in the download: (object)
  • lockedContent: whether to process locked content. Default: true
  • postInTier: see Filtering posts by tier
  • postsWithMediaType: sets the media type criteria for downloading posts. Values can be:
    • any: download posts regardless of the type of media they contain. Also applies to posts that do not contain any media.
    • none: only download posts that do not contain media.
    • Array<image | video | audio | attachment>: only download posts that contain the specified media type(s).
    Default: any
  • postsPublished: sets the publish date range for posts. Its value is an object with two properties: after and before (both a DateTime object). You can specify one or both to indicate an open-ended or closed date range.
  • campaignInfo: whether to save campaign info. Default: true
  • contentInfo: whether to save content info. Default: true
  • contentMedia: the type of content media to download (images, videos, audio, attachments, excluding previews). Values can be:
    • true: download all content media.
    • false: do not download content media.
    • Array<image | video | audio | attachment | file>: only download the specified media type(s).
    Default: true
  • previewMedia: the type of preview media to download, if available. Values can be:
    • true: download all preview media.
    • false: do not download preview media.
    • Array<image | video | audio>: only download the specified media type(s).
    Default: true
  • allMediaVariants: whether to download all media variants, if available. If false, only the best quality variant will be downloaded. Default: false
  • mediaByFilename: sets the filename pattern to match against media files. Those that do not match the provided pattern will not be downloaded. Its value is an object with three properties corresponding to the media types supported: images, audio and attachments. For example, to download only ZIP attachments, you would set the option like this:

    mediaByFilename: { attachments: '*.zip' }

    Internally, pattern matching is done by minimatch, which supports glob patterns.
request Rate limiting and retry on error: (object)
  • maxRetries: maximum number of retries if a request or download fails. Default: 3
  • maxConcurrent: maximum number of concurrent downloads. Default: 10
  • minTime: minimum time to wait between starting requests or downloads (milliseconds). Default: 333
fileExistsAction What to do when a target file already exists: (object)
  • info: in the context of saving info (such as campaign or post info), the action to take when a file belonging to the info already exists. Default: saveAsCopyIfNewer
  • infoAPI: API data is saved as part of info. Because it changes frequently, and usually used for debugging purpose only, you can set a different action when saving an API data file that already exists. Default: overwrite
  • content: in the context of downloading content, the action to take when a file belonging to the content already exists. Default: skip

Supported actions:

  • overwrite: overwrite existing file.
  • skip: skip saving the file.
  • saveAsCopy: save the file under incremented filename (e.g. "abc.jpg" becomes "abc (1).jpg").
  • saveAsCopyIfNewer: like saveAsCopy, but only do so if the contents have actually changed.
embedDownloaders External downloader for embedded videos. See External downloaders.
logger See Logger
dryRun Run without writing files to disk (except logs, if any). Default: false

Campaign directory name format

Format to apply when naming campaign directories. A format is a string pattern consisting of fields enclosed in curly braces.

What is a campaign directory?

When you download content, a directory is created for the campaign that hosts the content. Content directories, which stores the downloaded content, are then placed under the campaign directory. If campaign info could not be obtained from content, then content directory will be created directly under outDir.

A format must contain at least one of the following fields:

Characters enclosed in square brackets followed by a question mark denote conditional separators. If the value of a field could not be obtained or is empty, the conditional separator immediately adjacent to it will be omitted from the name.

Default: '{creator.vanity}[ - ]?{campaign.name}'
Fallback: 'campaign-{campaign.id}'

Content directory name format

Format to apply when naming content directories. A format is a string pattern consisting of fields enclosed in curly braces.

What is a content directory?

Content can be a post or product. A directory is created for each piece of content. Downloaded items for the content are placed under this directory.

A format must contain at least one of the following unique identifier fields:

In addition, a format can contain the following fields:

Characters enclosed in square brackets followed by a question mark denote conditional separators. If the value of a field could not be obtained or is empty, the conditional separator immediately adjacent to it will be omitted from the name.

Default: '{content.id}[ - ]?{content.name}'
Fallback: '{content.type}-{content.id}'

Media filename format

Filename format of a downloaded item. A format is a string pattern consisting of fields enclosed in curly braces.

A format must contain at least one of the following fields:

In addition, a format can contain the following fields:

If media.variant is not included in the format, it will be appended to it if allMediaVariants is true.

Sometimes media.filename could not be obtained, in which case it will be replaced with media.id, unless it is already present in the format.

Characters enclosed in square brackets followed by a question mark denote conditional separators. If the value of a field could not be obtained or is empty, the conditional separator immediately adjacent to it will be omitted from the name.

Default: '{media.filename}'
Fallback: '{media.type}-{media.id}'

Filtering posts by tier

To download posts belonging to specific tier(s), set the include.postsInTier option. Values can be:

To obtain the IDs of tiers for a particular creator, first get the campaign through PatreonDownloader.getCampaign(), then inspect the rewards property:

const signal = new AbortSignal(); // optional
const logger = new MyLogger(); // optional - see Logger section
const campaign = await PatreonDownloader.getCampaign('johndoe', signal, logger);

// Sometimes a creator is identified only by user ID, in which case you would do this:
// const campaign = await PatreonDownloader.getCampaign({ userId: '80821958' }, signal, logger);

const tiers = campaign.rewards;
tiers.forEach((tier) => {
  console.log(`${tier.id} - ${tier.title}`);
});

See Campaign, Reward.

External downloaders

You can specify external downloaders for embedded videos. Each entry in the embedDownloaders option is an object with the following properties:

Proprety Description
provider Name of the provider of embedded content. E.g. youtube, vimeo (case-insensitive)
exec The command to run to download the embedded content

exec can contain fields enclosed in curly braces. They will be replaced with actual values at runtime:

Field Description
post.id ID of the post containing the embedded video
embed.provider Name of the provider
embed.provider.url Link to the provider's site
embed.url Link to the video page supplied by the provider
embed.subject Subject of the video
embed.html The HTML code that embeds the video player on the Patreon page
dest.dir The directory where the video should be saved

For example usage of exec, see example-embed.conf.

External downloaders are not subject to request.maxRetries and fileExistsAction settings. This is because patreon-dl has no control over the downloading process nor knowledge about the outcome of it (including where and under what name the file was saved).

Configuring YouTube connection

In its simplest form, the process of connecting patreon-dl to a YouTube account is as follows:

  1. Obtain credentials by having the user visit a Google page that links his or her account to a 'device' (which in this case is actually patreon-dl).
  2. Save the credentials, as a JSON string, to a file.
  3. Pass the path of the file to PatreonDownloader.getInstance()

To obtain credentials, you can use the YouTubeCredentialsCapturer class:

import { YouTubeCredentialsCapturer } from 'patreon-dl';

// Note: you should wrap the following logic inside an async
// process, and resolve when the credentials have been saved.

const capturer = new YouTubeCredentialsCapturer();

/**
 * 'pending' event emitted when verification data is ready and waiting
 * for user to carry out the verification process.
 */
capturer.on('pending', (data) => {
  // `data` is an object: { verificationURL: <string>, code: <string> }
  // Use `data` to provide instructions to the user:
  console.log(
    `In a browser, go to the following Verification URL and enter Code:

    - Verification URL: ${data.verificationURL}
    - Code: ${data.code}

    Then wait for this script to complete.`);
});

/**
 * 'capture' event emitted when the user has completed verification and the 
 * credentials have been relayed back to the capturer.
 */
capturer.on('capture', (credentials) => {
  // `credentials` is an object which you need to save to file as JSON string.
  fs.writeFileSync('/path/to/yt-credentials.json', JSON.stringify(credentials));
  console.log('Credentials saved!');
});

// When you have added the listeners, start the capture process.
capturer.begin();

Then, pass the path of the file to PatreonDownloader.getInstance():

const downloader = await PatreonDownloader.getInstance(url, {
  ...
  pathToYouTubeCredentials: '/path/to/yt-credentials.json'
});

You should ensure the credentials file is writable, as it needs to be updated with new credentials when the current ones expire. The process of renewing credentials is done automatically by the downloader.

Logger

Logging is optional, but provides useful information about the download process. You can implement your own logger by extending the Logger abstract class:

import { Logger } from 'patreon-dl';

class MyLogger extends Logger {

  log(entry) {
    // Do something with log entry
  }

  // Called when downloader ends, so you can
  // clean up the logger process if necessary.
  end() {
    // This is not an abstract function, so you don't have to
    // implement it if there is no action to be taken here. Default is
    // to resolve right away.
    return Promise.resolve();
  }
}

Each entry passed to log() is an object with the following properties:

Built-in loggers

The patreon-dl library comes with the following Logger implementations that you may utilize:

Aborting

To prematurely end a download process, use AbortController to send an abort signal to the downloader instance.

const downloader = await PatreonDownloader.getInstance(...);
const abortController = new AbortController();
downloader.start({
    signal: abortController.signal
});

...

abortController.abort();

// Downloader aborts current and pending tasks, then ends.

Workflow and Events

Workflow

  1. Downloader analyzes given URL and determines what targets to fetch.
  2. Downloader begins fetching data from Patreon servers. Emits fetchBegin event.
  3. Downloader obtains the target(s) from the fetched data for downloading.
  4. For each target (which can be a campaign, product or post):
    1. Downloader emits targetBegin event.
    2. Downloader determines whether the target needs to be downloaded, based on downloader configuration and target info such as accessibility.
      • If target is to be skipped, downloader emits targetEnd event with isSkipped: true. It then proceeds to the next target, if any.
    3. If target is to be downloaded, downloader saves target info (subject to downloader configuration), and emits phaseBegin event with phase: saveInfo. When done, downloader emits phaseEnd event.
    4. Downloader begins saving media belonging to target (again, subject to downloader configuration). Emits phaseBegin event with phase: saveMedia.
      1. Downloader saves files that do not need to be downloaded, e.g. embedded video / link info.
      2. Downloader proceeds to download files (images, videos, audio, attachments, etc.) belonging to the target in batches. For each batch, downloader emits phaseBegin event with phase: batchDownload. When done, downloader emits phaseEnd event with phase: batchDownload.
        • In this phaseBegin event, you can attach listeners to the download batch to monitor events for each download. See Download Task Batch.
    5. Downloader emits phaseEnd event with phase: saveMedia.
    6. Downloader emits targetEnd event with isSkipped: false, and proceeds to the next target.
  5. When there are no more targets to be processed, or a fatal error occurred, downloader ends with end event.

Events

const downloader = await PatreonDownloader.getInstance(...);

downloader.on('fetchBegin', (payload) => {
    ...
});

downloader.start();

Each event emitted by a PatreonDownloader instance has a payload, which is an object with properties containing information about the event.

Event Description
fetchBegin

Emitted when downloader begins fetching data about target(s).

Payload properties:

  • targetType: the type of target being fetched; one of product, post or post.
targetBegin

Emitted when downloader begins processing a target.

Payload properties:
targetEnd

Emitted when downloader is done processing a target.

Payload properties:

  • target: the target processed; one of Campaign, Product or Post.
  • isSkipped: whether target was skipped.

If isSkipped is true, the following additional properties are available:

  • skipReason: the reason for skipping the target; one of the following enums:
    • TargetSkipReason.Inaccessible
    • TargetSkipReason.AlreadyDownloaded
    • TargetSkipReason.UnmetMediaTypeCriteria
  • skipMessage: description of the skip reason.
phaseBegin

Emitted when downloader begins a phase in the processing of a target.

Payload properties:

  • target: the subject target of the phase; one of Campaign, Product or Post.
  • phase: the phase that is about to begin; one of saveInfo or batchDownload.

If phase is batchDownload, the following additional property is available:

  • batch: an object representing the batch of downloads to be executed by the downloader. For monitoring downloads in the batch, see Download Task Batch.
phaseEnd

Emitted when a phase ends for a target.

Payload properties:

  • target: the subject target of the phase; one of Campaign, Product or Post.
  • phase: the phase that has ended; one of saveInfo or batchDownload.
end

Emitted when downloader ends.

Payload properties:

  • aborted: boolean indicating whether the downloader is ending because of an abort request
  • error: if downloader ends because of an error, then error will be the captured error. Note that error is not necessarily an Error object; it can be anything other than undefined.
  • message: short description about the event

Download Task Batch

Files are downloaded in batches. Each batch is provided in the payload of phaseBegin event with phase: batchDownload. You can monitor events of individual downloads in the batch as follows:

downloader.on('phaseBegin', (payload) => {
    if (payload.phase === 'batchDownload') {
        const batch = payload.batch;
        batch.on(event, listener);
    }
})

Note that you don't have to remove listeners yourself. They will be removed once the batch ends and is destroyed by the downloader.

Download Task

Each download task in a batch is represented by an object with the following properties:

Property Description
id ID assigned to the task.
src The source of the download; URL or otherwise file path if downloading video from a previously-downloaded m3u8 playlist.
srcEntity The Downloadable item from which the download task was created.
retryCount The current retry count if download failed previously.
resolvedDestFilename The resolved destination filename of the download, or null if it has not yet been resolved.
resolvedDestFilename The resolved destination file path of the download, or null if it has not yet been reoslved.
getProgress() Function that returns the download progress.

Events

Each event emitted by a download task batch has a payload, which is an object with properties containing information about the event.

Event Description
taskStart

Emitted when a download starts.

Payload properties:

  • task: the download task
taskProgress

Emitted when a download progress is updated.

Payload properties:

  • task: the download task
  • progress: (object)
    • destFilename: the destination filename of the download
    • destFilePath: the destination file path of the download
    • lengthUnit: the unit measuring the progress. Generally, it would be 'byte', but for videos the unit would be 'second'.
    • length: content length, measured in lengthUnit.
    • lengthDownloaded: length downloaded so far, measured in lengthUnit.
    • percent: percent downloaded
    • sizeDownloaded: size of file downloaded (kB)
    • speed: download speed (kB/s)

    Note: sometimes length is undefined, in which case percent will also be undefined.
taskComplete

Emitted when a download is complete.

Payload properties:

  • task: the download task
taskError

Emitted when a download error occurs.

Payload properties:

  • error: (object)
    • task: the download task
    • cause: Error object or undefined
  • willRetry: whether the download will be reattempted
taskAbort

Emitted when a download is aborted.

Payload properties:

  • task: the download task
taskSkip

Emitted when a download is skipped.

Payload properties:

  • task: the download task
  • reason: (object)
    • name: destFileExists or other
    • message: string indicating the skip reason

If reason.name is destFileExists, reason will also contain the following property:

  • existingDestFilePath: the existing file path that is causing the download to skip
taskSpawn

Emitted when a download task is spawned from another task.

Payload properties:

  • origin: the original download task
  • spawn: the spawned download task
complete

Emitted when the batch is complete and there are no more downloads pending.

Payload properties: none

Changelog

v2.1.0

v2.0.0

v1.7.0

v1.6.2

v1.6.1

v1.6.0

v1.5.0

v1.4.0

v1.3.0

v1.2.2

v1.2.1

v1.2.0

v1.1.1

v1.1.0

v1.0.1

v1.0.0