Youtube2zim

⚠️ Disclaimer: The main branch is currently unstable as the scraper is undergoing significant rework for the 3.0 release. If an important patch needs to be applied before the v3 release, please refer to the v2 branch.

youtube2zim allows you to create a ZIM file from a Youtube Channel/Username or one-or-more Playlists.

It downloads the video (webm or mp4 extension – optionally recompress them in lower-quality, smaller size), the thumbnails, the subtitles and the authors' profile pictures ; then, it create a static HTML files folder of it before creating a ZIM off of it.

youtube2zim adheres to openZIM's Contribution Guidelines.

youtube2zim has implemented openZIM's Python bootstrap, conventions and policies v1.0.2.

Requirements

• ffmpeg for video transcoding (only used with --lower-quality).
• curl and unzip to install Javascript dependencies. See get_js_deps.sh if you want to do it manually.

Installation

Here comes a few different ways to install youtube2zim.

Virtualenv

youtube2zim is a Python3 software. If you are not using the Docker image, you are advised to use it in a virtualenv to avoid installing software dependencies on your system. Hatch is the proper tool to create the virtualenv and install the software locally. Ensure to use proper Python version as well (see pyproject.toml).

If you do not already have it on your system, install hatch to build the software and manage virtual environments (you might be interested by our detailed Developer Setup as well, especially regarding how to configure hatch globally for proper detection of its virtual environments by Visual Studio Code).

pip3 install hatch

Start a hatch shell: this will install software including dependencies in an isolated virtual environment.

cd scraper
hatch shell

youtube2zim --help       # Display youtube2zim help

Docker

docker run -v my_dir:/output ghcr.io/openzim/youtube youtube2zim --help

Usage

youtube2zim uses Youtube API v3 to fetch data from Youtube. You thus need to provide an API_KEY to use the scraper.

To get an API:

1. Connect to Google Developers Console
2. Create a new Project then Select it.
3. When asked, choose Create Credentials and select the API Key type. (Credentials page)

youtube2zim --api-key "<your-api-key>" --type user --id "Vsauce" --name "vsauce"

Notes

• Your API_KEY is subject to usage quotas (10,000 requests/day) so use --only_test_branding when adjusting parameters and branding to not waste your quota.
• If you encounter issues reading ZIM files created using this scraper, please take a look at the Compatibility Matrix before opening a ticket.

youtube2zim-playlists

youtube2zim produces a single ZIM file for a youtube request (channel, user, playlist).

youtube2zim-playlists allows you to automatically create one ZIM file per playlist of a given channel or user instead.

This script is a wrapper around youtube2zim and is bundled with the main package.

The difference between a channel and a user is due to Youtube legacy. Some old users have to be searched as a user, while more recent ones have to be searched as a channel. Try your best bet, and if it fails try the other type.

Usage

youtube2zim-playlists --help

Sample usage:

youtube2zim-playlists --indiv-playlists --api-key XXX --type user --id Vsauce --playlists-name="vsauce_en_playlist-{playlist_id}"

Those are the required arguments for youtube2zim-playlists but you can also pass any regular youtube2zim argument. Those will be forwarded to youtube2zim (which will be run independently for each playlist).

Specificities:

• --title and --description are mutually exclusive with --playlists-title and --playlists-description.
• If using --title or --description, all your playlists ZIMs will have the same, static metadata. This is rarely wanted.
• --playlists-title and --playlists-description allows you to dynamically customize them via some playlist-related variables:
  • {title}: the playlist title
  • {description}: the playlist description
  • {slug}: slugified version of the playlist title
  • {playlist_id}: playlist ID on youtube
  • {creator_id}: playlist's owner channel/user ID.
  • {creator_name}: playlist's owner channel/user name.
• You can omit them and youtube2zim will auto-generate those.
• you must specify --playlists-name (supports variables listed above).
• --playlists-name is used to set the Name metadata of the ZIM (should be unique) and if not set separately, the output file name for the ZIM.
• --metadata-from allows to specify a path or URL to a JSON file specifying custom static metadata for individual playlists. Format:

{
    "<playlist-id>": {
        "name": "",
        "zim-file": "",
        "title": "",
        "description": "",
        "tags": "",
        "creator": "",
        "profile": "",
        "banner": ""
    }
}

All fields are optional and taken from command-line/default if not found. <playlist-id> represents the Youtube Playlist ID.

If you feel the need for setting additional details in this file, chances are you should run youtube2zim independently for that playlist (still possible!)

Development

Before contributing be sure to check out the CONTRIBUTING.md guidelines.

License

GPLv3 or later, see LICENSE for more details.