search

mozilla-services / shavar-list-creation

Script to transform the Disconnect block-list into Safebrowsing v2 format for Firefox Tracking Protection
Mozilla Public License 2.0
15 stars 20 forks source link

readme

shavar-list-creation

Build Status Coverage

This script fetches blocklist .json from urls (such as shavar-prod-lists) and generates safebrowsing-compatible digest list files to be served by shavar.

Requirements

Run

  1. (optional) Make a virtualenv for the project and activate it:

    virtualenv -p python3.8 shavar-list-creation
source shavar-list-creation/bin/activate

  2. Install required libraries:

    pip install -r requirements-test.txt

  3. Copy the sample_shavar_list_creation.ini file to shavar_list_creation.ini:

    cp sample_shavar_list_creation.ini shavar_list_creation.ini

  4. Run the unit tests (currently under development):

    python -m pytest -v --cov=. --cov-branch

  5. Run the lists2safebrowsing.py script:

    ./lists2safebrowsing.py

Usage

This is run by a Jenkins deployment job every 30 minutes that:

  1. Checks out this repository
  2. Checks out the shavar-list-creation-config repository
  3. Copies stage.ini or prod.ini to shavar_list_creation.ini
  4. Runs python lists2safebrowsing.py, which uploads updated safebrowsing list files to S3 for shavar. ⚠️ Since there is no concept of deploy for the scripts, the changes that are merged to main are immediately accessible Jenkins for both Stage and Production.