sqlite-utils

Python CLI utility and library for manipulating SQLite databases.

Some feature highlights

Pipe JSON (or CSV or TSV) directly into a new SQLite database file, automatically creating a table with the appropriate schema
Run in-memory SQL queries, including joins, directly against data in CSV, TSV or JSON files and view the results
Configure SQLite full-text search against your database tables and run search queries against them, ordered by relevance
Run transformations against your tables to make schema changes that SQLite ALTER TABLE does not directly support, such as changing the type of a column
Extract columns into separate tables to better normalize your existing data
Install plugins to add custom SQL functions and additional features

Read more on my blog, in this series of posts on New features in sqlite-utils and other entries tagged sqliteutils.

Installation

pip install sqlite-utils

Or if you use Homebrew for macOS:

brew install sqlite-utils

Using as a CLI tool

Now you can do things with the CLI utility like this:

$ sqlite-utils memory dogs.csv "select * from t"
[{"id": 1, "age": 4, "name": "Cleo"},
 {"id": 2, "age": 2, "name": "Pancakes"}]

$ sqlite-utils insert dogs.db dogs dogs.csv --csv
[####################################]  100%

$ sqlite-utils tables dogs.db --counts
[{"table": "dogs", "count": 2}]

$ sqlite-utils dogs.db "select id, name from dogs"
[{"id": 1, "name": "Cleo"},
 {"id": 2, "name": "Pancakes"}]

$ sqlite-utils dogs.db "select * from dogs" --csv
id,age,name
1,4,Cleo
2,2,Pancakes

$ sqlite-utils dogs.db "select * from dogs" --table
  id    age  name
----  -----  --------
   1      4  Cleo
   2      2  Pancakes

You can import JSON data into a new database table like this:

$ curl https://api.github.com/repos/simonw/sqlite-utils/releases \
    | sqlite-utils insert releases.db releases - --pk id

Or for data in a CSV file:

$ sqlite-utils insert dogs.db dogs dogs.csv --csv

sqlite-utils memory lets you import CSV or JSON data into an in-memory database and run SQL queries against it in a single command:

$ cat dogs.csv | sqlite-utils memory - "select name, age from stdin"

See the full CLI documentation for comprehensive coverage of many more commands.

Using as a library

You can also import sqlite_utils and use it as a Python library like this:

import sqlite_utils
db = sqlite_utils.Database("demo_database.db")
# This line creates a "dogs" table if one does not already exist:
db["dogs"].insert_all([
    {"id": 1, "age": 4, "name": "Cleo"},
    {"id": 2, "age": 2, "name": "Pancakes"}
], pk="id")

Check out the full library documentation for everything else you can do with the Python library.

Related projects

Datasette: A tool for exploring and publishing data
csvs-to-sqlite: Convert CSV files into a SQLite database
db-to-sqlite: CLI tool for exporting a MySQL or PostgreSQL database as a SQLite file
dogsheep: A family of tools for personal analytics, built on top of sqlite-utils

simonw / sqlite-utils