According to Charles, the Peewee ORM is currently going through a major 3.0 rewrite and isn't accepting many (any?) new features for the 2.x line. Because of this, I made Herman as a light fork of Peewee (2.8.1) implementing some syntactic sugar and minor tweaks I've liked from other ORMs.
This is mostly for my company's internal use, but anyone else is free to use it. I'm maintaining API compatibility as a drop in replacement for peewee 2.x.
.. image:: https://api.travis-ci.org/keredson/peewee.png?branch=master :target: https://travis-ci.org/keredson/peewee
.. code-block:: bash
sudo pip uninstall peewee sudo pip install herman
Or to upgrade:
.. code-block:: bash
sudo pip install herman --upgrade
There is no herman
package. Herman represents itself as peewee (with the same peewee __version__
) as a drop-in replacement. To check if you're running Peewee or Herman, check for peewee.__herman_version__
.
The SQL generated by an ORM isn't just an artifact to be consumed by the database. Developers need to be able to read it as well. The more readable it is and the closer it is to what one would naturally write, the less time it takes to manually evaluate / debug it.
For example, this query:
.. code-block:: python
Article.select()
.join(Person, join_type=pw.JOIN.LEFT_OUTER, on=(Article.author==Person.id))
.where(Article.author==derek)
In Peewee generates this:
.. code-block:: sql
SELECT "t1"."id", "t1"."title", "t1"."author_id", "t1"."editor_id"
FROM "article" AS t1
LEFT OUTER JOIN "person" AS t2 ON ("t1"."author_id" = "t2"."id")
WHERE ("t1"."author_id" = ?
Whereas Herman generates:
.. code-block:: sql
SELECT a1.id, a1.title, a1.author_id, a1.editor_id
FROM article AS a1
LEFT OUTER JOIN person AS p1 ON (a1.author_id = p1.id)
WHERE (a1.author_id = ?)
You'll notice that in Herman:
.. code-block:: python
Article.select().first()
In Peewee generates this:
.. code-block:: sql
SELECT "t1"."id", "t1"."title", "t1"."author_id", "t1"."editor_id" FROM "article" AS t1
Where in Herman this is generated:
.. code-block:: sql
SELECT a1.id, a1.title, a1.author_id, a1.editor_id FROM article AS a1 LIMIT 1
Notice the additional "LIMIT 1" in Herman. Peewee 2.8.1 rather loads entire result set and returns this first object. (It's my understanding future versions of Peewee will also add a "LIMIT 1".)
In Peewee you can create table aliases as a variable outside your query. For example:
.. code-block:: python
author_table = Person.alias()
Article.select()
.join(author_table)
.where(author_table.name=='Derek')
This gets you a very respectable:
.. code-block:: sql
SELECT "t1"."id", "t1"."title", "t1"."author_id", "t1"."editor_id"
FROM "article" AS t1
INNER JOIN "person" AS t2 ON ("t1"."author_id" = "t2"."id")
WHERE ("t2"."name" = ?)
But in some queries it's very useful to specify what the aliases are. In Herman you can do this:
.. code-block:: python
author_table = Person.alias('author')
Article.select()
.join(author_table)
.where(author_table.name=='Derek')
Which generates this:
.. code-block:: sql
SELECT a1.id, a1.title, a1.author_id, a1.editor_id
FROM article AS a1
INNER JOIN person AS author ON (a1.author_id = author.id)
WHERE (author."name" = ?)
These two statements are equivalent in Herman:
.. code-block:: python
author_table = Person.alias('author')
author_table = Person.as_('author')
And because aliases are named, you're no longer required to use the external variable:
.. code-block:: python
Article.select()
.join(Person.as_('author'))
.where(Person.as_('author').name=='Derek')
Whether in-lining the aliases makes it easier or harder to read is entirely dependent on the query and the code block it's in, but it's good to have the option. Both are supported in Herman.
If a query is otherwise invalid and there is only one possible interpretation of the query author's intent, automatically tie the un-aliased column to the appropriate alias. For example:
.. code-block:: python
Person.alias('xyz').select().where(Person.name == 'Derek')
Generates invalid SQL in Peewee (which is run on the database, which throws an exception):
.. code-block:: sql
SELECT "t1"."id", "t1"."name", "t1"."parent_id"
FROM "person" AS t1 WHERE ("t2"."name" = ?)
peewee.OperationalError: no such column: t2.name
But in Herman it's valid:
.. code-block:: sql
SELECT xyz.id, xyz."name", xyz.parent_id
FROM person AS xyz WHERE (xyz."name" = ?)
This would NOT work if more than one Person alias were included in the query.
Peewee will generate and run on the database SQL it knows is invalid. For example:
.. code-block:: python
Person.select().where(Article.title == 'xyz').first()
In Peewee will throw a database error:
.. code-block:: sql
SELECT "t1"."id", "t1"."name", "t1"."parent_id"
FROM "person" AS t1 WHERE ("t2"."title" = ?)
peewee.OperationalError: no such column: t2.title
This isn't good for two reasons. First, I don't like relying on the database to catch easily detectable errors for us. Second, the error is opaque and specific to the internal implementation details of peewee (the "t2").
Herman on the other hand will not generate the SQL at all, instead raising:
.. code-block:: python
peewee.ProgrammingError: <class '__main__.Article'> is not a part of this query
The get() method in Peewee adds a "LIMIT 1" and returns the first object. I feel this is incorrect behavior. The difference between first() and get() is get() should assert that only one matching record exists. (This is something Django got right IMO.) If my query criteria hasn't correctly isolated a unique row the ORM should throw an exception.
This is why Herman added a "LIMIT 2" to the SQL genned from get(), and does a check on the number of object returned. For example:
.. code-block:: python
derek = Person.create(name='Derek')
callie = Person.create(name='Callie')
Person.select().get()
Will throw:
.. code-block:: sql
peewee.DataError: Too many instances matching query exist:
SQL: SELECT p1.id, p1."name", p1.parent_id FROM person AS p1 LIMIT 2
Rather than returning a random Person object selected by the database.
In Herman, this:
.. code-block:: python
Person.ALL
Is the same as this:
.. code-block:: python
Person.select()
It's just a nomenclature I preferred from Django. I made it uppercase to prevent conflicts with any columns named "all", and to highlight that it's effectively a constant.
If I have a set of models:
.. code-block:: sql
class Person(pw.Model):
name = pw.CharField()
class Article(pw.Model):
title = pw.CharField()
author = pw.ForeignKeyField(db_column='author_id', rel_model=Person, to_field='id')
editor = pw.ForeignKeyField(db_column='editor_id', rel_model=Person, to_field='id', related_name='edited_articles', null=True)
class Reply(pw.Model):
text = pw.CharField()
article = pw.ForeignKeyField(db_column='article_id', rel_model=Article, to_field='id')
And I want to do something fancy like get all replies with their articles and authors and editors, in Peewee I have to do something like this:
.. code-block:: python
author_table = Person.alias() editor_table = Person.alias() replies = Reply.select(Reply, author_table, editor_table) .join(Article) .join(author_table, join_type=pw.JOIN.LEFT_OUTER, on=(author_table==Article.author)) .switch(Article) .join(editor_table, join_type=pw.JOIN.LEFT_OUTER, on=(editor_table==Article.editor)) .where(author_table.name=="Derek")
Which is all sorts of complicated. For instance:
Herman offers a simpler syntax:
.. code-block:: python
Reply.ALL .plus(Reply.article, Article.author.as('author')) .plus(Reply.article, Article.editor) .where(Person.as('author').name=="Derek")
Which generates the same SQL:
.. code-block:: sql
SELECT r1.id, r1."text", r1.article_id, a1.id, a1.title, a1.author_id, a1.editor_id, author.id, author."name", author.parent_id, p1.id, p1."name", p1.parent_id FROM reply AS r1 LEFT OUTER JOIN article AS a1 ON (r1.article_id = a1.id) LEFT OUTER JOIN person AS author ON (a1.author_id = author.id) LEFT OUTER JOIN person AS p1 ON (a1.editor_id = p1.id) WHERE (author."name" = ?)
The plus() method takes a variable number of ForeignKeyField objects which represent a path away from the primary query object (Reply in this case). For example:
.. code-block:: python
Reply.ALL.plus(Reply.article)
Gets all the replies with their associated articles.
.. code-block:: python
Reply.ALL.plus(Reply.article, Article.editor)
Gets all the replies with their associated articles and all the articles editors. Note that "Reply.article" is a foreign key from Reply to Article, and Article.editor is a foreign key from Article to Person. The list of foreign keys must create a logical path where the "to" type of one FK matches the "from" type of the next FK.
Following the same path twice is harmless. For instance:
.. code-block:: python
Reply.ALL.plus(Reply.article).plus(Reply.article)
Is no different than calling plus() once. This is why we can specify multiple paths that have some overlap, for example:
.. code-block:: python
Reply.ALL .plus(Reply.article, Article.author) .plus(Reply.article, Article.editor)
Which will join on the article table only once.
You can also alias your joined tables with:
.. code-block:: python
Article.author.as_('author')
Which allows you to reference it later in your conditional:
.. code-block:: python
.where(Person.as_('author').name=="Derek")
Herman's plus() also supports following foreign keys from one-to-many. Like:
.. code-block:: python
Article.ALL.plus(Reply.article)
Which internally does a prefetch to populate the article with all of its replies. There will be O(k) SQL statements executed, where k
is the number of two-many relationships. All of these queries will be grouped into one transaction to guarantee correctness.
IMPORTANT:
Remember that foreign keys represent edges in your object graph, and a call to `plus(*edges)` tells Herman to include that path from the object graph in your query.
This semantics for plus() have been co-opted from the DKO <https://github.com/keredson/DKO>
_ project, which I authored for my former employer. DKO's version of this syntax has been in broad production use since 2010 by hundreds of developers, accessing some of the largest (billions of rows) conventional relational databases that exist.
If you call:
.. code-block:: python
len(Article.select())
Peewee this will load a list of all objects, permanently cache said list, and then call len()
on that cache.
In Herman this will call count()
on the database and return the resulting integer. It does not build the list of objects in python nor cache anything. However, for backwards compatability, if something else has already populated the cached results of the query, it will call len()
on that.
The semantics behind Peewee's circular foreign key dependencies <http://docs.peewee-orm.com/en/latest/peewee/model.html#circular-foreign-key-dependencies>
_ get kind of unwieldy when you have more than a few models (and they're spread over multiple files). This is because the DeferredRelation object has to be defined, used, then the other model defined in another file, then set_model has to be called on the original, and then you're left with the object reference dangling around that has no purpose. IE the example in the docs:
.. code-block:: python
DeferredTweet = DeferredRelation()
class User(Model): username = CharField()
favorite_tweet = ForeignKeyField(DeferredTweet, null=True)
class Tweet(Model): message = TextField() user = ForeignKeyField(User, related_name='tweets')
DeferredTweet.set_model(Tweet)
Ours happens all in the model definition with an optional parameter given to DeferredRelation. Like:
.. code-block:: python
class User(Model): username = CharField()
favorite_tweet = ForeignKeyField(DeferredRelation('Tweet'), null=True)
class Tweet(Model): message = TextField() user = ForeignKeyField(User, related_name='tweets')
This removes the need for the extra variable in the global namespace and the coordination of it over multiple files. And since the parameter is optional, it is fully backwards-compatible with the old syntax.
Our patch for this has been incorporated upstream, so this is forwards-compatible too, following Peewee's next release.
If you try to do a IN operation on an empty list:
.. code-block:: python
User.select().where(User.id << [])
Peewee will generate the following SQL:
.. code-block:: sql
SELECT "t1"."id", "t1"."username" FROM "user" AS t1 WHERE ("t1"."id" IN ())
Which the database will reject as invalid, throwing an exception. We instead generate a "false" statement:
.. code-block:: sql
SELECT u1.id, u1.username FROM "user" AS u1 WHERE (0 = 1)
So you don't have to manually test for empty lists every time you use a SQL IN.
Peewee is a simple and small ORM. It has few (but expressive) concepts, making it easy to learn and intuitive to use.
tons of extensions available in the playhouse <http://docs.peewee-orm.com/en/latest/peewee/playhouse.html>
_
Postgresql HStore, JSON, arrays and more <http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#postgres-ext>
_SQLite full-text search, user-defined functions, virtual tables and more <http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#sqlite-ext>
_Schema migrations <http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#migrate>
and model code generator <http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#pwiz>
Connection pool <http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#pool>
_Encryption <http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#sqlcipher-ext>
_and much, much more... <http://docs.peewee-orm.com/en/latest/peewee/playhouse.html>
_New to peewee? Here is a list of documents you might find most helpful when getting started:
Quickstart guide <http://docs.peewee-orm.com/en/latest/peewee/quickstart.html#quickstart>
_ -- this guide covers all the essentials. It will take you between 5 and 10 minutes to go through it.Guide to the various query operators <http://docs.peewee-orm.com/en/latest/peewee/querying.html#query-operators>
_ describes how to construct queries and combine expressions.Field types table <http://docs.peewee-orm.com/en/latest/peewee/models.html#field-types-table>
_ lists the various field types peewee supports and the parameters they accept.For flask helpers, check out the flask_utils extension module <http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#flask-utils>
. You can also use peewee with the popular extension flask-admin <https://flask-admin.readthedocs.io/en/latest/>
to provide a Django-like admin interface for managing peewee models.
Defining models is similar to Django or SQLAlchemy:
.. code-block:: python
from peewee import *
from playhouse.sqlite_ext import SqliteExtDatabase
import datetime
db = SqliteExtDatabase('my_database.db')
class BaseModel(Model):
class Meta:
database = db
class User(BaseModel):
username = CharField(unique=True)
class Tweet(BaseModel):
user = ForeignKeyField(User, related_name='tweets')
message = TextField()
created_date = DateTimeField(default=datetime.datetime.now)
is_published = BooleanField(default=True)
Connect to the database and create tables:
.. code-block:: python
db.connect()
db.create_tables([User, Tweet])
Create a few rows:
.. code-block:: python
charlie = User.create(username='charlie')
huey = User(username='huey')
huey.save()
# No need to set `is_published` or `created_date` since they
# will just use the default values we specified.
Tweet.create(user=charlie, message='My first tweet')
Queries are expressive and composable:
.. code-block:: python
# A simple query selecting a user.
User.get(User.username == 'charles')
# Get tweets created by one of several users. The "<<" operator
# corresponds to the SQL "IN" operator.
usernames = ['charlie', 'huey', 'mickey']
users = User.select().where(User.username << usernames)
tweets = Tweet.select().where(Tweet.user << users)
# We could accomplish the same using a JOIN:
tweets = (Tweet
.select()
.join(User)
.where(User.username << usernames))
# How many tweets were published today?
tweets_today = (Tweet
.select()
.where(
(Tweet.created_date >= datetime.date.today()) &
(Tweet.is_published == True))
.count())
# Paginate the user table and show me page 3 (users 41-60).
User.select().order_by(User.username).paginate(3, 20)
# Order users by the number of tweets they've created:
tweet_ct = fn.Count(Tweet.id)
users = (User
.select(User, tweet_ct.alias('ct'))
.join(Tweet, JOIN.LEFT_OUTER)
.group_by(User)
.order_by(tweet_ct.desc()))
# Do an atomic update
Counter.update(count=Counter.count + 1).where(
Counter.url == request.url)
Check out the example app <http://docs.peewee-orm.com/en/latest/peewee/example.html>
_ for a working Twitter-clone website written with Flask.
Check the documentation <http://docs.peewee-orm.com/>
_ for more examples.
Specific question? Come hang out in the #peewee channel on irc.freenode.net, or post to the mailing list, http://groups.google.com/group/peewee-orm . If you would like to report a bug, create a new issue <https://github.com/coleifer/peewee/issues/new>
_ on GitHub.
.. image:: http://media.charlesleifer.com/blog/photos/wat.jpg
I've written a number of blog posts about building applications and web-services with peewee (and usually Flask). If you'd like to see some real-life applications that use peewee, the following resources may be useful:
Building a note-taking app with Flask and Peewee <http://charlesleifer.com/blog/saturday-morning-hack-a-little-note-taking-app-with-flask/>
as well as Part 2 <http://charlesleifer.com/blog/saturday-morning-hacks-revisiting-the-notes-app/>
and Part 3 <http://charlesleifer.com/blog/saturday-morning-hacks-adding-full-text-search-to-the-flask-note-taking-app/>
_.Analytics web service built with Flask and Peewee <http://charlesleifer.com/blog/saturday-morning-hacks-building-an-analytics-app-with-flask/>
_.Personalized news digest (with a boolean query parser!) <http://charlesleifer.com/blog/saturday-morning-hack-personalized-news-digest-with-boolean-query-parser/>
_.Using peewee to explore CSV files <http://charlesleifer.com/blog/using-peewee-to-explore-csv-files/>
_.Structuring Flask apps with Peewee <http://charlesleifer.com/blog/structuring-flask-apps-a-how-to-for-those-coming-from-django/>
_.Creating a lastpass clone with Flask and Peewee <http://charlesleifer.com/blog/creating-a-personal-password-manager/>
_.Building a web-based encrypted file manager with Flask, peewee and S3 <http://charlesleifer.com/blog/web-based-encrypted-file-storage-using-flask-and-aws/>
_.Creating a bookmarking web-service that takes screenshots of your bookmarks <http://charlesleifer.com/blog/building-bookmarking-service-python-and-phantomjs/>
_.Building a pastebin, wiki and a bookmarking service using Flask and Peewee <http://charlesleifer.com/blog/dont-sweat-small-stuff-use-flask-blueprints/>
_.Encrypted databases with Python and SQLCipher <http://charlesleifer.com/blog/encrypted-sqlite-databases-with-python-and-sqlcipher/>
_.Dear Diary: An Encrypted, Command-Line Diary with Peewee <http://charlesleifer.com/blog/dear-diary-an-encrypted-command-line-diary-with-python/>
_.Query Tree Structures in SQLite using Peewee and the Transitive Closure Extension <http://charlesleifer.com/blog/querying-tree-structures-in-sqlite-using-python-and-the-transitive-closure-extension/>
_.