Make lifter able to query any kind of data (LINQ)

At the moment, lifter is focused on querying against Python iterables.

However, there is an opportunity to make it much more than that. Some people, (e.g. on reddit) mentioned Microsoft .NET LINQ as a possible use case for lifter, and I recently stumbled accross pynq, an attempt to implement this in Python.

As you can see in the project wiki, there are indeed some similitude between our API, but LINQ (and therefore, pyqn) goals are much wider than lifter at the moment.

Basically, LINQ (Language Integrated Query) is a way to query any kind of provider, using the same API everytime. A provider can be a collection/python iterable, a REST API, a relational database or basically anything that can return results.

Of course, where not Microsoft, and we're building a package here, not a Python implementation, so we'll probably never reach this kind of clean syntax:

var results =  from u in users
               where u.age < 10 AND u.is_active = True
               select new (u.first_name, u.email);

But imagine if you could query an IMAP mailbox with lifter:


from lifter.backends.imap import ImapBackend, Email

backend = ImapBackend(host='imap.example.com', user='test', password='test')

# Get unread email from INBOX
backend.select(Email).filter(Email.unseen == True, Email.directory == 'Inbox')

Or a REST API:

from lifter.backends.http import RESTBackend, JSONModel

class MyStoreApi(RESTBackend):
    endpoint = 'https://store.example.com/api/'
    content_type = 'json'

class Product(JSONModel):
    path = 'products'

backend = MyStoreApi()
for product in backend.select(Product).all():
    print(product.title, product.description)

Or a SQL Database:

from lifter.backends.sql import SQLiteBackend, SQLModel

class User(SQLModel):
    table_name = 'users'

SQLiteBackend('/path/to/db').select(User).filter(User.is_active == False).count()

And, of course, a Python iterable:


from lifter.backends.python import PythonModel as APIResult

manager = APIResult.load(api_results)
sorted_results = manager.order_by(~APIResult.relevance)

I can imagine an infinite number of use cases for such a set of features. If I sum it up, lifter would grow from an iterable query engine, to a generic query engine.

What does it imply:

[x] All iterable-specific code should be move to a dedicated backend
[x] Queries should be backend independent, and will be intepreted by backend
[x] Backend results (SQL Query, HTTP JSON response, etc.) would be translated in plain python objects after the query is executed, that's probably a job for the Model class.

That would be pretty big change, but also an exciting one because it opens so many possibilities.

agateblue / lifter

Make lifter able to query any kind of data (LINQ) #33