Jaymon / prom

A PostgreSQL or SQLite orm for Python
MIT License
22 stars 4 forks source link

a small map/reduce like iterator #30

Closed Jaymon closed 7 years ago

Jaymon commented 8 years ago

it would be cool to have something that could break apart a table and each thread would work on a part of the table, so the idea would be:

# imagine the user table has 100 rows and is managed by the User prom.Orm child

def names_that_begin_with_f(u):
    return 1 if u.name.tolower().startswith("f") else 0

p = MR(threads=10, orm_class=User, target=names_that_begin_with_f)

p.run()

What the run() method will do is find out how many rows are in the table, and create how many threads were requested and then divide the table count by number of threads and tell each thread to work on a certain section of the table, then each thread will go through their section using all() and return the results.

There are times where we want to do some simple crunching and this would speed that up quite a bit.

Jaymon commented 7 years ago

This looks like a job for the multiprocessing module