performance could be improved

[danielhochman]

Potential performance improvements:

• [x] stop using OrderedDicts for model attributes
• [x] circumvent boto's request parsing e.g. #72
• [ ] use ujson to parse request bodies and for attribute (de)serialization
• [ ] enable gzip compression for requests to DynamoDB API
• [ ] allow custom ProjectionExpression in query api to only pull relevant attributes
• [ ] use ProjectionExpression to only select attributes that exist in model

[danielhochman]

cc @joshowen

[danielhochman]

Is the reason for using OrderedDicts purely cosmetic @jlafon?

[joshowen]

We've monkey patched in cyordereddict, and it has performed well (If we have to use OrderedDicts anywhere)

[joshowen]

The logging could be sped up a bit too by letting logger decide when to format log lines eg: log.debug("{0} committing batch operation".format(self.model)) -> log.debug("%s committing batch operation", self.model)

[jlafon]

@danielhochman OrderedDicts are used to make testing easier between Python 2 and 3 (the tests can assert that two dictionaries are equal by iterating over the key, value pairs in a deterministic order). We could remove OrderedDicts and write the tests differently.

[joshowen]

I have the following open, which should speed things up a little: https://github.com/jlafon/PynamoDB/pull/75 (Removes OrderedDict) https://github.com/jlafon/PynamoDB/pull/74 (Removes a bunch of string formatting in some cases)

[jlafon]

Thank you all for contributing. I've fixed up the tests to work with all of the improvements that you have made. All of the changes have been merged into the devel branch. If you have time, please test it to make sure everything works as expected for you.

[joshowen]

https://github.com/jlafon/PynamoDB/pull/78 (Use ujson if its available) https://github.com/jlafon/PynamoDB/pull/77/ (This has a performance improvement in that it does not compute a default value for attributes with non-None values) https://github.com/jlafon/PynamoDB/pull/79 (Add attributes_to_get and consistent_read to more of the API)

[joshowen]

@jlafon I think we've broken the BinaryAttribute. It looks like we're base64 decoding something that is no longer base64 encoded. Did botocore base64 encode binary values?

edit: These are fixed on https://github.com/jlafon/PynamoDB/pull/84

[joshowen]

We're using this session:

from botocore.vendored.requests import Session as BotoRequestsSession
from botocore.vendored.requests.adapters import HTTPAdapter as BotoHTTPAdapter

class OFSession(BotoRequestsSession):
    def __init__(self):
        super(OFSession, self).__init__()
        self.mount('https://', BotoHTTPAdapter(pool_maxsize=100))
        self.mount('http://', BotoHTTPAdapter(pool_maxsize=100))

with the changes in https://github.com/jlafon/PynamoDB/pull/91 to speed up a lot of calls to DynamoDB in greenlets