search

gintas / django-picklefield

A pickled object field for Django
MIT License
180 stars 47 forks source link

Deepcopy could be configurable #28

Closed jarshwah closed 7 years ago

jarshwah commented 7 years ago

We were running into some memory issues with many large pickles being created in a loop. Other than doing really stupid things before picklefield even got the value, we noted that the deepcopy in dbsafe_encode was responsible for generating quite a bit of junk and taking quite a lot of time.

The comments inside the method say:

We use deepcopy() here to avoid a problem with cPickle, where dumps can generate different character streams for same lookup value if they are referenced differently.

This seems reasonable. But how many people are doing string lookups on the result of their pickling? It would be great if we could disable the deepcopying via another kwarg option on the field. Currently, our work around is to replicate the dbsafe_encode in our user code so that it is decoded correctly on the way back from the database.

If you're ok with the idea, I'm happy to put a PR together to address this.

charettes commented 7 years ago

If you're ok with the idea, I'm happy to put a PR together to address this.

@jarshwah sure, please do!