search

Netflix / PigPen

Map-Reduce for Clojure
Apache License 2.0
565 stars 55 forks source link

Tune nippy performance #172

Closed mbossenbroek closed 8 years ago

mbossenbroek commented 8 years ago

cc @ptaoussanis (I updated a second location where freeze & thaw were used)

cc @bcotton

Changes:

Specifically:

  1. freeze {:skip-header? true} - saves 4 bytes per payload + slightly faster. This kills support for data versioning and dynamic compression selection.
  2. freeze {:compressor nil} - should significantly improve freeze performance (~20%) at an insignificant increase to payload size (~5%). A good trade assuming job data doesn't grow into the megabyte range.
  3. thaw {:compressor nil, :encryptor nil} - necessary because of [1]
  4. thaw {:v1-compatibility? false} - good hygiene if you're not thawing any legacy data
cloudbees-pull-request-builder commented 8 years ago

NetflixOSS » PigPen » PigPen-pull-requests #16 SUCCESS This pull request looks good