search

OptimalBits / bull

Premium Queue package for handling distributed jobs and messages in NodeJS.
Other
15.57k stars 1.43k forks source link

Support for alternative serialization techniques such as MessagePack #805

Open sloops77 opened 6 years ago

sloops77 commented 6 years ago

When there is a large payload JSON can be pretty verbose. Support for alternative serialization techniques for the jobData and/or serialization middlewares to support compression would be a great enhancement

manast commented 6 years ago

MessagePack would not be so suitable since you need to define a schema which makes the whole thing quite complex. It would need to be a schemaless format, also ideally at least as fast as JSON. Last I checked I could not find any...

sloops77 commented 6 years ago

messagepack is schemaless. https://github.com/pgriess/node-msgpack

sloops77 commented 6 years ago

also something interesting to use would be snappy. https://github.com/kesla/node-snappy

manast commented 6 years ago

oh yes, sorry I mixed messagepack with protobuffers. Are your biggest concern space usage in redis, bandwidth or speed? When I research this some time ago I concluded that the fastest format was JSON, although not the most compact one.

manast commented 6 years ago

Maybe the best way is as you propose, just keep using JSON, which is very fast in node and just compress the resulting string before storing in redis. We could just add a new option when creating a queue, 'compression: true' and compress all data in that case.

manast commented 6 years ago

or even better:

compression: {
  encode: myencoder,
  decode: mydecoder
}
btd commented 6 years ago

I think it is not right to add any possible feature user asked in 'core'. It is a very solid path to package bloat and support nightmare. I think @manast last suggestion the most elegant solution possible for now.