search

weo-edu / store

derived data stores
0 stars 0 forks source link

msgpack #5

Open joshrtay opened 9 years ago

joshrtay commented 9 years ago

Should we use msgpack for the body our message?

ashaffer commented 9 years ago

Hmm maybe. I mean definitely eventually we'll want that, but in the beginning it might be an unnecessary complication? Not sure.

I think the most important property for whatever format we choose will be efficient streaming parsing. I think JSON is somewhat bad at that. Consider a 1GB event log file:

JSON.parse(fs.readFileSync('1gb.log'))

Is probably not going to work very well. Efficient streaming seems like a big priority. Although it looks like there are streaming parsers for JSON and MessagePack.