Closed sampaioletti closed 8 years ago
Yes, FlatBuffers skips the parsing step, but is therefore slower than accessing native objects. I don't think it is fair to compare against native objects without including the time to parse/create them, no, and also not realistic, since you usually can't skip that step.
The only time when native objects are truly faster is when you have to access the data many times. In that case keeping things in FlatBuffers will actually be slower overal. If that is part of the performance critical part of your code, then you should copy data out of FlatBuffers.
Thanks @gwvo I didn't want to cloud the post with to much more info (was already too long) but at first I ran the test including the 'parsing' step and even at 10 items the test showed fb slower overall
fbAve:0.9600000000000364ms this:0.9600000000000364ms
jsAve:0.035000000000763976ms this:0.035000000000763976ms
And was slower at different levels of items in this scenario (100, 10000, etc)
I've updated the plunkr to reflect that.
I was merely trying to isolate out that the problem wasn't in the 'parsing' step.
I would only like to bring it up as we love the library and wanted to see if people far better with js than us had any thoughts on optimizing (or pointed out our tests were bogus). We will look at it and have done some CPU profiling and if we can reduce the number of function calls (maybe some arrays of offsets so we don't have to call functions to get offsets every time) it might help. But not trying to be critical just trying to open a discussion. The reduction in overhead on our server for the serialization step makes the reduced performance on the client more than worth it (since they process for free haha), and I guess thats a core point of the library.
I'll close this out so you don't have to think about it since you are aware that the access is less performant than native and its not really an issue, and if we come up with any performance increases in our work we'll shoot you a pull.
Thanks again!
It is possible that the JS implementation is particularly slow. I designed FlatBuffers originally for C++, and it may not fit JS quite as well (i.e. all the indirections confusing the JIT). Maybe the JS implementation can be optimized, or maybe a different API (that actually unpacks into JS objects would be better for this language.
I believe @evanw (who implemented the JS portion) at some point experimented with an object based API, though I am not sure where that went. @evanw: can you point us to that? Maybe someone would like to continue it.
Yeah i looked at the the Object API posts you mention, but when I went to @evanw s github all I could find was the normal flatbuffers.js file. If we can get ahold of his api I'd be happy to take a look at it and run the same tests to compare the differences, and look at doing a fork with the alternate api so we can see how it goes (since as he mentioned they are using a different library now).
Thanks
@gwvo had a thought while looking at some other stuff we were working on and based on your c++ comment. I might try next week if I have some time to run your c++ code through emscripten and see how efficient the generated js is given this scenario. I did a project a while back where we had a small utility library that we converted and the performance really surprised us. API was ugly from a js perspective, but it worked quite well. I don't think that that is a good long term strategy but if it proves more efficient it might give us a starting point for optimization in the existing code for 'free'. Now that I say that we could even try something similar given the golang generated code and gopherjs project. Both might just provide good test data as a starting point, Just a thought.
Curiosity got the best of me so I did a quick project with gopherJS (added it to plunkr), it wasn't faster in any case. However, an interesting thing happened I was having trouble with the gopherJS returning a string since the GO version returns a []byte...so I decided to just do numbers and eliminate looking up the SensorId string value, Instant difference in the 'official' implementation. Now for very small collections (say 10) it is barely slower and on some iterations actually is faster than the json (it is a little slow on first run as you would expect, but optimizes quickly). I think at that level I still have some problems with the way i'm running the test and that probably throws in some GC that makes the results inconclusive, but its night and day vs running with a string. Once you get over 100 items in the collection it becomes faster than the json version, go with 1000-100000 and fb its substantially faster and stays that way (meaning combined time, access is still slower but the tradeoff works now as you would expect). (updated plunkr with this)
So I still want to run the emscrpiten if I get time next week, but this is excellent news for the application I'm working on as it consists of mostly sending thousands of numeric values for realtime graphing purposes. We will just avoid Strings as much as possible or break them out on first pass into objects.
Another test we could run would be looking up multiple numeric values to make sure it scales...but that is another day.
So I would say from a JS optimization path we should look at how we are handling strings, the cost might be in the UTF8/16 decision tree or elsewhere..but I think it gives us a starting point. I'll continue to look into it, but I'm pretty pleased with the results. Again just my 2 cents.
Sorry to continue to post to a closed issue..but at them moment I think its a safe place to house my ramblings.
Thanks, that's some useful testing! Yes, indeed, looks like the manual UTF-8 decoding is slow.. I wonder how we can speed that up.
Not my wheelhouse, but I'll run some tests..quick google gives a couple ideas..perhaps http://ecmanaut.blogspot.com/2006/07/encoding-decoding-utf8-in-javascript.html?m=1 is worth a try. Glad to be of help. We try and give back if we are using a project. Between the performance and header only c++ this project fits us perfectly (tons of real time data, cross platform and embedded).
Hmm, using built-in functions would be great, bit worried that this is a two step conversion over octets, that might end up generating more garbage.. but who knows it is still faster than what we have. Also would need to check whether that works on Node.js.
Good to hear FlatBuffers works so well for you :)
I second what @sampaioletti is seeing. This is the small loadtest example i've created.
export class FlatbufferDemo extends React.Component {
constructor(props) {
super(props);
this.state = {};
}
async componentDidMount() {
cnsole.info('flatbuffers: ', flatbuffers);
const serialized = this.createMessage(1, 'Hello my name is Gagan');
const { index, text } = this.parseMessage(serialized);
cnsole.info('parsed: ', { index, text });
const N = 10000;
this.loadTestFb(N);
this.loadTestJson(N);
}
loadTestFb = (N) => {
let array;
{
const startTime = new Date().getTime();
array = xrange(0, N).toArray().map(x => this.createMessage(x, this.getStr()));
cnsole.info('Flatbuffer: Time taken in serializing: ', new Date().getTime() - startTime);
}
{
const startTime = new Date().getTime();
const parsed = array.map(a => this.parseMessage(a));
cnsole.info('Flatbuffer: Time taken in parsing: ', new Date().getTime() - startTime);
cnsole.info(array.length, array[10]);
cnsole.info(parsed.length, parsed[10]);
}
};
loadTestJson = (N) => {
let array;
{
const startTime = new Date().getTime();
array = xrange(0, N).toArray().map(x => JSON.stringify({ index: x, str: this.getStr() }));
cnsole.info('Json: Time taken in serializing: ', new Date().getTime() - startTime);
}
{
const startTime = new Date().getTime();
const parsed = array.map(a => JSON.parse(a));
cnsole.info('Json: Time taken in parsing: ', new Date().getTime() - startTime);
cnsole.info(array.length, array[10]);
cnsole.info(parsed.length, parsed[10]);
}
};
getStr = () => {
return 'Hello ' + Math.ceil(1000 * Math.random()) + ' world ' + Math.random();
};
createMessage = (index, str) => {
const builder = new flatbuffers.Builder(100);
const s = builder.createString(str);
helo.Message.startMessage(builder);
helo.Message.addIndex(builder, new flatbuffers.Long(index, index));
helo.Message.addText(builder, s);
const orc = helo.Message.endMessage(builder);
builder.finish(orc);
return builder.dataBuffer();
};
parseMessage = (serialized) => {
const parsed = helo.Message.getRootAsMessage(serialized);
const index = parsed.index().high;
const text = parsed.text();
return { index, text };
};
render() {
return (
<View>
</View>
);
}
}
Run it for 10,000 iterations and you get this: Flatbuffer: Time taken in serializing: 50 Flatbuffer: Time taken in parsing: 18 Json: Time taken in serializing: 18 Json: Time taken in parsing: 7
Maybe JSON library is too damn fast already !
You example json is just two fields { index, str } and is not characteristic of a typically json with nested objects and more complex parsing. So I would advise to make the data format a bit more complex to really test the difference. Also, you use a long for the index, which I don't find necessary for 10,000 iterations, so you create more work for FB because it has to do extra stuff to process a long in JS.
In the FB parsing, you also parse the buffer and then stick the values into a JSON object. So you are doing extra work that you would typically not perform when using FB. Generally you use work off the buffer and get the data you want when you need it. Perhaps you would put the data into another json object, but I wouldn't count that part as part of the FB parsing.
In the FB creation, you are recreating the builder each iteration, which will have overhead of allocating memory for the builder. I would construct one builder and pass it into the createMessage method, and just "clear" it each loop. That would be more typical of real-world usage.
new flatbuffers.Long(index, index)
is also incorrect. The two parameters are the upper and lower bytes of the long, and should not be the same value.
@dbaileychess : thanks for the pointers, will share the same loadtest with more fields and long fix
@gagangoku I'm interested in this. Were you able to do another benchmark?
Sometimes all business systems use FlatBuffers. If the browser wants to connect, it can only use FlatBuffers. 1] The server can first deserialize the FlatBuffers binary into objects, and then call the unpack method to obtain the corresponding entity objects. Then call the Json serialization library of the current programming language, serialize the entity object into Json, and then send it to the browser. The browser receives an instance class that Json parses into an object, and then the instance class at the end of T of as FlatBuffers. In this way, it automatically prompts each attribute. 1} Support unpack programming languages: c #, java, trust, ts, C++, dart, go, python, swift 2} Programming languages that do not support unpack: kotlin, lobster, lua, nim, php 2] It is more troublesome to convert Json to FlatBuffers. First, reverse the Json string into an entity class at the end of T, Then use the equivalent of NET reflection technology, dynamically calling the method of creating the FlatBuffers object, The corresponding attribute in the entity class instance at the end of T is dynamically passed in as a parameter.
Hello,
We have been using FB for a while now in some C++ and Go projects and have had spectacular performance, we now have a project that needs to communicate with those projects from Javascript in the browser. We have been doing some performance testing comparing JSON.parse with FB (totally non-scientific) and are seeing a bottleneck accessing values (using Chrome browser) whereas 'parsing' works as expected. Just wanted to see if anyone had any thoughts. We appreciate the help.
Given an .fbs of
we build 10000 entities in FB
and also build 10000 entities in a JSON string
getRandomIntInclusive is used to try and keep the v8 optimizer guessing;
afterwhich we alternate between the two building 10000 entities and parsing them (skipping the first two runs to account for JIT compilation)
For JSON
For FB
and get a console output like
if we run the time around the 'parsing' step i.e.
FB is orders of magnitude faster (as one would expect), but you loose all you gained when you have to access the values as in the above code. Does this seem like a fair comparison? After testing the 'parsing' step and getting the results we were expecting we were a little shocked by the spread (ran it at different counts with same basic result).
Sorry for the ugly code was just supposed to be a quick test(my JS is weak anyway). Appreciate any thoughts, including ones that start out with no no dummy (:
quick plunkr to illustrate http://embed.plnkr.co/dDiWvZ/