search

TenKeyAngle / dataturbine

Automatically exported from code.google.com/p/dataturbine
1 stars 0 forks source link

Overhead is very high #21

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?
1. Start a RBNB server
2. Start a RBNB Source generator and configure for 1 hz, 10 channels of 
floating point data. 
3. On another computer start a sink to download from all 10 channels above that 
also prints out the rate and bytes transferred.

What is the expected output? What do you see instead?
The expected output is around 40 bytes of data per second plus _some_ minimal 
amount of overhead. What is observed instead is around 6000 bytes of data per 
second.

What version of the product are you using? On what operating system?
Latest version of RBNB on Windows mainly, but OS doesn't matter.

Please provide any additional information below.
I looked at the data payload contents using Wireshark and observed around 3 to 
4 bytes were used for every '{' and '}' added. The raw data itself was minimal 
compared to the amount of overhead.  Also the channel names were transmitted 
constantly in plain text as part of the data payload. If you use long channel 
names you will experience even more overhead.

This is important to me because I use RBNB for a project at work and am 
producing over 1 GB of raw data per day and need to download some of this data 
over a 10 GB cellphone link to home base and can't afford the overhead required 
to send the data. Reducing the amount of overhead in the data payload would be 
greatly appreciated.

Original issue reported on code.google.com by RyanLind...@gmail.com on 25 Apr 2013 at 8:04

GoogleCodeExporter commented 9 years ago 
The 6000 bytes of data is partly due to the sample rate of 128 (default for 
RBNB source generator), but even if you reduce it to a sample rate of 4 (the 
minimum when using 1 Hz rate) you still see around 900 bytes per second 
compared to the raw content of 160 bytes (10 channels * 4 bytes per channel * 4 
samples per second) which means the overhead ratio is something like 5.6 bytes 
per byte of data which is still too much.

Original comment by RyanLind...@gmail.com on 25 Apr 2013 at 8:39