encoding/gb : create a lightweight package to replace encoding/binary to convert between go data and []byte freely

vipally commented 6 years ago

see https://groups.google.com/forum/#!topic/golang-nuts/Wk8qDdhbtUM

My trouble is that when I want to transport a go data on network between golang apps, but I can't find a perfect way to achieve this work. encoding/binary does not support enough types(eg:string), encoding/gob encoding too many byte numbers, and protocol buffer is not easy to use freely. So I have to design a new serialization package to fix this problem. My solution is to improve from std.binary, and design goal is to take both advantages of std.binary and gob. Finally I achieve it. Here is the main feature of this package.

as light-weight as std.binary
with full-type support like gob.
as high-performance as std.binary and gob.
encoding with fewer bytes than std.binary and gob.
use RegStruct to improving performance of struct encoding/decoding
recommended using in net protocol serialization and DB serialization usage: if bytes, err := binary.Encode(&data, nil); err==nil{ //... } if err := binary.Decode(bytes, &data); err==nil{ //... }

You can find the project here: https://github.com/vipally/binary I think it is a better way to instead std.binary to encoding/decoding between go data and byte slice. So I want to make it a std package.

vipally commented 6 years ago

benchmark disable serializer check BenchmarkStdWriteStruct-4 1000000 2107 ns/op 35.59 MB/s BenchmarkWriteStruct-4 500000 3566 ns/op 20.75 MB/s BenchmarkWriteRegedStruct-4 1000000 1415 ns/op 52.29 MB/s BenchmarkEncodeStruct-4 500000 3484 ns/op 21.24 MB/s BenchmarkEncodeRegedStruct-4 1000000 1300 ns/op 56.92 MB/s BenchmarkStdReadStruct-4 1000000 1416 ns/op 52.96 MB/s BenchmarkReadStruct-4 300000 4610 ns/op 16.05 MB/s BenchmarkReadRegedStruct-4 1000000 1511 ns/op 48.97 MB/s BenchmarkDecodeStruct-4 300000 4406 ns/op 16.79 MB/s BenchmarkDecodeRegedStruct-4 1000000 1319 ns/op 56.10 MB/s BenchmarkStdWriteInt1000-4 100000 23901 ns/op 167.35 MB/s BenchmarkWriteInt1000-4 50000 27321 ns/op 146.48 MB/s BenchmarkEncodeInt1000-4 50000 26401 ns/op 151.58 MB/s BenchmarkStdReadInt1000-4 100000 19461 ns/op 205.54 MB/s BenchmarkReadInt1000-4 50000 32481 ns/op 123.21 MB/s BenchmarkDecodeInt1000-4 50000 29561 ns/op 135.38 MB/s BenchmarkStdWriteString unsupported BenchmarkStdWriteString-4 2000000000 0.00 ns/op BenchmarkWriteString-4 5000000 257 ns/op 505.41 MB/s BenchmarkEncodeString-4 10000000 164 ns/op 788.79 MB/s BenchmarkStdReadString unsupported BenchmarkStdReadString-4 2000000000 0.00 ns/op BenchmarkReadString-4 10000000 230 ns/op 565.19 MB/s BenchmarkDecodeString-4 10000000 142 ns/op 909.68 MB/s BenchmarkPutUvarint32-4 50000000 27.0 ns/op 148.14 MB/s BenchmarkPutUvarint64-4 20000000 65.2 ns/op 122.79 MB/s

vipally commented 6 years ago

benchmark enable serializer check BenchmarkStdWriteStruct-4 1000000 2101 ns/op 35.70 MB/s BenchmarkWriteStruct-4 500000 3714 ns/op 19.92 MB/s BenchmarkWriteRegedStruct-4 1000000 1549 ns/op 47.77 MB/s BenchmarkEncodeStruct-4 500000 3502 ns/op 21.13 MB/s BenchmarkEncodeRegedStruct-4 1000000 1324 ns/op 55.89 MB/s BenchmarkStdReadStruct-4 1000000 1430 ns/op 52.44 MB/s BenchmarkReadStruct-4 300000 4796 ns/op 15.43 MB/s BenchmarkReadRegedStruct-4 1000000 1615 ns/op 45.82 MB/s BenchmarkDecodeStruct-4 300000 4480 ns/op 16.52 MB/s BenchmarkDecodeRegedStruct-4 1000000 1316 ns/op 56.23 MB/s BenchmarkStdWriteInt1000-4 100000 23851 ns/op 167.71 MB/s BenchmarkWriteInt1000-4 50000 27261 ns/op 146.80 MB/s BenchmarkEncodeInt1000-4 50000 26541 ns/op 150.78 MB/s BenchmarkStdReadInt1000-4 100000 19501 ns/op 205.12 MB/s BenchmarkReadInt1000-4 50000 32681 ns/op 122.45 MB/s BenchmarkDecodeInt1000-4 50000 29621 ns/op 135.10 MB/s BenchmarkStdWriteString unsupported BenchmarkStdWriteString-4 2000000000 0.00 ns/op BenchmarkWriteString-4 5000000 257 ns/op 505.81 MB/s BenchmarkEncodeString-4 10000000 167 ns/op 776.08 MB/s BenchmarkStdReadString unsupported BenchmarkStdReadString-4 2000000000 0.00 ns/op BenchmarkReadString-4 10000000 227 ns/op 571.90 MB/s BenchmarkDecodeString-4 10000000 144 ns/op 898.98 MB/s BenchmarkPutUvarint32-4 50000000 26.8 ns/op 149.36 MB/s BenchmarkPutUvarint64-4 20000000 65.3 ns/op 122.60 MB/s

vipally commented 6 years ago

improve encode uint32 BenchmarkStdWriteStruct-4 500000 2482 ns/op 30.22 MB/s BenchmarkWriteStruct-4 300000 4276 ns/op 17.30 MB/s BenchmarkWriteRegedStruct-4 1000000 1723 ns/op 42.95 MB/s BenchmarkEncodeStruct-4 300000 4146 ns/op 17.84 MB/s BenchmarkEncodeRegedStruct-4 1000000 1543 ns/op 47.96 MB/s BenchmarkStdReadStruct-4 1000000 1693 ns/op 44.30 MB/s BenchmarkReadStruct-4 300000 5580 ns/op 13.26 MB/s BenchmarkReadRegedStruct-4 1000000 1794 ns/op 41.25 MB/s BenchmarkDecodeStruct-4 300000 5400 ns/op 13.70 MB/s BenchmarkDecodeRegedStruct-4 1000000 1532 ns/op 48.30 MB/s BenchmarkStdWriteInt1000-4 50000 26441 ns/op 151.28 MB/s BenchmarkWriteInt1000-4 50000 23881 ns/op 167.58 MB/s BenchmarkEncodeInt1000-4 100000 22871 ns/op 174.98 MB/s BenchmarkStdReadInt1000-4 100000 22611 ns/op 176.90 MB/s BenchmarkReadInt1000-4 50000 38962 ns/op 102.71 MB/s BenchmarkDecodeInt1000-4 50000 33821 ns/op 118.33 MB/s BenchmarkStdWriteString unsupported BenchmarkStdWriteString-4 2000000000 0.00 ns/op BenchmarkWriteString-4 5000000 266 ns/op 487.23 MB/s BenchmarkEncodeString-4 10000000 173 ns/op 749.24 MB/s BenchmarkStdReadString unsupported BenchmarkStdReadString-4 2000000000 0.00 ns/op BenchmarkReadString-4 5000000 241 ns/op 538.94 MB/s BenchmarkDecodeString-4 10000000 153 ns/op 844.66 MB/s BenchmarkPutUvarint32-4 50000000 33.7 ns/op 118.76 MB/s BenchmarkPutUvarint64-4 20000000 82.7 ns/op 96.79 MB/s

vipally / go

encoding/gb : create a lightweight package to replace encoding/binary to convert between go data and []byte freely #5