search

influxdata / influxdb-comparisons

Code for comparison write ups of InfluxDB and other solutions
MIT License
306 stars 112 forks source link

OpenTSDB serializer speed up #160

Closed Sasasu closed 4 years ago

Sasasu commented 4 years ago 
before: 61M/s
after:  577M/s

test use ./data_gen --format opentsdb --balh balh | pv > /dev/null with i7-8550U and LPDDR3

cmd-influx commented 4 years ago

Mark - Take a look at the change, let me know if you have concerns.

Sasasu commented 4 years ago

I forgot the FieldKeys, now fix

➜ ./data_gen -format opentsdb | head
using random seed 424995067
2020/02/04 15:06:38 Using sampling interval 10s
2020/02/04 15:06:38 Using cardinality of 9
{"metric":"cpu.usage_user","timestamp":1514764800000,"value":60.4660287979619540,"tags":{"hostname":"host_0","region":"ap-northeast-1","datacenter":"ap-northeast-1a","rack":"72","os":"Ubuntu16.10","arch":"x86","team":"CHI","service":"10","service_version":"0","service_environment":"test"}}
{"metric":"cpu.usage_system","timestamp":1514764800000,"value":94.0509088045012476,"tags":{"hostname":"host_0","region":"ap-northeast-1","datacenter":"ap-northeast-1a","rack":"72","os":"Ubuntu16.10","arch":"x86","team":"CHI","service":"10","service_version":"0","service_environment":"test"}}
{"metric":"cpu.usage_idle","timestamp":1514764800000,"value":66.4560053218490481,"tags":{"hostname":"host_0","region":"ap-northeast-1","datacenter":"ap-northeast-1a","rack":"72","os":"Ubuntu16.10","arch":"x86","team":"CHI","service":"10","service_version":"0","service_environment":"test"}}
{"metric":"cpu.usage_nice","timestamp":1514764800000,"value":43.7714187186980155,"tags":{"hostname":"host_0","region":"ap-northeast-1","datacenter":"ap-northeast-1a","rack":"72","os":"Ubuntu16.10","arch":"x86","team":"CHI","service":"10","service_version":"0","service_environment":"test"}}
{"metric":"cpu.usage_iowait","timestamp":1514764800000,"value":42.4637497071265670,"tags":{"hostname":"host_0","region":"ap-northeast-1","datacenter":"ap-northeast-1a","rack":"72","os":"Ubuntu16.10","arch":"x86","team":"CHI","service":"10","service_version":"0","service_environment":"test"}}
{"metric":"cpu.usage_irq","timestamp":1514764800000,"value":68.6823072867109374,"tags":{"hostname":"host_0","region":"ap-northeast-1","datacenter":"ap-northeast-1a","rack":"72","os":"Ubuntu16.10","arch":"x86","team":"CHI","service":"10","service_version":"0","service_environment":"test"}}
{"metric":"cpu.usage_softirq","timestamp":1514764800000,"value":6.5637019217476222,"tags":{"hostname":"host_0","region":"ap-northeast-1","datacenter":"ap-northeast-1a","rack":"72","os":"Ubuntu16.10","arch":"x86","team":"CHI","service":"10","service_version":"0","service_environment":"test"}}
{"metric":"cpu.usage_steal","timestamp":1514764800000,"value":15.6519254732791246,"tags":{"hostname":"host_0","region":"ap-northeast-1","datacenter":"ap-northeast-1a","rack":"72","os":"Ubuntu16.10","arch":"x86","team":"CHI","service":"10","service_version":"0","service_environment":"test"}}
{"metric":"cpu.usage_guest","timestamp":1514764800000,"value":9.6969518914484567,"tags":{"hostname":"host_0","region":"ap-northeast-1","datacenter":"ap-northeast-1a","rack":"72","os":"Ubuntu16.10","arch":"x86","team":"CHI","service":"10","service_version":"0","service_environment":"test"}}
{"metric":"cpu.usage_guest_nice","timestamp":1514764800000,"value":30.0911860585287059,"tags":{"hostname":"host_0","region":"ap-northeast-1","datacenter":"ap-northeast-1a","rack":"72","os":"Ubuntu16.10","arch":"x86","team":"CHI","service":"10","service_version":"0","service_environment":"test"}}