search

nilsmagnus / grib

Golang GRIB2 parser
Other
57 stars 19 forks source link

Optimize memory-allocations a bit here and there #16

Closed nilsmagnus closed 5 years ago

nilsmagnus commented 5 years ago

Old performance-test results:

go tool pprof -top memprofile.out
File: gribtest.test
Type: alloc_space
Time: Feb 7, 2019 at 10:25pm (CET)
Showing nodes accounting for 9.66GB, 99.32% of 9.73GB total
Dropped 44 nodes (cum <= 0.05GB)
      flat  flat%   sum%        cum   cum%
    4.78GB 49.20% 49.20%     5.87GB 60.31%  github.com/nilsmagnus/grib/griblib.(*Data2).extractData
    2.36GB 24.26% 73.46%     2.36GB 24.26%  github.com/nilsmagnus/grib/griblib.(*Data2).scaleValues
    0.95GB  9.78% 83.24%     0.95GB  9.78%  github.com/nilsmagnus/grib/griblib.(*BitReader).readIntsBlock
    0.65GB  6.71% 89.95%     9.71GB 99.84%  github.com/nilsmagnus/grib/griblib.ReadMessage
    0.26GB  2.63% 92.58%     0.26GB  2.63%  github.com/nilsmagnus/grib/griblib.(*BitReader).readUintsBlock
    0.25GB  2.61% 95.19%     0.50GB  5.17%  github.com/nilsmagnus/grib/griblib.(*Data2).extractBitGroupParameters
    0.13GB  1.34% 96.52%     0.13GB  1.34%  github.com/nilsmagnus/grib/griblib.bitGroupParameter.zeroGroup
    0.10GB     1% 97.52%     0.10GB     1%  reflect.(*structType).Field
    0.09GB  0.87% 98.40%     0.18GB  1.86%  encoding/binary.Read
    0.04GB  0.46% 98.86%     0.18GB  1.82%  github.com/nilsmagnus/grib/griblib.ReadSection0
    0.04GB  0.46% 99.32%     8.88GB 91.30%  github.com/nilsmagnus/grib/griblib.readMessage
         0     0% 99.32%     0.10GB     1%  encoding/binary.dataSize
         0     0% 99.32%     0.10GB     1%  encoding/binary.sizeof
         0     0% 99.32%     0.09GB  0.93%  github.com/nilsmagnus/grib/griblib.(*Data2).extractGroupBitWidths
         0     0% 99.32%     0.08GB  0.78%  github.com/nilsmagnus/grib/griblib.(*Data2).extractGroupLengths
         0     0% 99.32%     0.08GB  0.85%  github.com/nilsmagnus/grib/griblib.(*Data2).extractGroupReferences
         0     0% 99.32%     8.75GB 90.00%  github.com/nilsmagnus/grib/griblib.ParseData3
         0     0% 99.32%     9.71GB 99.84%  github.com/nilsmagnus/grib/griblib.ReadMessages
         0     0% 99.32%     8.78GB 90.31%  github.com/nilsmagnus/grib/griblib.ReadSection7
         0     0% 99.32%     1.08GB 11.12%  github.com/nilsmagnus/grib/griblib.bitGroupParameter.readData
         0     0% 99.32%     0.89GB  9.16%  github.com/nilsmagnus/grib/griblib/gribtest.BenchmarkReadMessages
         0     0% 99.32%     0.55GB  5.66%  github.com/nilsmagnus/grib/griblib/gribtest.Test_message_to_png
         0     0% 99.32%     3.86GB 39.72%  github.com/nilsmagnus/grib/griblib/gribtest.Test_read_integrationtest_file
         0     0% 99.32%     3.81GB 39.18%  github.com/nilsmagnus/grib/griblib/gribtest.Test_read_integrationtest_file_hour0
         0     0% 99.32%     0.55GB  5.62%  github.com/nilsmagnus/grib/griblib/gribtest.Test_temperature_layers
         0     0% 99.32%     0.10GB     1%  reflect.(*rtype).Field
         0     0% 99.32%     0.87GB  8.94%  testing.(*B).launch
         0     0% 99.32%     0.89GB  9.16%  testing.(*B).runN
         0     0% 99.32%     8.83GB 90.82%  testing.tRunner

New performance-test results:

go tool pprof -top memprofile.out
File: gribtest.test
Type: alloc_space
Time: Feb 7, 2019 at 10:23pm (CET)
Showing nodes accounting for 7.82GB, 99.67% of 7.85GB total
Dropped 50 nodes (cum <= 0.04GB)
      flat  flat%   sum%        cum   cum%
    5.88GB 74.98% 74.98%     6.47GB 82.44%  github.com/nilsmagnus/grib/griblib.(*Data2).extractData
    0.75GB  9.51% 84.48%     0.75GB  9.51%  github.com/nilsmagnus/grib/griblib.(*Data2).scaleValues
    0.47GB  6.04% 90.53%     0.47GB  6.04%  github.com/nilsmagnus/grib/griblib.(*BitReader).readIntsBlock
    0.27GB  3.38% 93.91%     0.35GB  4.41%  github.com/nilsmagnus/grib/griblib.(*Data2).extractBitGroupParameters
    0.11GB  1.42% 95.32%     0.11GB  1.42%  github.com/nilsmagnus/grib/griblib.bitGroupParameter.zeroGroup
    0.08GB  1.05% 96.37%     0.08GB  1.05%  github.com/nilsmagnus/grib/griblib.(*BitReader).readUintsBlock
    0.07GB  0.94% 97.32%     7.83GB 99.80%  github.com/nilsmagnus/grib/griblib.ReadMessage
    0.06GB  0.82% 98.13%     0.06GB  0.82%  encoding/binary.Read
    0.06GB  0.78% 98.92%     0.06GB  0.79%  github.com/nilsmagnus/grib/griblib.makeBitReader
    0.06GB  0.75% 99.67%     7.76GB 98.86%  github.com/nilsmagnus/grib/griblib.readMessage
         0     0% 99.67%     7.61GB 96.98%  github.com/nilsmagnus/grib/griblib.ParseData3
         0     0% 99.67%     7.83GB 99.80%  github.com/nilsmagnus/grib/griblib.ReadMessages
         0     0% 99.67%     7.63GB 97.26%  github.com/nilsmagnus/grib/griblib.ReadSection7
         0     0% 99.67%     0.59GB  7.46%  github.com/nilsmagnus/grib/griblib.bitGroupParameter.readData
         0     0% 99.67%     1.56GB 19.90%  github.com/nilsmagnus/grib/griblib/gribtest.BenchmarkReadMessages
         0     0% 99.67%     0.37GB  4.70%  github.com/nilsmagnus/grib/griblib/gribtest.Test_message_to_png
         0     0% 99.67%     2.75GB 34.98%  github.com/nilsmagnus/grib/griblib/gribtest.Test_read_integrationtest_file
         0     0% 99.67%     2.72GB 34.72%  github.com/nilsmagnus/grib/griblib/gribtest.Test_read_integrationtest_file_hour0
         0     0% 99.67%     0.40GB  5.04%  github.com/nilsmagnus/grib/griblib/gribtest.Test_temperature_layers
         0     0% 99.67%     1.54GB 19.66%  testing.(*B).launch
         0     0% 99.67%     1.56GB 19.90%  testing.(*B).runN
         0     0% 99.67%     6.28GB 80.07%  testing.tRunner
rm profile.out memprofile.out
rm gribtest.test
nilsmagnus commented 5 years ago

What do you think @landru29, are these fixes correct? It seems that some more optimizations can be done, but I need to figure out a clever way to do that.

landru29 commented 5 years ago

Yeah ! It seems better like this !