search

bytedance / sonic

A blazingly fast JSON serializing & deserializing library
Apache License 2.0
6.71k stars 333 forks source link

Build memory usage spike when using `sonic` #543

Closed ryan961 closed 10 months ago

ryan961 commented 10 months ago

Go version: 1.21.3 Sonic version: v1.10.2

Go 1.21.3 build memory usage spike when using sonic, htop shows github.com/chenzhuoyu/iasm takes most memory. Tried lowering go version but still the same problem, finally removing sonic resolves it.

img_v2_e0a317e8-8332-4dc2-b26a-37b2f385c1ag

img_v2_ae2b67a6-0d4a-4a63-b7f0-896fb27f25dg

root     2405205 59.6 50.4 3235660 2031588 pts/1 Dl+  15:37   0:26 /usr/local/go/pkg/tool/linux_amd64/compile -o /tmp/go-build1839298967/b414/_pkg_.a -trimpath /tmp/go-build1839298967/b414=> -p [github.com/chenzhuoyu/iasm/x86_64](http://github.com/chenzhuoyu/iasm/x86_64) -lang=go1.16 -buildid cKq3gwXe-y-_95LWWpke/cKq3gwXe-y-_95LWWpke -goversion go1.21.3 -symabis /tmp/go-build1839298967/b414/symabis -c=2 -nolocalimports -importcfg /tmp/go-build1839298967/b414/importcfg -pack -asmhdr /tmp/go-build1839298967/b414/go_asm.h /data/share/projects/arpg/vendor/[github.com/chenzhuoyu/iasm/x86_64/arch.go](http://github.com/chenzhuoyu/iasm/x86_64/arch.go) /data/share/projects/arpg/vendor/[github.com/chenzhuoyu/iasm/x86_64/assembler.go](http://github.com/chenzhuoyu/iasm/x86_64/assembler.go) /data/share/projects/arpg/vendor/[github.com/chenzhuoyu/iasm/x86_64/assembler_alias.go](http://github.com/chenzhuoyu/iasm/x86_64/assembler_alias.go) /data/share/projects/arpg/vendor/[github.com/chenzhuoyu/iasm/x86_64/eface.go](http://github.com/chenzhuoyu/iasm/x86_64/eface.go) /data/share/projects/arpg/vendor/[github.com/chenzhuoyu/iasm/x86_64/encodings.go](http://github.com/chenzhuoyu/iasm/x86_64/encodings.go) /data/share/projects/arpg/vendor/[github.com/chenzhuoyu/iasm/x86_64/instructions.go](http://github.com/chenzhuoyu/iasm/x86_64/instructions.go) /data/share/projects/arpg/vendor/[github.com/chenzhuoyu/iasm/x86_64/instructions_table.go](http://github.com/chenzhuoyu/iasm/x86_64/instructions_table.go) /data/share/projects/arpg/vendor/[github.com/chenzhuoyu/iasm/x86_64/operands.go](http://github.com/chenzhuoyu/iasm/x86_64/operands.go) /data/share/projects/arpg/vendor/[github.com/chenzhuoyu/iasm/x86_64/pools.go](http://github.com/chenzhuoyu/iasm/x86_64/pools.go) /data/share/projects/arpg/vendor/[github.com/chenzhuoyu/iasm/x86_64/program.go](http://github.com/chenzhuoyu/iasm/x86_64/program.go) /data/share/projects/arpg/vendor/[github.com/chenzhuoyu/iasm/x86_64/registers.go](http://github.com/chenzhuoyu/iasm/x86_64/registers.go) /data/share/projects/arpg/vendor/[github.com/chenzhuoyu/iasm/x86_64/utils.go](http://github.com/chenzhuoyu/iasm/x86_64/utils.go)
liuq19 commented 10 months ago

maybe you could try the Pretouch to reduce the cpu/memory usage spike. More details can be found in Readme https://github.com/bytedance/sonic#pretouch

AsterDY commented 10 months ago

You can lower sonic version <1.10 but it's not recommended... Why does building memory usage matter to you?

ryan961 commented 10 months ago

Thanks for the reply. I checked the Pretouch mentioned above, but in my project I only use sonic.Unmarshal in one or two API interface places, which should not be the huge schema described in the doc.

Our current CI/CD pipeline is probably outdated, we directly go build on the demo machine. But the memory change caused by whether using sonic is too big, which makes the build basically get signal: killed.