support stackmap/gcdatamask/gcbssmask

cloudwego / goref

Go heap object reference analysis tool

Apache License 2.0

538 stars 17 forks source link

support stackmap/gcdatamask/gcbssmask #6

Closed jayantxie closed 2 days ago

jayantxie commented 2 months ago

Currently, the gcmask scanning only supports heap gcmask and does not support stackmap, gcbssmask/gcdatamask. This may lead to some memory being missed in extreme cases. FYI: https://github.com/golang/go/blob/master/src/runtime/heapdump.go#L270 https://github.com/golang/go/blob/master/src/runtime/heapdump.go#L445

jensenojs commented 2 months ago

Is this issue urgent? (I'm a bit worried that I won't be able to finish it in time after taking it over.)

and I haven't read the source code of Golang. What relevant documents or core concepts are helpful for understanding this project?

jayantxie commented 2 months ago

It's OK, I guess you're Chinese? Maybe you could refer to this book: https://draveness.me/golang/docs/part3-runtime/ch07-memory/golang-memory-allocator/ And I've wrote a article to explain the principle, which can be found in README doc.

jensenojs commented 2 months ago

Haha, yeah. Thanks so much. I'll be definitely submit a pull request to goref!

It's OK, I guess you're Chinese? Maybe you could refer to this book: https://draveness.me/golang/docs/part3-runtime/ch07-memory/golang-memory-allocator/ And I've wrote a article to explain the principle, which can be found in README doc.

Zxilly commented 1 month ago

I think it's important, with github.com/Zxilly/go-size-analyzer goref can only recognize a small subset of the memory used. Specifically, only about 300m of the 2000m memory footprint can be recognized.

jayantxie commented 1 month ago

I think it's important, with github.com/Zxilly/go-size-analyzer goref can only recognize a small subset of the memory used. Specifically, only about 300m of the 2000m memory footprint can be recognized.

@Zxilly Unknown types of memory can be identified through gc root mask, but the type information cannot be displayed. Can you provide corresponding core files and executable files? I will analyze the cause of this case and fix it

Zxilly commented 1 month ago

I was using attach to collect the info, so no core file can be provided.

jayantxie commented 1 month ago

Is there any way for me to reproduce your scene? BTW, you can also manually obtain the core file using the gcore ${PID} command

Zxilly commented 1 month ago

I didn't know gcore before, I'll return later with the coredump.

Zxilly commented 1 month ago

The coredump file is huge, I extreme compressed this file, but it still exceeded the limit allowed by GitHub. The link is valid for 7 days.

https://www.transfernow.net/dl/20240819pxX6giXG

jayantxie commented 1 month ago

Thank you so much! I'll figure out the problem.

jayantxie commented 1 month ago

The go version of the given core file is 1.23, which is not supported by goref yet. I am already planning to support it.

jayantxie commented 2 weeks ago

@Zxilly Have you seen the heap profile? I have confirmed that the total heap span for your core file is 700MB. The 360mb collected by goref is within the normal range. You can use the command free -h to check if cache is occupying other memory.

jayantxie commented 1 week ago

@Zxilly I have supported the gcrootmask and rescanned the core file you sent. The scanned memory result is the same with the previous. And I have verified that the total memory of all spans in the heap is 676mb, so this should be the actual memory.

Without dwarf:

jayantxie commented 2 days ago

Done: https://github.com/cloudwego/goref/pull/26