Memory Leak Bugs Reported by PyRefCon(static analyzer report)

nonsense-j commented 1 year ago

Bugs Overview

PyRefCon(by @Snape3058) is a static analyzer tool to detect memory-related bugs (refcounting issues) by tracing reference count of python objects. After analyzing Paddle 2.5(latest) by PyRefCon, we found 36 bugs, all of which are categorized as memory leak issues. After mannually reviewed based on our knowledge, we divided them into these groups by their leak location:

C1: PADDLE_THROW Leak. For this case, program will directly throw an exception by PADDLE_THROW without decreasing the reference count. Report count: 25.
C2: PADDLE_ENFORCE Leak. For this case, program will indirectly throw an exception by PADDLE_ENFORCE_XX assertions without decreasing the reference count. Report count: 4.
C3: Pure Leak. For this case, program will leave the current scope without decreasing the reference count. Report count: 7.

All reports can be accessed in our reports archive. In detail, we will introduce every bug report organized by their located files for easy revision.

Fix Suggestions

To fix the above issues, we suggest using a smart pointer implementation to replace the raw pointers in code to automatically decrease the reference count after exceptions are thrown. e.g. https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/lib/core/safe_pyobject_ptr.h

nonsense-j commented 1 year ago

Submitted by mistake, this issue will be reopened later.

wanghuancoder commented 1 year ago

非常感谢您的建议！回头看，确实智能指针更好管理一些！这是早期我设计时的失误。目前这个改成只能指针工作量不小，我们将尽早排期优化！

Snape3058 commented 1 year ago

除了 pure leak 是引用計數忘記遞減外，其他都是臨時使用的對象在拋出異常的情況下會導致引用計數未被遞減的情況。可以根據工具報告了的問題，對項目中的其他各處的使用情況進行進一步檢查。感謝回覆，還希望能在對上述問題進行核對後給出確認的結果。如果可以的話，希望確認結果能使用英文進行回覆。其他討論可以繼續使用中文

TimeYWL commented 9 months ago

Is there any workaround method?

Snape3058 commented 9 months ago

Is there any workaround method?

The fix suggestion part, at the end of this issue report, can be a workaround solution.

Using RAII-based monitor objects, such as smart pointers may be helpful to automatically decrease the reference count when the function exits unexpectedly.

TimeYWL commented 9 months ago

When I run my demo, the memory keeps increasing without out any exception:

class dropout(paddle.nn.Layer):
    def __init__(self):
        super().__init__()
        self.scale = 1.0
        self.bias = 0.5
    def forward(self, x):
        class c_dropout_eager(PyLayer):
            @staticmethod
            def forward(ctx, tensor):
                return _legacy_C_ops.scale(tensor, 'scale', self.scale, 'bias', self.bias)
            @staticmethod
            def backward(ctx, dy):
                return _legacy_C_ops.scale(tensor, 'scale', self.scale, 'bias', self.bias)
        return c_dropout_eager.apply(x)

drop_demo = dropout()

for i in tqdm(range(500000)):
    np_data = np.random.rand(512, 4096)

    output = drop_demo(paddle.to_tensor(np_data, dtype='float32'))

How it happend?

Snape3058 commented 9 months ago

When I run my demo, the memory keeps increasing without out any exception:

These reports are generated with our static analyzer. Hence, we do not know how to trigger them with actual Python code.

Besides, the aforementioned bugs happen when an exception is thrown (for bugs not categorized as "pure leak"). If no exceptions are thrown during execution, the corresponding execution path may not follow the one we reported.

For details of the reported path, please click the "report-xxxxxx" link of each report.

PaddlePaddle / Paddle