pytorch / pytorch

Tensors and Dynamic neural networks in Python with strong GPU acceleration
https://pytorch.org
Other
80.14k stars 21.54k forks source link

Unable to record Memory consumption with `torch.cuda.memory._record_memory_history()` #128131

Open GeJulia opened 4 weeks ago

GeJulia commented 4 weeks ago

🐛 Describe the bug

I get the following issue when I want to record my memory with torch.cuda.memory._record_memory_history()

line 696, in _record_memory_history_impl
    _C._cuda_recordMemoryHistory(enabled_, record_context, record_context_cpp,
RuntimeError: low_pc_ <= addr && addr <= high_pc_ INTERNAL ASSERT FAILED at "../torch/csrc/profiler/unwind/fde.h":205, please report a bug to PyTorch. NOT IN RANGE?

​My environment includes

torch                     2.1.0.dev20230405+cu118          pypi_0    pypi
torch-dct                 0.1.6                    pypi_0    pypi
torchaudio                2.1.0.dev20230405+cu118          pypi_0    pypi
torchmetrics              0.11.4                   pypi_0    pypi
torchvision               0.16.0.dev20230405+cu118          pypi_0    pypi

Versions

Versions of relevant libraries: [pip3] numpy==1.24.4 [pip3] pytorch-ignite==0.4.12 [pip3] pytorch-lightning==2.0.2 [pip3] pytorch-triton==2.1.0+46672772b4 [pip3] torch==2.1.0.dev20230405+cu118 [pip3] torch-dct==0.1.6 [pip3] torchaudio==2.1.0.dev20230405+cu118 [pip3] torchmetrics==0.11.4 [pip3] torchvision==0.16.0.dev20230405+cu118 [conda] numpy 1.24.1 pypi_0 pypi [conda] pytorch-ignite 0.4.12 pypi_0 pypi [conda] pytorch-lightning 2.0.2 pypi_0 pypi [conda] pytorch-triton 2.1.0+46672772b4 pypi_0 pypi [conda] torch 2.1.0.dev20230405+cu118 pypi_0 pypi [conda] torch-dct 0.1.6 pypi_0 pypi [conda] torchaudio 2.1.0.dev20230405+cu118 pypi_0 pypi [conda] torchmetrics 0.11.4 pypi_0 pypi [conda] torchvision 0.16.0.dev20230405+cu118 pypi_0 pypi

cc @ptrblck @msaroufim

janeyx99 commented 4 weeks ago

Can you try this on the latest torch?