Open krono opened 4 months ago
superg
commented
4 months ago Hi @krono, Thank you for the report. Is the issue easily reproducible? Would it be possible to request nv-hostengine core dump?
EDIT: follow up questions Do you get any syslog kernel error messages for NVLink in 16:21 - 16:29 timeframe?
krono
commented
4 months ago Hi @superg (somehow I don't get gh mails anymore, sorry)
Reproducible? for me everytime. Coredump:
core.nv-hostengine.0.35a3b73c95c04716880c91638ed46a93.3124092.1709284067000000000000.lz4.zip
superg
commented
4 months ago Thank you for the dumps, I am currently looking into it. Will share my findings here.
superg
commented
4 months ago I narrowed the search down to one of the NSCQ observe callbacks in: https://github.com/NVIDIA/DCGM/blob/master/modules/nvswitch/DcgmNvSwitchManager.cpp#L851C45-L851C53
To understand more I would like to request debug level logs when the crash happen, here's how to do it:
Make sure nvidia-dcgm service is running (nv-hostengine), execute dcgmi set --logging-severity DEBUG, that will set nv-hostengine logging level to DEBUG.
Next, reproduce the crash and share /var/nv-hostengine.log (feel free to clear it beforehand if needed).
krono
commented
4 months ago Hi, here's the log nv-hostengine.log
krono
commented
4 months ago I do not see the log point log_debug("Attaching to NvSwitches"); being hit…
superg
commented
4 months ago Thank you for the logs. indeed it crashed in another NSCQ observe callback: https://github.com/NVIDIA/DCGM/blob/master/modules/nvswitch/FieldDefinitions.cpp#L164
I am currently looking into the chain of events that led into this, will reply once I have more information.
superg
commented
3 months ago Unfortunately we aren't able to reproduce this issue internally. However we've added better debugging to help diagnose such issues in the future and at some point it will be merged to GitHub.
superg
commented
3 months ago Unfortunately we aren't able to reproduce this issue internally. However we've added better debugging to help diagnose such issues in the future and at some point it will be merged to GitHub.
krono
commented
3 months ago Is there any way i can debug that? Like a step through debugger?
superg
commented
3 months ago Yes, basically you will have to build debug DCGM with symbols. Then you will be able to use GDB, step through code and inspect variables etc. Put a breakpoint here: https://github.com/NVIDIA/DCGM/blob/master/modules/nvswitch/FieldDefinitions.cpp#L164 Just want to mention that this is pretty advanced and involves using ./build.sh script and docker dcgmbuild container (our build is containerized) and running nv-hostengine locally.
krono
commented
3 months ago So here we are.
Debuggin around this:
auto cb = [](const indexTypes... indicies,
nscq_rc_t rc,
TempData<nscqFieldType, storageType, is_vector, indexTypes...>::cbType in,
NscqDataCollector<TempData<nscqFieldType, storageType, is_vector, indexTypes...>> *dest) {
if (dest == nullptr)
{
log_error("NSCQ passed dest = nullptr");
return;
}
dest->callCounter++;
if (NSCQ_ERROR(rc))
{
log_error("NSCQ {} passed error {}", dest->nscqPath, (int)rc);
TempData<nscqFieldType, storageType, is_vector, indexTypes...> item;
item.CollectFunc(dest, indicies...);
return;
}
TempData<nscqFieldType, storageType, is_vector, indexTypes...> item; /* BREAKPOINT HERE */
item.CollectFunc(dest, in, indicies...);
};shows:
This is more or less expected.
"/{nvswitch}/id/phys_id"fieldId to be 863, but it is proabbly garbage: "32767"The callCounter looks goofy, too.
Most important, the nscqPath is not the expected "/{nvswitch}/id/phys_id" but rather the value?
-=-=-=-
It seem that there's something wrong in my /usr/lib/x86_64-linux-gnu/libnvidia-nscq.so.2, because it looks like the library is just calling this with
broken info.
Lib info:
So the promblem is probably not DCGM but rahter lib NSCQ?
superg
commented
3 months ago Hi, thank you for the details! Let me process this information and get back to you.
superg
commented
3 months ago @krono , I apologize for the long wait. We've managed to reproduce the issue on our side. While our call stack is different, the source of the problem is very likely to be the same and your observations on std::vector<> with garbage supports it. I believe that the fix that we're working on will resolve it.
krono
commented
3 months ago thanks :)
krono
commented
2 months ago Hi @superg , any news or any place I can read up on the issue here?
superg
commented
2 months ago Hi @krono, We have an internal tracking ticket for this issue and an assigned developer, this is still work in progress.
The issue is with the callback signature (after all the template instantiations), we use:
void callback(const nscq_uuid_t* device, nscq_rc_t rc, std::vector<nscq_error_t>, void *data)
whereas NSCQ expects:
void callback(const nscq_uuid_t* device, nscq_rc_t rc, const nscq_error_t error, void* data)
for a given path type.
Callback code has to be rewritten for the second signature.
krono
commented
2 months ago oh my.
Which component will need updat? DCGM or NSCQ?
superg
commented
2 months ago That's in DCGM.
krono
commented
2 months ago Thanks! I'll keep watching this space
krono
commented
1 month ago I have now a second machine that fell victim to that problem: HGX-Bases system, similarly configured.
krono
commented
2 weeks ago Hey, any news?
superg
commented
2 weeks ago @krono, the issue is identified and we are working on a fix. The current ETA is August.
Running a
3.3.5-3.4.0exporter on a 3.3.5 host-engine as shipped via nvidia-ubuntu-repos SEGFAULTs the Host-engine.Is there something I can do? Shour that be reported to the exporter instead?
Logs:
dmesg crash info
``` Feb28 16:22] nvidia-nvswitch5: open (major=510) [ +0,042810] nvidia-nvswitch4: open (major=510) [ +0,042606] nvidia-nvswitch0: open (major=510) [ +0,042409] nvidia-nvswitch2: open (major=510) [ +0,042448] nvidia-nvswitch1: open (major=510) [ +0,042372] nvidia-nvswitch3: open (major=510) [Feb28 16:29] nv-hostengine[1280071]: segfault at 28 ip 00007f09f65c74b2 sp 00007f09f61e2ba0 error 6 in libdcgmmodulenvswitch.so.3.3.5[7f09f658c000+f8000] [ +0,000008] Code: 7d b8 44 88 6d b0 e8 7d 0a ff ff 48 8b 45 a8 48 8b 73 18 48 89 45 c0 48 3b 73 20 0f 84 df 00 00 00 66 0f 6f 45 b0 48 83 c6 18 <0f> 11 46 e8 48 8b 45 c0 48 89 46 f8 48 89 73 18 48 8d 65 d8 5b 41 [ +0,155916] nvidia-nvswitch3: release (major=510) [ +0,000005] nvidia-nvswitch1: release (major=510) [ +0,000002] nvidia-nvswitch2: release (major=510) [ +0,000003] nvidia-nvswitch0: release (major=510) [ +0,000002] nvidia-nvswitch4: release (major=510) [ +0,000002] nvidia-nvswitch5: release (major=510) ```journal for exporter and hostengine
``` Feb 28 16:21:57 gx01 systemd[1]: Started NVIDIA DCGM service. Feb 28 16:21:58 gx01 nv-hostengine[1280055]: DCGM initialized Feb 28 16:21:58 gx01 nv-hostengine[1280055]: Started host engine version 3.3.5 using port number: 5555 Feb 28 16:29:08 gx01 systemd[1]: Started DCGM Exporter. Feb 28 16:29:08 gx01 dcgm-exporter[1298060]: time="2024-02-28T16:29:08+01:00" level=info msg="Starting dcgm-exporter" Feb 28 16:29:08 gx01 dcgm-exporter[1298060]: time="2024-02-28T16:29:08+01:00" level=info msg="Attemping to connect to remote hostengine at localhost:5555" Feb 28 16:29:08 gx01 dcgm-exporter[1298060]: time="2024-02-28T16:29:08+01:00" level=info msg="DCGM successfully initialized!" Feb 28 16:29:09 gx01 dcgm-exporter[1298060]: time="2024-02-28T16:29:09+01:00" level=info msg="Collecting DCP Metrics" Feb 28 16:29:09 gx01 dcgm-exporter[1298060]: time="2024-02-28T16:29:09+01:00" level=info msg="Falling back to metric file '/net/mgmtdelab/pool/html/dcgm/current/counters.csv'" Feb 28 16:29:09 gx01 dcgm-exporter[1298060]: time="2024-02-28T16:29:09+01:00" level=info msg="Initializing system entities of type: GPU" Feb 28 16:29:11 gx01 dcgm-exporter[1298060]: time="2024-02-28T16:29:11+01:00" level=info msg="Initializing system entities of type: NvSwitch" Feb 28 16:29:11 gx01 dcgm-exporter[1298060]: time="2024-02-28T16:29:11+01:00" level=info msg="Initializing system entities of type: NvLink" Feb 28 16:29:11 gx01 dcgm-exporter[1298060]: time="2024-02-28T16:29:11+01:00" level=info msg="Initializing system entities of type: CPU" Feb 28 16:29:11 gx01 dcgm-exporter[1298060]: time="2024-02-28T16:29:11+01:00" level=info msg="Not collecting CPU metrics; Error retrieving DCGM MIG hierarchy: This request is serviced by a module of DCGM that is not currently loaded" Feb 28 16:29:11 gx01 dcgm-exporter[1298060]: time="2024-02-28T16:29:11+01:00" level=info msg="Initializing system entities of type: CPU Core" Feb 28 16:29:11 gx01 dcgm-exporter[1298060]: time="2024-02-28T16:29:11+01:00" level=info msg="Not collecting CPU Core metrics; Error retrieving DCGM MIG hierarchy: This request is serviced by a module of DCGM that is not currently loaded" Feb 28 16:29:14 gx01 dcgm-exporter[1298060]: time="2024-02-28T16:29:14+01:00" level=warning msg="can not destroy group" error="Error destroying group: Host engine connection invalid/disconnected" groupID="{21}" Feb 28 16:29:14 gx01 dcgm-exporter[1298060]: time="2024-02-28T16:29:14+01:00" level=warning msg="Cannot destroy field group." error="Host engine connection invalid/disconnected" Feb 28 16:29:14 gx01 dcgm-exporter[1298060]: time="2024-02-28T16:29:14+01:00" level=fatal msg="Failed to watch metrics: Error watching fields: Host engine connection invalid/disconnected" Feb 28 16:29:14 gx01 systemd[1]: dcgm-exporter.service: Main process exited, code=exited, status=1/FAILURE Feb 28 16:29:14 gx01 systemd[1]: dcgm-exporter.service: Failed with result 'exit-code'. Feb 28 16:29:14 gx01 systemd[1]: nvidia-dcgm.service: Main process exited, code=killed, status=11/SEGV Feb 28 16:29:14 gx01 systemd[1]: nvidia-dcgm.service: Failed with result 'signal'. ```Versions
``` # dcgm-exporter -v --debug DCGM Exporter version 3.3.5-3.4.0 # dcgmi -v Version : 3.3.5 Build ID : 14 Build Date : 2024-02-24 Build Type : Release Commit ID : 93088b0e1286c6e7723af1930251298870e26c19 Branch Name : rel_dcgm_3_3 CPU Arch : x86_64 Build Platform : Linux 4.15.0-180-generic #189-Ubuntu SMP Wed May 18 14:13:57 UTC 2022 x86_64 CRC : 08a0d9624b562a1342bf5f8828939294 ```apt-cache policy datacenter-gpu-manager
``` # apt-cache policy datacenter-gpu-manager datacenter-gpu-manager: Installed: 1:3.3.5 Candidate: 1:3.3.5 Version table: *** 1:3.3.5 580 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 100 /var/lib/dpkg/status 1:3.3.3 580 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 1:3.3.1 580 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 1:3.3.0 580 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 1:3.2.6 580 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 1:3.2.5 580 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 1:3.2.3 580 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 1:3.1.8 580 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 1:3.1.7 580 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 1:3.1.6 580 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 1:3.1.3 580 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 1:3.0.4 580 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 1:2.4.8 600 600 https://repo.download.nvidia.com/baseos/ubuntu/focal/x86_64 focal-updates/common amd64 Packages 1:2.4.7 600 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 600 https://repo.download.nvidia.com/baseos/ubuntu/focal/x86_64 focal-updates/common amd64 Packages 1:2.4.6 580 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 1:2.4.5 600 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 600 https://repo.download.nvidia.com/baseos/ubuntu/focal/x86_64 focal-updates/common amd64 Packages 1:2.3.6 600 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 600 https://repo.download.nvidia.com/baseos/ubuntu/focal/x86_64 focal-updates/common amd64 Packages 1:2.3.5 580 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 1:2.3.4 580 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 1:2.3.2 600 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 600 https://repo.download.nvidia.com/baseos/ubuntu/focal/x86_64 focal-updates/common amd64 Packages 1:2.3.1 600 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 600 https://repo.download.nvidia.com/baseos/ubuntu/focal/x86_64 focal-updates/common amd64 Packages 1:2.2.9 600 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 600 https://repo.download.nvidia.com/baseos/ubuntu/focal/x86_64 focal-updates/common amd64 Packages 1:2.2.8 580 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 1:2.2.3 580 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 1:2.1.8 580 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 1:2.1.7 580 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 1:2.1.4 580 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 1:2.0.15 580 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 1:2.0.14 600 600 https://repo.download.nvidia.com/baseos/ubuntu/focal/x86_64 focal-updates/common amd64 Packages 1:2.0.13 600 580 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages 600 https://repo.download.nvidia.com/baseos/ubuntu/focal/x86_64 focal/common amd64 Packages ```OS info
``` # cat /etc/dgx-release DGX_NAME="DGX Server" DGX_PRETTY_NAME="NVIDIA DGX Server" DGX_SWBUILD_DATE="2020-10-26-11-53-11" DGX_SWBUILD_VERSION="5.0.0" DGX_COMMIT_ID="7501dff" DGX_PLATFORM="DGX Server for DGX A100" DGX_SERIAL_NUMBER="XXXXXXXXXXXX" DGX_OTA_VERSION="5.0.5" DGX_OTA_DATE="XXXXXXXXXXXXXXXXX" DGX_OTA_VERSION="5.1.1" DGX_OTA_DATE="XXXXXXXXXXXXXXXXX" DGX_OTA_VERSION="5.2.0" DGX_OTA_DATE="XXXXXXXXXXXXXXXXX" DGX_OTA_VERSION="5.3.1" DGX_OTA_DATE="XXXXXXXXXXXXXXXXX" DGX_OTA_VERSION="5.5.1" DGX_OTA_DATE="XXXXXXXXXXXXXXXXX" ```