Start the recompiled dcgm-exporter fails to collect GPU metrics with an error

[15234660879]

Ask your question

GPU-info

Locate the code shown below, and its print is as follows

INFO[0000] Starting dcgm-exporter
INFO[0000] DCGM successfully initialized!
INFO[0000] Collecting DCP Metrics
INFO[0000] Falling back to metric file '/etc/dcgm-exporter/default-counters.csv' INFO[0000] Initializing system entities of type: GPU
INFO[0001] Not collecting NvSwitch metrics; no fields to watch for device type: 3 INFO[0001] Not collecting NvLink metrics; no fields to watch for device type: 6 INFO[0001] Not collecting CPU metrics; no fields to watch for device type: 7 INFO[0001] Not collecting CPU Core metrics; no fields to watch for device type: 8 INFO[0010] mi Entity isGPU 0 [100 101 140 150 155 156 200 201 202 203 204 206 207 230 251 252 449 526 393 394 395 1] result / C.DCGM_ST_OK -10 0 INFO[0010] =========================: Object is in an undefined state

[nvvfedorov]

@15234660879, Try to run the dcgm-exporter with the following command line options: --enable-dcgm-log --dcgm-log-level=DEBUG

[15234660879]

Thank you for your reply. I use the command --enable-dcgm-log --dcgm-log-level=DEBUG, and the ERROR log output is as follows 2024-07-31 09:29:09.173 ERROR [383:399] [[NvSwitch]] Could not load NSCQ. dlwrap_attach ret: Can not access a needed shared library (-79): If this system has NvSwitches, please ensure that the package libnvidia-nscq is installed on your system and that the service user has permissions to access it. [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/modules/nvswitch/DcgmNvSwitchManager.cpp:798] [DcgmNs::DcgmNvSwitchManager::AttachToNscq] 2024-07-31 09:29:09.173 ERROR [383:399] [[NvSwitch]] AttachToNscq() returned -25 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/modules/nvswitch/DcgmNvSwitchManager.cpp:632] [DcgmNs::DcgmNvSwitchManager::Init] 2024-07-31 09:29:09.173 ERROR [383:399] [[NvSwitch]] Could not initialize switch manager. Ret: DCGM library could not be found [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/modules/nvswitch/DcgmModuleNvSwitch.cpp:34] [DcgmNs::DcgmModuleNvSwitch::DcgmModuleNvSwitch] 2024-07-31 09:29:09.173 ERROR [383:405] [[NvSwitch]] Not attached to NvSwitches. Aborting [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/modules/nvswitch/DcgmNvSwitchManager.cpp:975] [DcgmNs::DcgmNvSwitchManager::ReadNvSwitchStatusAllSwitches] 2024-07-31 09:29:09.518 ERROR [383:406] Unexpected null fieldMeta for field 150 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.518 ERROR [383:406] Unexpected null fieldMeta for field 155 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.518 ERROR [383:406] Unexpected null fieldMeta for field 156 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.518 ERROR [383:406] Unexpected null fieldMeta for field 155 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.518 ERROR [383:406] Unexpected null fieldMeta for field 156 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.518 ERROR [383:406] Unexpected null fieldMeta for field 155 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.518 ERROR [383:406] Unexpected null fieldMeta for field 156 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.518 ERROR [383:406] Unexpected null fieldMeta for field 140 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.518 ERROR [383:406] Unexpected null fieldMeta for field 150 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.518 ERROR [383:406] Unexpected null fieldMeta for field 155 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.518 ERROR [383:406] Unexpected null fieldMeta for field 140 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.518 ERROR [383:406] Unexpected null fieldMeta for field 155 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.518 ERROR [383:406] Unexpected null fieldMeta for field 155 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.520 ERROR [383:406] Unhandled field value output type: ¾ [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5762] [DcgmCacheManager::ActuallyUpdateGpuFieldValues] 2024-07-31 09:29:09.522 ERROR [383:406] Unhandled field value output type: ¾ [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5762] [DcgmCacheManager::ActuallyUpdateGpuFieldValues] 2024-07-31 09:29:09.523 ERROR [383:406] Unhandled field value output type: ¾ [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5762] [DcgmCacheManager::ActuallyUpdateGpuFieldValues] 2024-07-31 09:29:09.525 ERROR [383:406] Unhandled field value output type: ¾ [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5762] [DcgmCacheManager::ActuallyUpdateGpuFieldValues] 2024-07-31 09:29:09.527 ERROR [383:406] Unhandled field value output type: ¾ [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5762] [DcgmCacheManager::ActuallyUpdateGpuFieldValues] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 150 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 150 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 150 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 156 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 156 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 140 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 140 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 140 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 150 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 150 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 140 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 155 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 156 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 140 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 150 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 140 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 156 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 155 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 156 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 150 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 155 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 156 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 155 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 156 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 155 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 156 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 140 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 150 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 155 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 140 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 155 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:09.527 ERROR [383:406] Unexpected null fieldMeta for field 155 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5273] [DcgmCacheManager::ActuallyUpdateAllFields] 2024-07-31 09:29:10.173 INFO [383:404] DcgmCacheManagerEventThread ended [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:12537] [DcgmCacheManagerEventThread::run] 2024-07-31 09:29:10.173 DEBUG [383:404] Thread id 4026529344 stopped [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/common/DcgmThread/DcgmThread.cpp:314] [DcgmThread::RunInternal] 2024-07-31 09:29:10.174 DEBUG [383:397] Thread 140101564757568 had m_alreadyJoined 0 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/common/DcgmThread/DcgmThread.cpp:237] [DcgmThread::Wait] 2024-07-31 09:29:10.174 INFO [383:397] Event thread was stopped normally. [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:1907] [DcgmCacheManager::Shutdown] 2024-07-31 09:29:10.174 INFO [383:406] Cache manager update thread ending [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmCacheManager.cpp:5987] [DcgmCacheManager::run] 2024-07-31 09:29:10.174 DEBUG [383:406] Thread id 4009743936 stopped [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/common/DcgmThread/DcgmThread.cpp:314] [DcgmThread::RunInternal] 2024-07-31 09:29:10.175 DEBUG [383:397] Thread 140101547972160 had m_alreadyJoined 0 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/common/DcgmThread/DcgmThread.cpp:237] [DcgmThread::Wait] 2024-07-31 09:29:10.175 DEBUG [383:397] Thread 140101547972160 had m_alreadyJoined 1 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/common/DcgmThread/DcgmThread.cpp:237] [DcgmThread::Wait] 2024-07-31 09:29:10.176 DEBUG [383:397] Entering RemoveAllTrackedRequests [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmHostEngineHandler.cpp:3988] [DcgmHostEngineHandler::RemoveAllTrackedRequests] 2024-07-31 09:29:10.179 DEBUG [383:397] embedded host engine cleaned up [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmApi.cpp:5218] [dcgmStopEmbedded] 2024-07-31 09:29:10.179 DEBUG [383:397] Before dcgmapiFreeClientHandler [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmApi.cpp:5426] [dcgmShutdown] 2024-07-31 09:29:10.179 INFO [383:397] Another thread freed the client handler for us. [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmApi.cpp:239] [dcgmapiFreeClientHandler] 2024-07-31 09:29:10.179 DEBUG [383:397] After dcgmapiFreeClientHandler [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmApi.cpp:5428] [dcgmShutdown] 2024-07-31 09:29:10.179 DEBUG [383:397] dcgmShutdown completed successfully [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmApi.cpp:5463] [dcgmShutdown] 2024-07-31 09:29:20.001 DEBUG [383:399] Entering dcgmEntityGetLatestValues(dcgmHandle_t pDcgmHandle, dcgm_field_entity_group_t entityGroup, int entityId, unsigned short fieldIds[], unsigned int count, dcgmFieldValue_v1 values[]) (2147483647 1 0 0xc001fa5898 4 0xc002432000) [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/entry_point.h:341] [dcgmEntityGetLatestValues] 2024-07-31 09:29:20.001 ERROR [383:399] DcgmHostEngineHandler::Instance() returned NULL [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmApi.cpp:306] [processModuleCommandAtEmbeddedHostEngine] 2024-07-31 09:29:20.001 ERROR [383:399] dcgmModuleSendBlockingFixedRequest returned -10 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/src/DcgmApi.cpp:694] [helperGetLatestValuesForFields] 2024-07-31 09:29:20.001 DEBUG [383:399] Returning -10 [/workspaces/dcgm-rel_dcgm_3_3-postmerge@2/dcgmlib/entry_point.h:341] [dcgmEntityGetLatestValues] result / C.DCGM_ST_OK -10 0 INFO[0010] =========================: Object is in an undefined state

[glowkey]

This is a very strange error. You say that you've recompiled dcgm-exporter? Does the error happen with the official versions? It's very hard to determine what might be going wrong here.