Open biesbjerg opened 6 years ago
Interesting, I saw the same symptoms but thought the bug is in minotaur. It's possible a gpustatd bug is behind it.
Seem to be an edge case where the output of:
/usr/bin/nvidia-settings -q fans
doesn't have an entry for every device on the system.
no wait, that's wrong. it looks like a device being returned by -q fans is not present in the output from nvidia-smi -L.
Did the device fall off the bus? That is a bastard of a problem to deal with gracefully, and it does happen with mining rigs where everything is a bodge (and in all mining rigs, almost everything is a bodge, starting with PCIe risers).
Does
dmesg | grep Xid
show anything interesting?
The sanest bodge for a workaround I have come up with is using swatch to watch syslog and issue
echo b > /proc/sysrq-trigger
when an error occurs. It's pretty awful (or DevOps as fook, depending on your point of view).
Or is this a mismatch between nvidia-smi and xorg.conf?
dmesg
doesn't show anything:
[midas@midas ~]$ sudo dmesg | grep Xid
[midas@midas ~]$
My best guess then is a xorg.conf mismatch with nvidia-smi.
Looks okay to me:
[midas@midas ~]$ cat /etc/X11/xorg.conf
Section "ServerLayout"
Identifier "Layout0"
Screen 0 "Screen0" 0 0
Screen 1 "Screen1" RightOf "Screen0"
Screen 2 "Screen2" RightOf "Screen1"
Screen 3 "Screen3" RightOf "Screen2"
Screen 4 "Screen4" RightOf "Screen3"
Screen 5 "Screen5" RightOf "Screen4"
Option "Xinerama" "0"
EndSection
Section "Device"
Identifier "Device0"
Driver "nvidia"
VendorName "NVIDIA Corporation"
BoardName "GeForce GTX 1080 Ti"
Option "UseEDID" "false"
Option "AllowEmptyInitialConfiguration" "yes"
Option "ConnectToAcpid" "off"
Option "NoLogo" "1"
Option "Coolbits" "28"
Option "RegistryDwords" "PowerMizerEnable=0x1; PerfLevelSrc=0x2222; PowerMizerDefaultAC=0x1"
BusID "PCI:1:0:0"
EndSection
Section "Device"
Identifier "Device1"
Driver "nvidia"
VendorName "NVIDIA Corporation"
BoardName "GeForce GTX 1080 Ti"
Option "UseEDID" "false"
Option "AllowEmptyInitialConfiguration" "yes"
Option "ConnectToAcpid" "off"
Option "NoLogo" "1"
Option "Coolbits" "28"
Option "RegistryDwords" "PowerMizerEnable=0x1; PerfLevelSrc=0x2222; PowerMizerDefaultAC=0x1"
BusID "PCI:3:0:0"
EndSection
Section "Device"
Identifier "Device2"
Driver "nvidia"
VendorName "NVIDIA Corporation"
BoardName "GeForce GTX 1080 Ti"
Option "UseEDID" "false"
Option "AllowEmptyInitialConfiguration" "yes"
Option "ConnectToAcpid" "off"
Option "NoLogo" "1"
Option "Coolbits" "28"
Option "RegistryDwords" "PowerMizerEnable=0x1; PerfLevelSrc=0x2222; PowerMizerDefaultAC=0x1"
BusID "PCI:5:0:0"
EndSection
Section "Device"
Identifier "Device3"
Driver "nvidia"
VendorName "NVIDIA Corporation"
BoardName "GeForce GTX 1080 Ti"
Option "UseEDID" "false"
Option "AllowEmptyInitialConfiguration" "yes"
Option "ConnectToAcpid" "off"
Option "NoLogo" "1"
Option "Coolbits" "28"
Option "RegistryDwords" "PowerMizerEnable=0x1; PerfLevelSrc=0x2222; PowerMizerDefaultAC=0x1"
BusID "PCI:6:0:0"
EndSection
Section "Device"
Identifier "Device4"
Driver "nvidia"
VendorName "NVIDIA Corporation"
BoardName "GeForce GTX 1080 Ti"
Option "UseEDID" "false"
Option "AllowEmptyInitialConfiguration" "yes"
Option "ConnectToAcpid" "off"
Option "NoLogo" "1"
Option "Coolbits" "28"
Option "RegistryDwords" "PowerMizerEnable=0x1; PerfLevelSrc=0x2222; PowerMizerDefaultAC=0x1"
BusID "PCI:7:0:0"
EndSection
Section "Device"
Identifier "Device5"
Driver "nvidia"
VendorName "NVIDIA Corporation"
BoardName "GeForce GTX 1080 Ti"
Option "UseEDID" "false"
Option "AllowEmptyInitialConfiguration" "yes"
Option "ConnectToAcpid" "off"
Option "NoLogo" "1"
Option "Coolbits" "28"
Option "RegistryDwords" "PowerMizerEnable=0x1; PerfLevelSrc=0x2222; PowerMizerDefaultAC=0x1"
BusID "PCI:8:0:0"
EndSection
Section "Monitor"
Identifier "Monitor0"
VendorName "Unknown"
ModelName "Unknown"
HorizSync 28.0 - 33.0
VertRefresh 43.0 - 72.0
Option "DPMS"
EndSection
Section "Monitor"
Identifier "Monitor1"
VendorName "Unknown"
ModelName "Unknown"
HorizSync 28.0 - 33.0
VertRefresh 43.0 - 72.0
Option "DPMS"
EndSection
Section "Monitor"
Identifier "Monitor2"
VendorName "Unknown"
ModelName "Unknown"
HorizSync 28.0 - 33.0
VertRefresh 43.0 - 72.0
Option "DPMS"
EndSection
Section "Monitor"
Identifier "Monitor3"
VendorName "Unknown"
ModelName "Unknown"
HorizSync 28.0 - 33.0
VertRefresh 43.0 - 72.0
Option "DPMS"
EndSection
Section "Monitor"
Identifier "Monitor4"
VendorName "Unknown"
ModelName "Unknown"
HorizSync 28.0 - 33.0
VertRefresh 43.0 - 72.0
Option "DPMS"
EndSection
Section "Monitor"
Identifier "Monitor5"
VendorName "Unknown"
ModelName "Unknown"
HorizSync 28.0 - 33.0
VertRefresh 43.0 - 72.0
Option "DPMS"
EndSection
Section "Screen"
Identifier "Screen0"
Device "Device0"
Monitor "Monitor0"
DefaultDepth 24
SubSection "Display"
Depth 24
EndSubSection
EndSection
Section "Screen"
Identifier "Screen1"
Device "Device1"
Monitor "Monitor1"
DefaultDepth 24
SubSection "Display"
Depth 24
EndSubSection
EndSection
Section "Screen"
Identifier "Screen2"
Device "Device2"
Monitor "Monitor2"
DefaultDepth 24
SubSection "Display"
Depth 24
EndSubSection
EndSection
Section "Screen"
Identifier "Screen3"
Device "Device3"
Monitor "Monitor3"
DefaultDepth 24
SubSection "Display"
Depth 24
EndSubSection
EndSection
Section "Screen"
Identifier "Screen4"
Device "Device4"
Monitor "Monitor4"
DefaultDepth 24
SubSection "Display"
Depth 24
EndSubSection
EndSection
Section "Screen"
Identifier "Screen5"
Device "Device5"
Monitor "Monitor5"
DefaultDepth 24
SubSection "Display"
Depth 24
EndSubSection
EndSection
[midas@midas ~]$ nvidia-smi
Fri Mar 2 11:20:22 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.30 Driver Version: 390.30 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 108... On | 00000000:01:00.0 Off | N/A |
| 28% 50C P2 136W / 139W | 2812MiB / 11176MiB | 100% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX 108... On | 00000000:03:00.0 Off | N/A |
| 31% 52C P2 138W / 139W | 2656MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 GeForce GTX 108... On | 00000000:05:00.0 Off | N/A |
| 28% 51C P2 138W / 139W | 2656MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 GeForce GTX 108... On | 00000000:06:00.0 Off | N/A |
| 28% 51C P2 137W / 139W | 2656MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 4 GeForce GTX 108... On | 00000000:07:00.0 Off | N/A |
| 28% 51C P2 138W / 139W | 2656MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 5 GeForce GTX 108... On | 00000000:08:00.0 Off | N/A |
| 31% 51C P2 139W / 139W | 2656MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
Yeah, that does look right...
I have just had wrong power limit again, but no similar gpustatd crash in the same time period. So the gpustatd crash and power limits being set wrong are not directly related.
And no PCIe errors on the device that was mis-set. This looks like at least two separate bugs. The power setting one is quite critical. Until that is fixed, it may be safer and possibly even more profitable to just run ethminer standalone with a static power limit.
minotaur.log
2018-03-02 12:58:59: [info] initialising
2018-03-02 12:59:00: [info] new version available: v1.1.2
2018-03-02 12:59:00: [info] scanning devices
2018-03-02 12:59:00: [warning] gpustatd metrics file is stale (>5s): /var/run/gpustatd/5.yml
2018-03-02 12:59:00: [warning] gpustatd metrics file is stale (>5s): /var/run/gpustatd/4.yml
2018-03-02 12:59:00: [warning] gpustatd metrics file is stale (>5s): /var/run/gpustatd/3.yml
2018-03-02 12:59:00: [warning] gpustatd metrics file is stale (>5s): /var/run/gpustatd/2.yml
2018-03-02 12:59:00: [warning] gpustatd metrics file is stale (>5s): /var/run/gpustatd/1.yml
2018-03-02 12:59:00: [warning] gpustatd metrics file is stale (>5s): /var/run/gpustatd/0.yml
2018-03-02 12:59:00: [info] found 6 nvidia GPUs
2018-03-02 12:59:00: [info] device 0 [1080Ti]: ccminer: blake2s,blake256r8,decred,cryptonight,x13,keccak,x11gost,neoscrypt
2018-03-02 12:59:00: [info] device 0 [1080Ti]: ccminer: quark,nist5,qubit,skunk,lbry,lyra2rev2,x15
2018-03-02 12:59:00: [info] device 0 [1080Ti]: ethminer: daggerhashimoto
2018-03-02 12:59:00: [info] device 0 [1080Ti]: excavator: blake2s,daggerhashimoto_sia,cryptonight,daggerhashimoto,keccak
2018-03-02 12:59:00: [info] device 0 [1080Ti]: excavator: sia,neoscrypt,daggerhashimoto_decred,daggerhashimoto_pascal,pascal
2018-03-02 12:59:00: [info] device 0 [1080Ti]: excavator: decred,nist5,equihash,lbry,lyra2rev2
2018-03-02 12:59:00: [info] device 0 [1080Ti]: ccminer2: blake2s,keccak,quark,x11gost,qubit,lbry,nist5
2018-03-02 12:59:00: [info] device 1 [1080Ti]: as above
2018-03-02 12:59:00: [info] device 2 [1080Ti]: as above
2018-03-02 12:59:00: [info] device 3 [1080Ti]: as above
2018-03-02 12:59:00: [info] device 4 [1080Ti]: as above
2018-03-02 12:59:00: [info] device 5 [1080Ti]: as above
2018-03-02 12:59:00: [info] you have calibration data for all supported algorithms :)
2018-03-02 12:59:00: [info] retrieving state from miner backends
2018-03-02 12:59:00: [info] device 0 [1080Ti]: most profitable is now: ethermine/daggerhashimoto in region: None using ethminer
2018-03-02 12:59:00: [info] device 1 [1080Ti]: most profitable is now: ethermine/daggerhashimoto in region: None using ethminer
2018-03-02 12:59:00: [info] device 2 [1080Ti]: most profitable is now: ethermine/daggerhashimoto in region: None using ethminer
2018-03-02 12:59:00: [info] device 3 [1080Ti]: most profitable is now: ethermine/daggerhashimoto in region: None using ethminer
2018-03-02 12:59:00: [info] device 4 [1080Ti]: most profitable is now: ethermine/daggerhashimoto in region: None using ethminer
2018-03-02 12:59:00: [info] device 5 [1080Ti]: most profitable is now: ethermine/daggerhashimoto in region: None using ethminer
2018-03-02 12:59:01: [info] device 0 [1080Ti]: starting algorithm daggerhashimoto with ethminer [pool=ethermine] [profile=1080ti_oc_daggerhas
himoto] [region=None]
2018-03-02 12:59:01: [info] device 1 [1080Ti]: starting algorithm daggerhashimoto with ethminer [pool=ethermine] [profile=1080ti_oc_daggerhas
himoto] [region=None]
2018-03-02 12:59:01: [warning] gpustatd metrics file is stale (>5s): /var/run/gpustatd/5.yml
2018-03-02 12:59:01: [info] device 2 [1080Ti]: starting algorithm daggerhashimoto with ethminer [pool=ethermine] [profile=1080ti_oc_daggerhas
himoto] [region=None]
2018-03-02 12:59:01: [info] device 5 [1080Ti]: starting algorithm daggerhashimoto with ethminer [pool=ethermine] [profile=1080ti_oc_daggerhas
himoto] [region=None]
2018-03-02 12:59:01: [info] device 3 [1080Ti]: starting algorithm daggerhashimoto with ethminer [pool=ethermine] [profile=1080ti_oc_daggerhas
himoto] [region=None]
2018-03-02 12:59:01: [warning] gpustatd metrics file is stale (>5s): /var/run/gpustatd/4.yml
2018-03-02 12:59:01: [info] device 4 [1080Ti]: starting algorithm daggerhashimoto with ethminer [pool=ethermine] [profile=1080ti_oc_daggerhas
himoto] [region=None]
2018-03-02 12:59:01: [warning] gpustatd metrics file is stale (>5s): /var/run/gpustatd/3.yml
2018-03-02 12:59:01: [warning] gpustatd metrics file is stale (>5s): /var/run/gpustatd/2.yml
2018-03-02 12:59:01: [warning] gpustatd metrics file is stale (>5s): /var/run/gpustatd/1.yml
2018-03-02 12:59:01: [warning] gpustatd metrics file is stale (>5s): /var/run/gpustatd/0.yml
2018-03-02 12:59:01: [warning] failed to set power limit on device 4 (check we have +s on nvidia-smi)
2018-03-02 12:59:21: [warning] gpustatd metrics file is stale (>5s): /var/run/gpustatd/5.yml
2018-03-02 12:59:21: [warning] gpustatd metrics file is stale (>5s): /var/run/gpustatd/4.yml
2018-03-02 12:59:21: [warning] gpustatd metrics file is stale (>5s): /var/run/gpustatd/3.yml
2018-03-02 12:59:21: [warning] gpustatd metrics file is stale (>5s): /var/run/gpustatd/2.yml
2018-03-02 12:59:21: [warning] gpustatd metrics file is stale (>5s): /var/run/gpustatd/1.yml
2018-03-02 12:59:21: [warning] gpustatd metrics file is stale (>5s): /var/run/gpustatd/0.yml
Seems I have a crashing GPU
[midas@midas ~]$ sudo su gpustatd
bash-4.2$ gpustatd
info: gpustatd 1.1.4 starting up
info: scanning devices
Traceback (most recent call last):
File "/home/mock/rpmbuild/BUILD/gpustatd-1.1.4/gpustatd.py", line 210, in <module>
File "/home/mock/rpmbuild/BUILD/gpustatd-1.1.4/gpustatd.py", line 33, in __init__
File "/home/mock/rpmbuild/BUILD/gpustatd-1.1.4/singleton.py", line 5, in __call__
File "/home/mock/rpmbuild/BUILD/gpustatd-1.1.4/nvidia.py", line 63, in __init__
KeyError: 4
bash-4.2$ nvidia-smi -L
GPU 0: GeForce GTX 1080 Ti (UUID: GPU-29c69203-c1cd-73d6-7776-82b222317e43)
GPU 1: GeForce GTX 1080 Ti (UUID: GPU-f0043b8c-451a-645b-e0f7-d4164bff9443)
GPU 2: GeForce GTX 1080 Ti (UUID: GPU-408fdbc5-7d5a-67a9-97cc-e71214807d2a)
GPU 3: GeForce GTX 1080 Ti (UUID: GPU-87241d91-62ff-7633-b776-d69789b3bf12)
Unable to determine the device handle for gpu 0000:07:00.0: GPU is lost. Reboot the system to recover this GPU
GPU 5: GeForce GTX 1080 Ti (UUID: GPU-653ef5a7-d43e-c784-146d-d850bdee0521)
Time to dial back the OC. What OC offsets do you use?
The question is, does one crashed GPU somehow prevent gputstatd from updating metrics for all devices? That probably shouldn't be the case.
Yeah, just dialed OC back a bit, for equihash and daggerhashimoto (the algos that have been running the last couple of hours)
device_profiles:
default:
algorithm: all
device: all
1080ti_oc:
algorithm: all
device: 1080ti
gpu_clock_offset: 115
memory_clock_offset: 1500
1080ti_oc_equihash:
algorithm: equihash
device: 1080ti
gpu_clock_offset: 150 # 155
memory_clock_offset: 1580 # 1620
1080ti_oc_neoscrypt:
algorithm: neoscrypt
device: 1080ti
gpu_clock_offset: 105
memory_clock_offset: 450
1080ti_oc_x11gost:
algorithm: x11gost
device: 1080ti
gpu_clock_offset: 155
memory_clock_offset: 1620
1080ti_oc_lyra2rev2:
algorithm: lyra2rev2
device: 1080ti
gpu_clock_offset: 175
memory_clock_offset: 1650
1080ti_oc_daggerhashimoto:
algorithm: daggerhashimoto
device: 1080ti
gpu_clock_offset: 115 # 125
memory_clock_offset: 1130 # 1150
Nvidia API doesn't degrade gracefully, when a GPU crashes, most control goes away, and X usually goes with it.
+1500 RAM is borderline at best on 1080Ti. On Asus, stick with +823, any more and performance gets worse. On other cards, +1000 is safe in P2. Relatively few cards can handle +1500 stably. Any more and you are heading for serious instability very quickly.
+150 on GPU is also borderline. Stick with +125 with any sign of trouble.
And I'm not convinced that a profile per algorithm is more than a huge waste of time for a placebo effect.
And API will probably never get graceful degradation because the problem goes away without OC and fan control. No need for Xorg even without those.
I find it really depends on the algo. I have manually calibrated my cards, algo vs. OC and think I've found the 'sweet spot' performance wise. Stability no guaranteed of course.
Anyway, after rebooting, my GPU still doesn't show up. I think maybe the GPU or riser has gone bad.
After running for some time I often get this (pulled from
/var/log/messages
):This, of course, causes all sorts of trouble for minotaur. I had negative profits because it was mining
daggerhashimoto
at 277 W instead of the expected 139 W etc.