Out of memory crash

[florensie]

2018-04-30 14:11:53,150 DEBUG: [14:11:53][0x00007f77174bf700][info] net | sia | New job '00000060efe0c20e', diff=2
2018-04-30 14:11:53,229 DEBUG: [14:11:53][0x00007f771b798140][info] net | sia | Share #42 accepted
2018-04-30 14:11:53,589 DEBUG: [14:11:53][0x00007f77174bf700][info] net | daggerhashimoto | New job '0000003ddaf88b8a', diff=0.25
2018-04-30 14:11:57,824 DEBUG: [14:11:57][0x00007f771b798140][info] net | daggerhashimoto | New job '0000003ddaf89e29', diff=0.25
2018-04-30 14:12:01,977 DEBUG: [14:12:01][0x00007f77174bf700][info] net | daggerhashimoto | New job '0000003ddaf8b75d', diff=0.25
2018-04-30 14:12:02,959 DEBUG: [14:12:02][0x00007f771b798140][info] net | daggerhashimoto | New job '0000003ddaf95502', diff=0.25
2018-04-30 14:12:10,193 DEBUG: [14:12:10][0x00007f77174bf700][info] net | daggerhashimoto | New job '0000003ddafa268c', diff=0.25
2018-04-30 14:12:13,113 DEBUG: [14:12:13][0x00007f77174bf700][info] net | sia | Share #43 accepted
2018-04-30 14:12:14,606 DEBUG: [14:12:14][0x00007f771b798140][info] net | daggerhashimoto | New job '0000003ddafa3a0f', diff=0.25
2018-04-30 14:12:15,464 DEBUG: [14:12:15][0x00007f77174bf700][info] net | daggerhashimoto | New job '0000003ddafa4214', diff=0.25
2018-04-30 14:12:15,576 DEBUG: [14:12:15][0x00007f7709dff700][fatal] wrkr0-0 | CUDA error 'out of memory' in func 'run' line 695
2018-04-30 14:12:15,577 DEBUG: [14:12:15][0x00007f771b798140][info] Shutting down
2018-04-30 14:12:15,578 DEBUG: [14:12:15][0x00007f771b798140][info] api | Closing
2018-04-30 14:12:15,683 DEBUG: 
2018-04-30 14:14:17,319 INFO: Switching nvidia_GPU-4719bfd3-10d6-e79e-3dd1-092d3774e3dc from excavator_daggerhashimoto_sia to excavator_lyra2rev2 (0.096 -> 0.106 mBTC/day)
Traceback (most recent call last):
  File "./nuxhashd.py", line 259, in <module>
    main()
  File "./nuxhashd.py", line 77, in main
    do_mining(nx_settings, nx_benchmarks, all_devices)
  File "./nuxhashd.py", line 236, in do_mining
    algorithm.set_devices(my_devices)
  File "/home/florens/nuxhash/miners/excavator.py", line 213, in set_devices
    self._attach_device(device)
  File "/home/florens/nuxhash/miners/excavator.py", line 219, in _attach_device
    raise miner.MinerNotRunning('could not connect to excavator')
miners.miner.MinerNotRunning

The script had been running flawlessly on a 6GB GTX 1060 for over 24 hours until this happened. Could be a memory leak or just an issue with CUDA? I'm running the latest CUDA on Arch linux. Maybe someting should be implemented to restart if CUDA runs out of memory instead of shutting down the excavator and crashing to program as a result.

[blacktigersoftware]

I second this bug (enhancement). If we could only respawn the commandline..... Problem occurs on a 1060 6Gb SC. Occurred last when switching from DaggarHashimoto to LyraRev2.

[YoRyan]

Fixed, thanks for reporting. I've noticed this on my rig too. Besides, in general, you shouldn't blindly trust your mining program to run flawlessly...