(-217:Gpu API call) invalid argument in function 'operator()' Exception will be ignored.

[TonyTheTaiga]

• OpenCV => 4.5.1
• Operating System / Platform => Linux 4.9.140-tegra aarch64
• Compiler => c++ (ver 7.5.0)
• Cuda 10.2
• cuDNN 8.0.0

Detailed description

When I go to shutdown my application this error is called and I've no clue as to why. I am using the cuda backend and running a yolo_v3 model. Has anyone else seen this warning pop up?

[ WARN:0] global /opencv-build/build/opencv-4.5.1/modules/dnn/src/cuda4dnn/csl/memory.hpp (67) operator() Device memory deallocation failed in deleter. OpenCV(4.5.1) /opencv-build/build/opencv-4.5.1/modules/dnn/src/cuda4dnn/csl/memory.hpp:61: error: (-217:Gpu API call) invalid argument in function 'operator()' Exception will be ignored.

[YashasSamaga]

Do you use CUDA elsewhere in your application? Do you use multithreading? Please provide a reproducer if possible.

[CloudRobot]

I meet same problem, and I find Net should be defined in main function ,and can not be used as global variables. if i use it as global variable ,it find this:

WARN:6] global /data/image/opencv/opencv-4.5.2/modules/dnn/src/cuda4dnn/csl/memory.hpp (67) operator() Device memory deallocation failed in deleter. OpenCV(4.5.2) /data/image/opencv/opencv-4.5.2/modules/dnn/src/cuda4dnn/csl/memory.hpp:61: error: (-217:Gpu API call) driver shutting down in function 'operator()' Exception will be ignored.

terminate called after throwing an instance of 'cv::dnn::cuda4dnn::csl::CUDAException' what(): OpenCV(4.5.2) /data/image/opencv/opencv-4.5.2/modules/dnn/src/cuda4dnn/csl/memory.hpp:293: error: (-217:Gpu API call) driver shutting down in function '~MemoryLockGuard'

[YashasSamaga]

@CloudRobot Thanks for the tip. I have reproduced the problem. It happens only when the net object is a global variable. I will look into it and try to find the cause and then resolve it.

For the time being, please clear the global Net object by assigning an empty network to stop the errors/warnings.

// this forces the CUDA backend to clean-up before your main function exits
net = cv::dnn::Net();

[YashasSamaga]

Related: https://stackoverflow.com/questions/35815597/cuda-call-fails-in-destructor

I think this cannot be fixed in OpenCV code. The problem is caused by the CUDA compiler silently inserting CUDA objects (for initialization and whatnot) into the global scope. So what essentially could be happening is that the CUDA context objects are destroyed before the Net object is destroyed. In this case, the CUDA backend will invoke CUDA runtime API to clean up in a corrupt context which causes the errors that you see.

[CloudRobot]

Related: https://stackoverflow.com/questions/35815597/cuda-call-fails-in-destructor

I think this cannot be fixed in OpenCV code. The problem is caused by the CUDA compiler silently inserting CUDA objects (for initialization and whatnot) into the global scope. So what essentially could be happening is that the CUDA context objects are destroyed before the Net object is destroyed. In this case, the CUDA backend will invoke CUDA runtime API to clean up in a corrupt context which causes the errors that you see.

Thanks Very Much ! I solved it using point. in the first, cv::dnn::Net *net = new cv::dnn::Net(); in the end, delete net;

[YashasSamaga]

@CloudRobot You can do it without using pointers:

cv::dnn::Net net;
int main ()
{
    // initialize your network, do inference
    net = cv::dnn::Net(); // this forces the previous instance to destruct
    return 0;
}

RAII based solution:

class NetGuard {
public:
    NetGuard(cv::dnn::Net& net) p_net(net) { }
    NetGuard(const NetGuard&) = delete;
    ~NetGuard() { p_net = cv::dnn::Net(); }

    NetGuard& operator=(const NetGuard&) = delete;

private:
    cv::dnn::Net& p_net;
};

cv::dnn::Net net;
int main () {
    NetGuard guard(net);
    // use 'net', initialize, do inference
    // NetGuard will ensure cv::dnn::Net is destroyed before 'main' finishes
    return 0;
}

I suggest the RAII based solution as it will also handle early returns, exceptions, etc.

[CloudRobot]

@CloudRobot You can do it without using pointers:

cv::dnn::Net net;
int main ()
{
    // initialize your network, do inference
    net = cv::dnn::Net(); // this forces the previous instance to destruct
    return 0;
}

RAII based solution:

class NetGuard {
public:
    NetGuard(cv::dnn::Net& net) p_net(net) { }
    NetGuard(const NetGuard&) = delete;
    ~NetGuard() { net = cv::dnn::Net(); }

    NetGuard& operator=(const NetGuard&) = delete;

private:
    cv::dnn::Net& p_net;
};

cv::dnn::Net net;
int main () {
    NetGuard(net);
    // use 'net', initialize, do inference
    // NetGuard will ensure cv::dnn::Net is destroyed before 'main' finishes
    return 0;
}

I suggest the RAII based solution as it will also handle early returns, exceptions, etc.

Thanks very much.

[CloudRobot]

it can not run ,could you help me. i use opencv4.5.2.

the error is:

expected ‘;’ at end of member declaration NetGuard(cv::dnn::Net& net) p_net(cv::dnn::Net net) { } ‘cv::dnn::dnn4_v20210301::Net& NetGuard::pp_net’ conflicts with a previous declaration cv::dnn::Net& pp_net; error ‘net’ not declared in this scope ~NetGuard() { net = cv::dnn::Net();

@YashasSamaga Thanks a lot.

[YashasSamaga]

@CloudRobot Sorry, I had made a typo: net should be p_net in the destructor. I haven't tested the code but it should be easy to get it working.

[CloudRobot]

@CloudRobot Sorry, I had made a typo: net should be p_net in the destructor. I haven't tested the code but it should be easy to get it working.

Thank a lot. I change It and could run .

 public:
     NetGuard(cv::dnn::Net& net) : p_net(net) { } ;
     NetGuard(const NetGuard&) = delete;
     ~NetGuard() { p_net = cv::dnn::Net(); }

     NetGuard& operator=(const NetGuard&) = delete;

 private:
     cv::dnn::Net& p_net;
 };

 cv::dnn::Net net;
 int main () {
     NetGuard NetGuard(net);
     // use 'net', initialize, do inference
     // NetGuard will ensure cv::dnn::Net is destroyed before 'main' finishes
     return 0;
 }```

[mochechan]

To share my experiences:

This is my opencv with yolov4 code modified from Asadullah-Dal17.

#!/usr/bin/python3
import os,cv2,numpy

# https://github.com/Asadullah-Dal17/yolov4-opencv-python/blob/master/yolov4.py
class yolov4:
    def __init__(self,names,weights,cfg,debug=False):
        if not os.path.isfile(names):
            raise ValueError("not exist:",names)
        if not os.path.isfile(weights):
            raise ValueError("not exist:",weights)
        if not os.path.isfile(cfg):
            raise ValueError("not exist:",cfg)
        if not all([os.path.isfile(x) for x in [names,weights,cfg]]):
            raise ValueError("All names,weights,cfg files should be prepared.")

        self.names = [cname.strip() for cname in open(names,"r").readlines()]
        c = open(cfg,"r").readlines()
        self.width = int([x.strip() for x in c if x.find("width") == 0][0].split("=")[1])
        self.height = int([x.strip() for x in c if x.find("height") == 0][0].split("=")[1])
        #print("nn input size",self.width, self.height)

        net = cv2.dnn.readNet(weights, cfg) # opencv should be compiled with cuda
        net.setPreferableBackend(cv2.dnn.DNN_BACKEND_CUDA)
        net.setPreferableTarget(cv2.dnn.DNN_TARGET_CUDA_FP16)
        self.model = cv2.dnn_DetectionModel(net)
        self.model.setInputParams(size=(self.width, self.height), scale=1/255, swapRB=True)

    def infer(self,img,conf_thresh=0.25,nms_thresh=0.4)->list:
        classes, scores, bboxes = self.model.detect(img, conf_thresh, nms_thresh)
        objects = []
        for (classid, score, bbox) in zip(classes, scores, bboxes):
            #print(isinstance(classid,int),type(classid))
            if isinstance(classid,numpy.int32):
                objects.append({"id":classid, "name":self.names[classid], "probability": score, "x":bbox[0], "y":bbox[1], "w":bbox[2], "h":bbox[3]})
            elif isinstance(classid,list):
                objects.append({"id":classid[0], "name":self.names[classid[0]], "probability": score[0], "x":bbox[0], "y":bbox[1], "w":bbox[2], "h":bbox[3]})
            else:
                print("internal error",type(classid))
                raise ValueError("incorrect type of classid")
        return objects

if __name__ == '__main__': # usage & test
    import os,sys,time,datetime,re,json,logging
    print(cv2.__version__,cv2.__file__)
    #y4 = yolov4("deprecated/darknet/cfg/coco.names","./yolov4.weights","./yolov4.cfg")
    y4 = yolov4("deprecated/darknet/cfg/coco.names","./yolov4-sam-mish.weights","./yolov4-sam-mish.cfg")
    img = cv2.imread("dog.jpg")

My python3 code uses Flask to call the above code, raises almost identical error messages, and then crashes. It spent me a lot of time to find countermeasures.

terminate called after throwing an instance of 'cv::dnn::cuda4dnn::csl::CUDAException'               
  what():  OpenCV(4.6.0-dev) /home/a/repos/opencv/modules/dnn/src/cuda4dnn/csl/memory.hpp:293: error:
(-217:Gpu API call) operation not permitted when stream is capturing in function '~MemoryLockGuard'

A workable solution is to avoid multi-thread since I have no idea to use mutex to protect the yolov4 so far.

app.run(debug=True,host='0.0.0.0',port=8080,threaded=False,ssl_context=('cert.pem', 'privkey.pem'))