Megvii-BaseDetection / YOLOX

YOLOX is a high-performance anchor-free YOLO, exceeding yolov3~v5 with MegEngine, ONNX, TensorRT, ncnn, and OpenVINO supported. Documentation: https://yolox.readthedocs.io/
Apache License 2.0
9.15k stars 2.15k forks source link

When running the Core ML model converted with coremltools in Swift, the objectness score behaves oddly only on the actual device #1780

Closed hiroalchem closed 4 weeks ago

hiroalchem commented 4 weeks ago

I wanted to use YOLOX for iOS, so I created a yolox.mlpackage using coremltools and developed a Swift app. Although there were no issues in the simulator, when I tested it on an actual device (iPhone 15 Pro), I noticed that the objectness score was abnormally high. Here is the code for loading the model and using it in the app. When I run this code, the objectness score exceeds 1, although other values do not show any abnormalities. Could anyone provide some advice?

    private func loadModel() {
        do {
            let model = try yolox()
            self.model = try VNCoreMLModel(for: model.model)
        } catch {
            errorMessage = "Failed to load model: \(error.localizedDescription)"
            print(errorMessage!)
        }
    }

    private func classifyImage() {
        print("classifyImage called")
        guard let uiImage = image, let ciImage = CIImage(image: uiImage) else {
            print("Image is not available for classification")
            return
        }

        guard let validModel = model else {
            print("Model is not loaded or available for classification")
            return
        }

        let request = VNCoreMLRequest(model: validModel) { request, error in
            if let error = error {
                print("Classification failed with error: \(error.localizedDescription)")
                return
            }

            guard let results = request.results as? [VNCoreMLFeatureValueObservation] else {
                print("Classification returned no results or wrong results type.")
                return
            }

            print("Total results returned: \(results.count)")
            if results.isEmpty {
                print("No results were returned.")
            } else {
                let boxes = self.postProcessFeatureValueObservations(results)
                DispatchQueue.main.async {
                    self.displayResults(boxes: boxes, on: uiImage)
                }
                print(boxes)
            }
        }

        let handler = VNImageRequestHandler(ciImage: ciImage)
        do {
            try handler.perform([request])
            print("Classification request performed")
        } catch {
            print("Failed to perform classification: \(error.localizedDescription)")
        }
    }

    func postProcessFeatureValueObservations(_ observations: [VNCoreMLFeatureValueObservation]) -> [BoundingBox] {
        var boxes: [BoundingBox] = []

        for observation in observations {
            guard let multiArray = observation.featureValue.multiArrayValue else { continue }
            let count = multiArray.count / 6  
            for i in 0..<count {
                let index = i * 6
                // セーフに各要素の値を取得
                let centerX = CGFloat(multiArray[index].floatValue)
                let centerY = CGFloat(multiArray[index + 1].floatValue)
                let width = CGFloat(multiArray[index + 2].floatValue)
                let height = CGFloat(multiArray[index + 3].floatValue)
                let score = Float(multiArray[index + 4].floatValue)
                let classScore = Float(multiArray[index + 5].floatValue)

                let x1 = centerX - width / 2.0
                let y1 = centerY - height / 2.0
                let x2 = centerX + width / 2.0
                let y2 = centerY + height / 2.0

                let rect = CGRect(x: x1, y: y1, width: x2 - x1, height: y2 - y1)
                if score * classScore > 0.1 { 
                    print(score)
                    boxes.append(BoundingBox(classScore: classScore, score: score, rect: rect))
                    if score > 1{
                        print("over1")
                    }
                }
            }
        }
hiroalchem commented 4 weeks ago

I passed "compute_precision=coremltools.precision.FLOAT32" when I convert model, then problem solved.