Open VojtenkoRN opened 1 year ago
To debug this problem, the first observation from the output picture is that the bounding boxes are not at the right positions. So in the detection result DetectedObjects
, you can check the x, y, w, h of the boundingBoxes
to see if they are indeed wrong numbers. You can also in the same DetectedObjects
check the classNames
. After this, you will know if it is purely plotting problem (since the bounding boxes numbers should be compatible with the image size), or the postprocessing.
You can take a look at this example: /djl/examples/src/main/java/ai/djl/examples/inference/MaskDetection.java
. It is similar to your usecase. Check how the images are rescaled, if needed. The relevant PR is #2452 which contains detailed .md file.
Also the first check might be the check on the onnx model. You can use python to load it and inference, and see what the correct boundingBoxes
and classNames
should be.
1) I've added optArgument("optApplyRatio", true)
and .optArgument("rescale", true)
but it doesn't change anything.
2) As i said earlier, there were too many objects detected and all of them had wrong bounding boxes (result for video frame above)
3) I changed torchscript to onnx and installed TensorRT. I also rewrote code like your example (branch onnx-model).
It throws File not found: ~/.djl.ai/cache/repo/model/undefined/ai/djl/localmodelzoo/51b9a350d1dd717c7e5d1c1c133e6afb8d7c12f0/synset.txt
. I haven't found in docs how to specify synset for YoloV5TranslatorFactory. But if I put synset.txt into that path in djl.id/cache (which is rather uncomfortable) it throws UnsupportedOperationException: This NDArray implementation does not currently support this operation
when trying to predict.
Synset.txt:
A few obvious issue:
First of all, .optArgument()
only affects TranslatorFactory, you are directly passing Translator. In your case, you need configure your Translator directly:
Translator<Image, DetectedObjects> translator = YoloV5Translator
.builder()
.setPipeline(pipeline)
.optSynset(Synset.asNameList())
.optThreshold(THRESHOLD)
.optRescaleSize(IMAGE_SIZE, IMAGE_SIZE)
.optApplyRatio(true)
.optOutputType(YoloV5Translator.YoloOutputType.AUTO)
.build();
Another small issue is you have the following code:
.optDevice(Device.gpu())
.optDevice(Device.cpu())
Thanks for advice! I rewrote Translator as you said and changed model for using it on gpu (my bad). The problems still remain :(
Detected:
On screen:
UPD: All changes committed
I tried your model and it seems working fine:
Path imageFile = Paths.get("src/test/resources/dog_bike_car.jpg");
Image img = ImageFactory.getInstance().fromFile(imageFile);
Criteria<Image, DetectedObjects> criteria =
Criteria.builder()
.optApplication(Application.CV.OBJECT_DETECTION)
.setTypes(Image.class, DetectedObjects.class)
.optModelPath(Paths.get("video-detection-example/model/yolov5x.torchscript"))
.optEngine("PyTorch")
.optArgument("width", "640")
.optArgument("height", "640")
.optArgument("resize", "true")
.optArgument("rescale", "true")
.optArgument("optApplyRatio", "true")
.optArgument("threshold", "0.4")
.optArgument("synsetUrl", "https://djl-ai.s3.amazonaws.com/mlrepo/model/cv/object_detection/ai/djl/pytorch/classes_coco.txt")
.optTranslatorFactory(new YoloV5TranslatorFactory())
.optProgress(new ProgressBar())
.build();
try (ZooModel<Image, DetectedObjects> model = criteria.loadModel();
Predictor<Image, DetectedObjects> predictor = model.newPredictor()) {
DetectedObjects detection = predictor.predict(img);
saveBoundingBoxImage(img, detection);
return detection;
}
Ok, thanks a lot! I'll check it as best I can.
But is there any other way to pass synset list except .optArgument("synsetUrl", "...")
for using in .optTranslatorFactory(new YoloV5TranslatorFactory())
? Because put synset.txt in ~/djl.id/cache/.../
is rather uncomfortable :(
@VojtenkoRN
Yes. you can use .optArgument("synset", "dog,cat,car,...")
, but we didn't do any csv like comma escape, so it has limitation.
Thank you all!
Option 1 (was):
Pipeline pipeline = new Pipeline()
.add(new Resize(IMAGE_SIZE))
.add(new ToTensor());
Translator<Image, DetectedObjects> translator = YoloV5Translator
.builder()
.setPipeline(pipeline)
.optSynset(Synset.asNameList())
.optThreshold(THRESHOLD)
.optRescaleSize(IMAGE_SIZE, IMAGE_SIZE)
.optApplyRatio(true)
.optOutputType(YoloV5Translator.YoloOutputType.AUTO)
.build();
Criteria<Image, DetectedObjects> criteria = Criteria.builder()
.setTypes(Image.class, DetectedObjects.class)
.optModelUrls(modelPath)
.optModelName(modelName)
.optDevice(Device.gpu())
.optApplication(Application.CV.OBJECT_DETECTION)
.optTranslator(translator)
.optEngine(Engine.getDefaultEngineName())
.optProgress(new ProgressBar())
.build();
Option 2 (become, synset passed from code, master)
Criteria<Image, DetectedObjects> criteria = Criteria.builder()
.setTypes(Image.class, DetectedObjects.class)
.optModelUrls(modelPath)
.optModelName(modelName)
.optDevice(Device.gpu())
.optApplication(Application.CV.OBJECT_DETECTION)
.optEngine(Engine.getDefaultEngineName())
.optArgument("width", IMAGE_SIZE)
.optArgument("height", IMAGE_SIZE)
.optArgument("resize", "true")
.optArgument("rescale", "true")
.optArgument("optApplyRatio", "true")
.optArgument("threshold", THRESHOLD)
.optArgument("synset", Synset.asString())
.optTranslatorFactory(new YoloV5TranslatorFactory())
.optProgress(new ProgressBar())
.build();
Result:
Option 3 (become, synset passed from txt, synset-in-txt)
final var synsetUrl = Path.of(synsetPath).toUri().toString();
Criteria<Image, DetectedObjects> criteria = Criteria.builder()
.setTypes(Image.class, DetectedObjects.class)
.optModelUrls(modelPath)
.optModelName(modelName)
.optDevice(Device.gpu())
.optApplication(Application.CV.OBJECT_DETECTION)
.optEngine(Engine.getDefaultEngineName())
.optArgument("width", IMAGE_SIZE)
.optArgument("height", IMAGE_SIZE)
.optArgument("resize", "true")
.optArgument("rescale", "true")
.optArgument("optApplyRatio", "true")
.optArgument("threshold", THRESHOLD)
.optArgument("synsetUrl", synsetUrl)
.optTranslatorFactory(new YoloV5TranslatorFactory())
.optProgress(new ProgressBar())
.build();
Result (this workaround seems to be working):
But it seems a bit odd to me that option 1 and option 2 doesn't work, while option 3 does. Although outwardly there is not much difference between them. Especially when you consider that all classes seem to be defined correctly in option 2
Thanks for help again! If you think that it isn't a bug - you can close that issue :)
I don't see any difference between all options, I just test with the following code, and works fine:
String url = "https://djl-ai.s3.amazonaws.com/mlrepo/model/cv/object_detection/ai/djl/pytorch/classes_coco.txt";
List<String> list;
try (InputStream is = new URL(url).openStream()) {
list = Utils.readLines(is);
}
String synset = String.join(",", list);
and:
String url = "https://djl-ai.s3.amazonaws.com/mlrepo/model/cv/object_detection/ai/djl/pytorch/classes_coco.txt";
List<String> list;
try (InputStream is = new URL(url).openStream()) {
list = Utils.readLines(is);
}
Pipeline pipeline = new Pipeline()
.add(new Resize(640))
.add(new ToTensor());
Translator<Image, DetectedObjects> translator = YoloV5Translator
.builder()
.setPipeline(pipeline)
.optSynset(list)
.optThreshold(0.4f)
.optRescaleSize(640, 640)
.optApplyRatio(true)
.optOutputType(YoloV5Translator.YoloOutputType.AUTO)
.build();
Description
In my quarkus project I'm trying to use DJL with a yolov5x model for multiple objects detection in a video. I passed all ms coco class names to the synset and then wanted to filter a result. But too many objects were detected and all of them had wrong bounding boxes
Expected Behavior
Correct detection with correct bounding boxes
Error image example
How to Reproduce?
Example
Steps to reproduce
Just run the app as the README said
What have you tried to solve it?
I've tried to pass single class name to synset (80 times), but I need several classes and it starts to detect all objects as passed class.
Environment Info
Engine.debugEnvironment()
result: