iChizer0
commented
5 days ago Hi @ThomasBroch, we have written a minimal sample code for you, hopefully it resolves your confusion.
static void __task(void*) {
auto device = Device::getInstance();
auto sensors = device->getSensors();
MA_ASSERT(!sensors.empty());
auto sensor = sensors.front();
auto rc = sensor->init(1);
MA_ASSERT(rc == MA_OK);
auto models = device->getModels();
MA_ASSERT(!models.empty());
auto model = models.front();
MA_ASSERT(model.addr != nullptr);
auto engine = ma::engine::EngineTFLite();
rc = engine.init();
MA_ASSERT(rc == MA_OK);
rc = engine.load(model.addr, model.size);
MA_ASSERT(rc == MA_OK);
auto algo = ma::ModelFactory::create(&engine);
MA_ASSERT(algo != nullptr);
auto camera = static_cast<ma::Camera*>(sensor);
rc = camera->startStream(ma::Camera::StreamMode::kRefreshOnReturn);
MA_ASSERT(rc == MA_OK);
printf("Running algo: %d\n", algo->getType());
for (;;) {
auto frame = ma_img_t{};
rc = camera->retrieveFrame(frame, MA_PIXEL_FORMAT_AUTO);
if (rc != MA_OK) {
continue;
}
switch (algo->getType()) {
case MA_MODEL_TYPE_PFLD:
case MA_MODEL_TYPE_IMCLS:
case MA_MODEL_TYPE_YOLOV8_POSE:
// ...
break;
case MA_MODEL_TYPE_FOMO:
case MA_MODEL_TYPE_YOLOV5:
case MA_MODEL_TYPE_YOLOV8:
case MA_MODEL_TYPE_NVIDIA_DET:
case MA_MODEL_TYPE_YOLO_WORLD: {
auto D = static_cast<ma::model::Detector*>(algo);
rc = D->run(&frame);
if (rc != MA_OK) {
continue;
}
auto results = D->getResults();
for (auto& result : results) {
result.x *= frame.width;
result.y *= frame.height;
result.w *= frame.width;
result.h *= frame.height;
result.score *= 100;
}
printf("Results size: %d\n", static_cast<int>(
std::distance(results.begin(), results.end())
));
for (auto& result : results) {
printf("x: %f, y: %f, w: %f, h: %f, score: %f\n", result.x, result.y, result.w, result.h, result.score);
}
}
default:
// ...
break;
}
camera->returnFrame(frame);
}
camera->stopStream();
}You can test this by replacing the corresponding function in the following branch with the code above:
If you wish to customize the data source of your images, we recommend you to read the source code of https://github.com/Seeed-Studio/SSCMA-Micro/tree/refactor/sscma-deploy. And if there's any further questions, please contact us again, thank you!
ThomasBroch
Describe the feature Please provide example firmware code for running swift yolo models, made with model assistant, to run them on the Grove Vision Module V2.
Motivation Through model assistant, swift yolo models can be trained that can be deployed to the Grove Vision AI Module V2. There appears to be no firmware source code example available for running these swift yolo models.
There are various firmware example apps available in the firmware repository, but there don't appear to be any for running swift yolo. Whenever you load a swift yolo through the sensecraft web toolkit, it also downloads and flashes a firmware image to run the model, but the source code of that firmware is not available.
Additional context We are working on a custom implementation where we would like to send an image over serial to the grove, rather than using a camera module. For that, we need to write our own firmware, but we have not been able to find example firmware for running swift yolo.
I have been in contact about this with support and product line managers. They asked me to detail my issue here.