wwdok
commented
2 years ago I replaced tf.gather_nd with tf.gather and some other tf ops, and try to replace tf.gather with some other tf ops, then reexport the onnx model, but the model size become double size larger, so please consider support gather op in the future (gather_nd can be implemented based on gather), we need a op that can select element by index, this is a common op in tf and pytorch .
I quit implementing tf.gather with other tf ops, swith to implement it in ncnn custom layer :
This is the onnx that have tf.gather but not have tf.gather_nd :
model_float32_wo_gathernd_onnx.zip
This is the converted param and bin:
model_float32_wo_gathernd_param.zip
model_float32_wo_gathernd_bin.zip
This is the cumtom gather op:
#include "gather.h"
namespace ncnn {
Gather::Gather()
{
one_blob_only = true;
}
int Gather::forward(const std::vector<Mat>& bottom_blobs, std::vector<Mat>& top_blobs, const Option& opt) const
{
const Mat& params = bottom_blobs[0];
const Mat& indices = bottom_blobs[1];
int indices_rank = indices.dims;
if(indices_rank == 1){
Mat& top_blob = top_blobs[0];
top_blob.create(indices.w, params.h, params.elemsize, opt.blob_allocator);
if (top_blob.empty())
return -100;
// index params by indices, then store in top_blob
for(int i = 0; i < indices.w; i++){
// get index from indices at i
int index = indices[i];
// copy one value each time
memcpy(top_blob.row(i), params.row(index), params.elemsize*params.h);
}
}
if (indices_rank == 2)
{
Mat& top_blob = top_blobs[0];
top_blob.create(indices.w, indices.h, params.h, params.elemsize, opt.blob_allocator);
if (top_blob.empty())
return -100;
// index params by indices, then store in top_blob
#pragma omp parallel for num_threads(opt.num_threads)
for (int i = 0; i < indices.h; i++)
{
for (int j = 0; j < indices.w; j++)
{
// get index from indices at i,j
int index = indices[i*indices.h+j];
float* absolute_index = (float*)top_blob.data + i*indices.h+j;
memcpy(absolute_index, params.row(index), params.elemsize*params.h);
}
}
}
return 0;
}
DEFINE_LAYER_CREATOR(Gather)
} // namespace ncnn#pragma once
#include "layer.h"
namespace ncnn{
class Gather : public Layer
{
public:
Gather();
virtual int forward(const std::vector<Mat>& bottom_blobs, std::vector<Mat>& top_blobs, const Option& opt) const;
};
} // namespace ncnnThen I use followig code to test it:
#include "ncnn/net.h"
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>
int main(int argc, char** argv)
{
const char* imagepath = "C:/Users/wadewang/Pictures/face.bmp";
cv::Mat img = cv::imread(imagepath, 1);
if (img.empty())
{
fprintf(stderr, "cv::imread %s failed/n", imagepath);
return -1;
}
cv::imshow("IMAGE",img);
cv::waitKey(0);
ncnn::Net face_mesh_net;
face_mesh_net.load_param("E:/Repo/ncnn/ncnn/build/tools/onnx/model_float32_wo_gathernd.param");
face_mesh_net.load_model("E:/Repo/ncnn/ncnn/build/tools/onnx/model_float32_wo_gathernd.bin");
const int target_size = 192;
int img_w = img.cols;
int img_h = img.rows;
ncnn::Mat in = ncnn::Mat::from_pixels_resize(img.data, ncnn::Mat::PIXEL_BGR2RGB, img_w, img_h, target_size, target_size);
const float mean_vals[3] = {0.f, 0.f, 0.f};
const float norm_vals[3] = {1.0 / 255.0, 1.0 / 255.0, 1.0 / 255.0};
in.substract_mean_normalize(mean_vals, norm_vals);
ncnn::Extractor ex = face_mesh_net.create_extractor();
ex.input("input_1", in);
ncnn::Mat out;
ex.extract("output_mesh_identity", out);
// print values in mat
int width = out.w;
int height = out.h;
int channels = out.c;
printf("whc shape:(%d, %d, %d) \n", width, height, channels);
for (int c = 0; c < out.c; c++)
{
for (int i = 0; i < out.h; i++)
{
const float* values = out.row(i);
for (int j = 0; j < out.w; j++)
{
printf("in channel %d height %d widdth %d: %f \n", c,i,j, values[j]);
}
}
}
return 0;
}It output a lot of -nan(ind):
How to debug this problem ? BTW, I think it may not the problem of custom layer - gather, because the gather layer is after the exteacted blob output_mesh_identity.
Some output of first conv of onnx model :
nihui
error log | 日志或报错信息 | ログ
model | 模型 | モデル
~1. original model https://github.com/PINTO0309/PINTO_model_zoo/files/8357832/model_float32.onnx.zip
how to reproduce | 复现步骤 | 再現方法
~I upload model to convertmodel.com, it reports those errors. I also try onnx-simplifier, but it still can not simplify any of these unsupported ops. I download converted param and bin files, try to modify the param file according this article《手工优化ncnn模型结构》, but this model is more complicated, it is hard for me to manually modify it correctly. I use netron and Ctrl F, search out some of related ops in converted param file:~
~For the first graph, I modify: from~
to
~I don't know if it is correct. For the second graph, I don't know how to modify the GatherND. As far as I understand, to modify the param, we need to find the input node and output node, then merge out those intermediate ops, but in this case, it need other input(as the yellow indicate), so is it possible to merge them ? Following graph is corresponding part of onnx model:~