gyr-kdgc commented 3 years ago

01/12 11:39:25: Launching 'app' on HUAWEI EML-AL00. Install successfully finished in 1 s 701 ms. $ adb shell am start -n "com.mindspore.classification/com.mindspore.classification.widget.CameraActivity" -a android.intent.action.MAIN -c android.intent.category.LAUNCHER Connected to process 6425 on device 'huawei-eml_al00-CLB0218804004918'. Capturing and displaying logcat messages from application. This behavior can be disabled in the "Logcat output" section of the "Debugger" settings page. I/.classificatio: Late-enabling -Xcheck:jni E/.classificatio: Unknown bits set in runtime_flags: 0x8000 I/.classificatio: Reinit property: dalvik.vm.checkjni= false E/libc: Access denied finding property "runtime.mmitest.isrunning" D/ActivityThread: Attach thread to application I/.classificatio: QarthPatchMonintor::Init QarthPatchMonintor::StartWatch QarthPatchMonintor::WatchPackage: /data/hotpatch/fwkhotpatch/ QarthPatchMonintor::CheckAndWatchPatch: /data/hotpatch/fwkhotpatch/com.mindspore.classification QarthPatchMonintor::CheckAndWatchPatch: /data/hotpatch/fwkhotpatch/all QarthPatchMonintor::Run I/.classificatio: QarthPatchMonintor::Reading QarthPatchMonintor::CheckNotifyEvent QarthPatchMonintor::CheckNotifyEvent before read D/OpenGLRenderer: disableOutlineDraw is true I/HwApiCacheMangerEx: apicache path=/storage/emulated/0 state=mounted key=com.mindspore.classification#10219#256 apicache path=/storage/emulated/0 state=mounted key=com.mindspore.classification#10219#0 I/AwareBitmapCacher: init processName:com.mindspore.classification pid=6425 uid=10219 E/AwareLog: AtomicFileUtils: readFileLines file not exist: android.util.AtomicFile@6f503ce E/AwareLog: AtomicFileUtils: readFileLines file not exist: android.util.AtomicFile@ccf92ef V/ActivityThread: callActivityOnCreate D/CameraActivity: onCreate V/HwWidgetFactory: : successes to get AllImpl object and return.... W/.classificatio: Accessing hidden method Landroid/view/View;->computeFitSystemWindows(Landroid/graphics/Rect;Landroid/graphics/Rect;)Z (greylist, reflection, allowed) W/.classificatio: Accessing hidden method Landroid/view/ViewGroup;->makeOptionalFitsSystemWindows()V (greylist, reflection, allowed) D/ActivityThread: add activity client record, r= ActivityRecord{b81e1d7 token=android.os.BinderProxy@b9724cd {com.mindspore.classification/com.mindspore.classification.widget.CameraActivity}} token= android.os.BinderProxy@b9724cd I/TrackingMobile: load libiMindSpore.so successfully.

W/MS_LITE: [mindspore/lite/src/model_common.cc:121] ImportFromBuffer] model version is MindSpore Lite 1.1.0, inference version is MindSpore Lite 1.0.0 not equal D/CameraFragment: Loading model return value: true I/CameraManagerGlobal: Connecting to camera service D/OpenGLRenderer: disableOutlineDraw is true D/HiTouch_PressGestureDetector: onAttached, package=com.mindspore.classification, windowType=1, mHiTouchRestricted=false I/iGraphics: [0020080c] pn: com.mindspore.classification, p: 6425 [0030080c] no spt app: com.mindspore.classification D/mali_winsys: EGLint new_window_surface(egl_winsys_display , void , EGLSurface, EGLConfig, egl_winsys_surface , EGLBoolean) returns 0x3000 D/CameraFragment: displayRotation: 0 D/HwFrameworkSecurityPartsFactory: HwFrameworkSecurityPartsFactory in. I/HwFrameworkSecurityPartsFactory: add HwFrameworkSecurityPartsFactory to memory. I/CameraManager: open camera: 0, package name: com.mindspore.classification I/HwCameraUtil: notifySurfaceFlingerCameraStatus : isFront = false , isOpend = true W/Gralloc3: mapper 3.x is not supported W/Gralloc3: allocator 3.x is not supported E/: APS:IFLoad:importExternalFunctions, search function createNewHwApsUtils failed, dlsym err:undefined symbol: createNewHwApsUtils** D/: APS:importExternalFunctions OK I/HwViewRootImpl: removeInvalidNode jank list is null I/MSJNI: MindSpore get session. Number of tensor elements:320 MindSpore scores[1] : [0.741127] MindSpore scores[2] : [2.156142] MindSpore scores[3] : [0.751505] MindSpore scores[7] : [1.094427]

A/libc: Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x3dcd163bbd200de0 in tid 6521 (MINDSPORE), pid 6425 (.classification)

I/CameraDevice-JV-0: close camera: 0, package name: com.mindspore.classification I/HwCameraUtil: notifySurfaceFlingerCameraStatus : isFront = false , isOpend = false

我在mobilenetv2模型部署代码的基础上进行修改，将模型文件改为自己的模型，并且修改了相应的类别的路径，原先的mobilenetv2部署都正常，更改之后就出现以上错误，并且手机上的app闪退，请问是什么问题？还需要修改哪些地方的代码？我用的是MNIST数据集训练出来的模型进行的测试。

gyr-kdgc commented 3 years ago

我修改了图像预处理的参数之后，模型不闪退了，A/libc: Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x3dcd163bbd200de0 in tid 6521 (MINDSPORE), pid 6425 (.classification)错误没有了，但是上面的其他错误还是存在。后来我又用另一个二分类模型进行测试，又有新的报错： W/libc: malloc(18446744073708478804) failed: returning null pointer E/MS_LITE: [mindspore/lite/src/tensor.cc:328] MallocData] Malloc tensor data failed, size=18446744073708478804 W/MS_LITE: [mindspore/lite/src/tensor.cc:353] MutableData] Malloc data failed

memcpy(inTensor->MutableData(), dataHWC, inputDims.channel inputDims.width inputDims.height * sizeof(float)); 发现是这条语句，分配内存错误，请问是什么原因？

chaosisnotopen commented 3 years ago

1、闪退的问题，提供的信息太少，看不出来 2、内存失败，看你提供的信息，应该是shape参数有问题如果可能，可以提供下你的模型和改的代码吗？

gyr-kdgc commented 3 years ago

1、闪退的问题，提供的信息太少，看不出来 2、内存失败，看你提供的信息，应该是shape参数有问题如果可能，可以提供下你的模型和改的代码吗？ --------------------------------------------------修改的代码部分-------------------------------------------------------------- static const int RET_CATEGORY_SUM = 2;//类别数 //类别名称 static const char *labels_name_map[RET_CATEGORY_SUM] = { "yes", "no"}; //类别阈值映射 static float g_thres_map[RET_CATEGORY_SUM] = { 0.2, 0.2, };

bool PreProcessImageData(const LiteMat &lite_mat_bgr, LiteMat lite_norm_mat_ptr) { MS_PRINT("---------------PreProcessImageData running--------------"); bool ret = false; LiteMat lite_mat_resize; LiteMat &lite_norm_mat_cut = lite_norm_mat_ptr; ret = ResizeBilinear(lite_mat_bgr, lite_mat_resize, 299, 299); if (!ret) { MS_PRINT("ResizeBilinear error"); return false; } LiteMat lite_mat_convert_float; ret = ConvertTo(lite_mat_resize, lite_mat_convert_float, 1.0 / 255.0); if (!ret) { MS_PRINT("ConvertTo error"); return false; } LiteMat lite_mat_cut; ret = Crop(lite_mat_convert_float, lite_mat_cut, 0, 0, 299, 299); if (!ret) { MS_PRINT("Crop error"); return false; } std::vector means = {0.5, 0.5, 0.5}; std::vector stds = {0.5, 0.5, 0.5}; SubStractMeanNormalize(lite_mat_cut, lite_norm_mat_cut, means, stds); return true; }

就修改了以上代码，其他代码没有修改。闪退问题应该就是shape参数设置不合理导致的，后来的MNIST模型已经可以部署，但是换了另一个二分类模型之后就出现以上的内存问题。我的模型是tensorflow的pb文件转换过来的，

gyr-kdgc commented 3 years ago

def img_pre_proc(filename): try: img = cv2.imread(filename) img = cv2.resize(img, (299, 299)) img = img / 255.0 img = img - 0.5 img = img * 2 return img, 1 except: return 0, 0

使用的是inceptionv4训练的分类网络，训练过程中的图像预处理就这些

AssassinGQ commented 3 years ago

训练完的模型能否用可视化工具打开一下，看一下输入的shape是多少，或者在代码里加一下打印看一下整图输入的tensor的shape。可以在CompileGraph接口被调用之后，memcpy(inTensor->MutableData(), dataHWC, inputDims.channel inputDims.width inputDims.height * sizeof(float));这一句代码之前，打印一下inTensor的shape。麻烦把打印出来的shape发出来看一下

gyr-kdgc commented 3 years ago

训练完的模型能否用可视化工具打开一下，看一下输入的shape是多少，或者在代码里加一下打印看一下整图输入的tensor的shape。可以在CompileGraph接口被调用之后，memcpy(inTensor->MutableData(), dataHWC, inputDims.channel inputDims.width inputDims.height * sizeof(float));这一句代码之前，打印一下inTensor的shape。麻烦把打印出来的shape发出来看一下

我直接使用原来的pb模型推理，打印的输入图片尺寸是（299，299，3）；在memcpy(inTensor->MutableData(), dataHWC,inputDims.channel inputDims.width inputDims.height * sizeof(float));这句代码之前打印了inTensor的shape，是==============[-1,299,299,3]============（打印的代码为MS_PRINT("==============[%d,%d,%d,%d]============",inTensor->shape().at(0), inTensor->shape().at(1), inTensor->shape().at(2), inTensor->shape().at(3));）这么看二者的输入图片形状是一样的啊

gyr-kdgc commented 3 years ago

还想问以下如果我想对输入的图片进行HWC到CHW的转换，我应该怎么做？我引入了std::shared_ptr HWC2CHW();看了说明，返回的是一个算子，但是应该怎么用这个算子呢？

AssassinGQ commented 3 years ago

训练完的模型能否用可视化工具打开一下，看一下输入的shape是多少，或者在代码里加一下打印看一下整图输入的tensor的shape。可以在CompileGraph接口被调用之后，memcpy(inTensor->MutableData(), dataHWC, inputDims.channel inputDims.width inputDims.height * sizeof(float));这一句代码之前，打印一下inTensor的shape。麻烦把打印出来的shape发出来看一下

我直接使用原来的pb模型推理，打印的输入图片尺寸是（299，299，3）；在memcpy(inTensor->MutableData(), dataHWC,inputDims.channel inputDims.width inputDims.height * sizeof(float));这句代码之前打印了inTensor的shape，是==============[-1,299,299,3]============（打印的代码为MS_PRINT("==============[%d,%d,%d,%d]============",inTensor->shape().at(0), inTensor->shape().at(1), inTensor->shape().at(2), inTensor->shape().at(3));）这么看二者的输入图片形状是一样的啊

如果inTensor的shape是[-1,299,299,3]的话，说明原始模型需要Resize，如果在没有Resize的情况下直接调用MutableData，会导致异常的内存分配，这个校验在最新版本里已经加了，不知道你用的是什么版本。不过无论如何，这里都会异常退出，如果要正常执行，需要在CompileGraph之后，MutableData之前调用LiteSession的Resize接口。Resize的使用方式：https://www.mindspore.cn/tutorial/lite/zh-CN/r1.1/use/runtime_cpp.html#id8

gyr-kdgc commented 3 years ago

训练完的模型能否用可视化工具打开一下，看一下输入的shape是多少，或者在代码里加一下打印看一下整图输入的tensor的shape。可以在CompileGraph接口被调用之后，memcpy(inTensor->MutableData(), dataHWC, inputDims.channel inputDims.width inputDims.height * sizeof(float));这一句代码之前，打印一下inTensor的shape。麻烦把打印出来的shape发出来看一下

我直接使用原来的pb模型推理，打印的输入图片尺寸是（299，299，3）；在memcpy(inTensor->MutableData(), dataHWC,inputDims.channel inputDims.width inputDims.height * sizeof(float));这句代码之前打印了inTensor的shape，是==============[-1,299,299,3]============（打印的代码为MS_PRINT("==============[%d,%d,%d,%d]============",inTensor->shape().at(0), inTensor->shape().at(1), inTensor->shape().at(2), inTensor->shape().at(3));）这么看二者的输入图片形状是一样的啊

如果inTensor的shape是[-1,299,299,3]的话，说明原始模型需要Resize，如果在没有Resize的情况下直接调用MutableData，会导致异常的内存分配，这个校验在最新版本里已经加了，不知道你用的是什么版本。不过无论如何，这里都会异常退出，如果要正常执行，需要在CompileGraph之后，MutableData之前调用LiteSession的Resize接口。Resize的使用方式：https://www.mindspore.cn/tutorial/lite/zh-CN/r1.1/use/runtime_cpp.html#id8

D/: APS:importExternalFunctions OK I/HwViewRootImpl: removeInvalidNode jank list is null I/MSJNI: MindSpore get session. E/MS_LITE: [mindspore/lite/src/ops/stack.cc:100] InferShape] All input shape size should be the same! [mindspore/lite/src/sub_graph_kernel.cc:138] ReSize] InferShape failed, type: Stack [mindspore/lite/src/scheduler.cc:82] ReSizeKernels] ReSize node CpuFP32SubGraph failed [mindspore/lite/src/ops/stack.cc:100] InferShape] All input shape size should be the same! [mindspore/lite/src/sub_graph_kernel.cc:138] ReSize] InferShape failed, type: Stack [mindspore/lite/src/scheduler.cc:82] ReSizeKernels] ReSize node CpuFP32SubGraph failed [mindspore/lite/src/lite_session.cc:529] Resize] restore kernel size fail!ret: -1 I/MSJNI: ==============[-1,299,299,3]============ W/libc: malloc(18446744073708478804) failed: returning null pointer E/MS_LITE: [mindspore/lite/src/tensor.cc:328] MallocData] Malloc tensor data failed, size=18446744073708478804 W/MS_LITE: [mindspore/lite/src/tensor.cc:353] MutableData] Malloc data failed =====================代码============================== auto mSession = labelNet->session(); if (mSession == nullptr) { MS_PRINT("MindSpore error, Session is a nullptr."); return NULL; } MS_PRINT("MindSpore get session.");

auto msInputs = mSession->GetInputs(); if (msInputs.size() == 0) { MS_PRINT("MindSpore error, msInputs.size() equals 0."); return NULL; }

std::vector resize_shape = {1, 299, 299, 3}; std::vector<std::vector> new_shapes; new_shapes.push_back(resize_shape); mSession->Resize(msInputs, new_shapes);

auto inTensor = msInputs.front();

float dataHWC = reinterpret_cast<float >(lite_norm_mat_cut.dataptr); // Copy dataHWC to the model input tensor. MS_PRINT("==============[%d,%d,%d,%d]============",inTensor->shape().at(0), inTensor->shape().at(1), inTensor->shape().at(2), inTensor->shape().at(3)); memcpy(inTensor->MutableData(), dataHWC, inputDims.channel inputDims.width inputDims.height * sizeof(float));

我添加了Resize代码，但是好像报错了，这好像没有resize成功

AssassinGQ commented 3 years ago

确实是resize失败了，可以把模型中stack算子前后的部分截图出来看一下吗

gyr-kdgc commented 3 years ago

确实是resize失败了，可以把模型中stack算子前后的部分截图出来看一下吗 extern "C" JNIEXPORT jlong JNICALL Java_com_mindspore_classification_gallery_classify_TrackingMobile_loadModel(JNIEnv *env, jobject thiz, jobject model_buffer, jint num_thread) { if (nullptr == model_buffer) { MS_PRINT("error, buffer is nullptr!"); return (jlong) nullptr; } jlong bufferLen = env->GetDirectBufferCapacity(model_buffer); if (0 == bufferLen) { MS_PRINT("error, bufferLen is 0!"); return (jlong) nullptr; }

char *modelBuffer = CreateLocalModelBuffer(env, model_buffer); if (modelBuffer == nullptr) { MS_PRINT("modelBuffer create failed!"); return (jlong) nullptr; }

// To create a mindspore network inference environment. void *labelEnv = new void ; MSNetWork labelNet = new MSNetWork; labelEnv = labelNet;

mindspore::lite::Context *context = new mindspore::lite::Context; context->threadnum = num_thread; context->devicelist[0].deviceinfo.cpu_deviceinfo.cpu_bindmode = mindspore::lite::NO_BIND; context->devicelist[0].deviceinfo.cpu_deviceinfo.enablefloat16 = false; context->devicelist[0].devicetype = mindspore::lite::DT_CPU;

labelNet->CreateSessionMS(modelBuffer, bufferLen, context); delete context;

if (labelNet->session() == nullptr) { MS_PRINT("MindSpore create session failed!."); delete labelNet; delete labelEnv; return (jlong) nullptr; }

if (model_buffer != nullptr) { env->DeleteLocalRef(model_buffer); }

return (jlong) labelEnv; }

extern "C" JNIEXPORT jstring JNICALL Java_com_mindspore_classification_gallery_classify_TrackingMobile_runNet(JNIEnv *env, jclass type, jlong netEnv, jobject srcBitmap) { LiteMat lite_mat_bgr, lite_norm_mat_cut;

if (!BitmapToLiteMat(env, srcBitmap, &lite_mat_bgr)) { MS_PRINT("BitmapToLiteMat error"); return NULL; } if (!PreProcessImageData(lite_mat_bgr, &lite_norm_mat_cut)) { MS_PRINT("PreProcessImageData error"); return NULL; }

ImgDims inputDims; inputDims.channel = lite_norm_matcut.channel; inputDims.width = lite_norm_matcut.width; inputDims.height = lite_norm_matcut.height;

// Get the mindsore inference environment which created in loadModel(). void labelEnv = reinterpret_cast<void >(netEnv); if (labelEnv == nullptr) { MS_PRINT("MindSpore error, labelEnv is a nullptr."); return NULL; } MSNetWork labelNet = static_cast<MSNetWork >(*labelEnv);

auto mSession = labelNet->session(); if (mSession == nullptr) { MS_PRINT("MindSpore error, Session is a nullptr."); return NULL; } MS_PRINT("MindSpore get session.");

auto msInputs = mSession->GetInputs(); if (msInputs.size() == 0) { MS_PRINT("MindSpore error, msInputs.size() equals 0."); return NULL; } //------------------对原始模型的输入进行resize--------------------- std::vector resize_shape = {1, 224, 224, 3}; std::vector<std::vector> new_shapes; new_shapes.push_back(resize_shape); mSession->Resize(msInputs, new_shapes); //--------------------------------------------------------------- auto inTensor = msInputs.front();

float dataHWC = reinterpret_cast<float >(lite_norm_mat_cut.dataptr); // Copy dataHWC to the model input tensor. MS_PRINT("==============[%d,%d,%d,%d]============",inTensor->shape().at(0), inTensor->shape().at(1), inTensor->shape().at(2), inTensor->shape().at(3)); memcpy(inTensor->MutableData(), dataHWC, inputDims.channel inputDims.width inputDims.height * sizeof(float));

// After the model and image tensor data is loaded, run inference. auto status = mSession->RunGraph();

if (status != mindspore::lite::RET_OK) { MS_PRINT("MindSpore run net error."); return NULL; }

/**

Get the mindspore inference results.
Return the map of output node name and MindSpore Lite MSTensor. / auto names = mSession->GetOutputTensorNames(); std::unordered_map<std::string, mindspore::tensor::MSTensor > msOutputs; for (const auto &name : names) { auto temp_dat = mSession->GetOutputByTensorName(name); msOutputs.insert(std::pair<std::string, mindspore::tensor::MSTensor *> {name, temp_dat}); }

std::string resultStr = ProcessRunnetResult(::RET_CATEGORY_SUM, ::labels_name_map, msOutputs); const char *resultCharData = resultStr.c_str(); return (env)->NewStringUTF(resultCharData); }

是这一部分吗？改动的就这些，其他的都没改动

mindspore-ai / mindspore

模型部署到手机上报错 #104