Open whalefa1I opened 3 years ago
saudet
commented
3 years ago Can you try to set the "org.bytedeco.javacpp.nopointergc" system property to "true" and see if that helps? It may be that the reference to one of those objects get lost, which could end up in JavaCPP deallocating memory prematurely when GC is used.
whalefa1I
commented
3 years ago Can you try to set the "org.bytedeco.javacpp.nopointergc" system property to "true" and see if that helps? It may be that the reference to one of those objects get lost, which could end up in JavaCPP deallocating memory prematurely when GC is used.
Will the "org.bytedeco.javacpp.nopointergc" setting and try ( PointerScope scope = new PointerScope()) conflict?
saudet
commented
3 years ago No, why would they conflict? If you're still getting a crash, the issue is somewhere else.
whalefa1I
commented
3 years ago No, why would they conflict? If you're still getting a crash, the issue is somewhere else.
Thank you for your reply! I will set the system property today and re-test.
whalefa1I
commented
3 years ago well,there is another fatal error... 1.log A fatal error has been detected by the Java Runtime Environment:
SIGSEGV (0xb) at pc=0x00007f14d50520d0, pid=3028, tid=0x00007f13904f7700 # JRE version: Java(TM) SE Runtime Environment (8.0_111-b14) (build 1.8.0_111-b14) Java VM: Java HotSpot(TM) 64-Bit Server VM (25.111-b14 mixed mode linux-amd64 compressed oops) Problematic frame: v ~StubRoutines::jbyte_disjoint_arraycopy
Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
If you would like to submit a bug report, please visit: http://bugreport.java.com/bugreport/crash.jsp --------------- T H R E A D ---------------
Current thread (0x00007f138c419800): JavaThread "grpc-executor-353" daemon [_thread_in_Java, id=15465, stack(0x00007f13903f7000,0x00007f13904f8000)]
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00007f1360000040
Registers: RAX=0x00007f1360001003, RBX=0x0000000000000000, RCX=0x0000000000000fc3, RDX=0xfffffffffffffe0c RSP=0x00007f13904f5880, RBP=0x00007f13904f5880, RSI=0x00000007b05efd85, RDI=0x00007f1360000ff8 R8 =0x00000007b05efe80, R9 =0x000000000004b000, R10=0x00007f14d5053080, R11=0x00000007b05eed58 R12=0x0000000000000000, R13=0x000000000004b000, R14=0x0000000000000fc3, R15=0x00007f138c419800 RIP=0x00007f14d50520d0, EFLAGS=0x0000000000010286, CSGSFS=0x0000000000000033, ERR=0x0000000000000004 TRAPNO=0x000000000000000e
Top of Stack: (sp=0x00007f13904f5880) 0x00007f13904f5880: 00000007b05eed58 00007f14d66e8236 0x00007f13904f5890: 00000007b05efe80 0000000700000000 0x00007f13904f58a0: 00000007b05eda18 00000022d5f38332 0x00007f13904f58b0: 00000006c01c7938 00000007a67e02c8 0x00007f13904f58c0: 0000000000000004 00007f14d58daf58 0x00007f13904f58d0: 00000007a67e02c8 00000ffca67e03f8 0x00007f13904f58e0: 0000000000000012 00007f14d5db1df8 0x00007f13904f58f0: 00000007b05ee088 00000007b05eed58 0x00007f13904f5900: 0004b01a00000001 00000000f4cfc059 0x00007f13904f5910: 00000007a67e02c8 00000007b05eed58 0x00007f13904f5920: 0000000000000008 0004b01a00000000 0x00007f13904f5930: 000000000004b01a 0000000000000000 0x00007f13904f5940: 00000007b05eed58 00007f14d6142808 0x00007f13904f5950: 00000007b05eed58 00000007b05eed58 0x00007f13904f5960: 0004b01a00000018 0000000e0004b01a 0x00007f13904f5970: 00000007b05eed58 00000007b05ee088 0x00007f13904f5980: 00000007a67e03d0 d8038f7800000000 0x00007f13904f5990: 00000000d819a654 00007f14d537b6f0 0x00007f13904f59a0: 00000007b05eed58 00000006c0d64410 0x00007f13904f59b0: d81a1092d81ac882 f60bdc11d819a654 0x00007f13904f59c0: 0000000a00000002 00000007b05eed58 0x00007f13904f59d0: 00000006c0d08490 f60bdc11d81ac882 0x00007f13904f59e0: 0000000600000007 00007f14d58d2494 0x00007f13904f59f0: 00000007a67e03d0 0000000000000001 0x00007f13904f5a00: 000000000004b026 00007f14d5b84068 0x00007f13904f5a10: 00000000d80c9670 00000007b05efe58 0x00007f13904f5a20: 00000007b05efe28 00000007b05eed58 0x00007f13904f5a30: 0000000000000000 00007f14d5e1f428 0x00007f13904f5a40: 00000006c0004638 00000006c0cd32a0 0x00007f13904f5a50: 0000000000001000 00007f14d63e045c 0x00007f13904f5a60: 00000007b05efe38 00000006c0cd32a0 0x00007f13904f5a70: 00000007b05ee4a0 00007f14d6c4ba74
Instructions: (pc=0x00007f14d50520d0) 0x00007f14d50520b0: 46 08 48 83 c7 02 48 83 c6 02 f7 c1 01 00 00 00 0x00007f14d50520c0: 74 06 8a 47 08 88 46 08 48 33 c0 c9 c3 66 66 90 0x00007f14d50520d0: 48 8b 44 d7 e8 48 89 44 d6 e8 48 8b 44 d7 f0 48 0x00007f14d50520e0: 89 44 d6 f0 48 8b 44 d7 f8 48 89 44 d6 f8 48 8b
Register to memory mapping:
RAX=0x00007f1360001003 is an unknown value RBX=0x0000000000000000 is an unknown value RCX=0x0000000000000fc3 is an unknown value RDX=0xfffffffffffffe0c is an unknown value RSP=0x00007f13904f5880 is pointing into the stack for thread: 0x00007f138c419800 RBP=0x00007f13904f5880 is pointing into the stack for thread: 0x00007f138c419800 RSI=0x00000007b05efd85 is pointing into object: 0x00000007b04c6e60
[error occurred during error reporting (printing register info), id 0xb]
Stack: [0x00007f13903f7000,0x00007f13904f8000], sp=0x00007f13904f5880, free space=1018k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) v ~StubRoutines::jbyte_disjoint_arraycopy J 7751 C2 com.google.protobuf.CodedOutputStream$OutputStreamEncoder.write(Ljava/nio/ByteBuffer;)V (193 bytes) @ 0x00007f14d66e8236 [0x00007f14d66e8140+0xf6] J 8254 C2 org.tensorflow.framework.TensorProto.writeTo(Lcom/google/protobuf/CodedOutputStream;)V (735 bytes) @ 0x00007f14d5db1df8 [0x00007f14d5db19c0+0x438] J 7976 C2 com.google.protobuf.FieldSet.writeElementNoTag(Lcom/google/protobuf/CodedOutputStream;Lcom/google/protobuf/WireFormat$FieldType;Ljava/lang/Object;)V (396 bytes) @ 0x00007f14d6142808 [0x00007f14d61422a0+0x568] J 9461 C2 com.google.protobuf.MapEntry.writeTo(Lcom/google/protobuf/CodedOutputStream;)V (17 bytes) @ 0x00007f14d537b6f0 [0x00007f14d537b4a0+0x250] J 4220 C2 com.google.protobuf.CodedOutputStream$OutputStreamEncoder.writeMessage(ILcom/google/protobuf/MessageLite;)V (12 bytes) @ 0x00007f14d5b84068 [0x00007f14d5b83e80+0x1e8] J 6552 C1 tensorflow.serving.Predict$PredictRequest.writeTo(Lcom/google/protobuf/CodedOutputStream;)V (129 bytes) @ 0x00007f14d63e045c [0x00007f14d63df020+0x143c] J 12829 C2 com.google.protobuf.AbstractMessageLite.writeTo(Ljava/io/OutputStream;)V (24 bytes) @ 0x00007f14d6c76dc0 [0x00007f14d6c76b20+0x2a0] J 12834 C2 io.grpc.internal.AbstractStream.writeMessage(Ljava/io/InputStream;)V (44 bytes) @ 0x00007f14d6c89388 [0x00007f14d6c88c40+0x748] J 12854 C2 io.grpc.stub.ClientCalls.asyncUnaryRequestCall(Lio/grpc/ClientCall;Ljava/lang/Object;Lio/grpc/stub/ClientCalls$StartableListener;)V (32 bytes) @ 0x00007f14d6cb9da4 [0x00007f14d6cb9a60+0x344] J 11820 C2 io.grpc.stub.ClientCalls.futureUnaryCall(Lio/grpc/ClientCall;Ljava/lang/Object;)Lcom/google/common/util/concurrent/ListenableFuture; (24 bytes) @ 0x00007f14d62412c0 [0x00007f14d6241100+0x1c0] J 9644 C2 io.grpc.stub.ClientCalls.blockingUnaryCall(Lio/grpc/Channel;Lio/grpc/MethodDescriptor;Lio/grpc/CallOptions;Ljava/lang/Object;)Ljava/lang/Object; (134 bytes) @ 0x00007f14d5c047b4 [0x00007f14d5c040a0+0x714] J 9656 C2 com.em.api.model.impl.FaceModelServiceImpl.getDataFromLandMarkModel_aroundBody3$advice(Lcom/em/api/model/impl/FaceModelServiceImpl;Ltensorflow/serving/Predict$PredictRequest;Lorg/aspectj/lang/JoinPoint;Lcom/em/api/common/aspect/CatAspect;Lorg/aspectj/lang/ProceedingJoinPoint;)Ljava/lang/Object; (296 bytes) @ 0x00007f14d58bfb04 [0x00007f14d58bf420+0x6e4] J 12876 C1 com.em.api.algo.impl.FaceProcessorSerivceImpl.getLandmarkResultNew(Lcom/em/api/face/MotionRequest;Lorg/bytedeco/opencv/opencv_core/IplImage;IILjava/util/Map;)Ljava/util/Map; (1366 bytes) @ 0x00007f14d6ccf51c [0x00007f14d6cca320+0x51fc] j com.em.api.algo.impl.FaceProcessorSerivceImpl.detectShake(Lcom/em/api/face/MotionRequest;)Lcom/em/api/face/MotionResponse;+544 J 12927 C1 com.em.api.algo.impl.FaceProcessorSerivceImpl.detectMotion_aroundBody2(Lcom/em/api/algo/impl/FaceProcessorSerivceImpl;Lcom/em/api/face/MotionRequest;Lorg/aspectj/lang/JoinPoint;)Lcom/em/api/face/MotionResponse; (364 bytes) @ 0x00007f14d6c06f44 [0x00007f14d6c03320+0x3c24] J 12941 C1 com.em.api.algo.impl.FaceProcessorSerivceImpl.detectMotion_aroundBody3$advice(Lcom/em/api/algo/impl/FaceProcessorSerivceImpl;Lcom/em/api/face/MotionRequest;Lorg/aspectj/lang/JoinPoint;Lcom/em/api/common/aspect/CatAspect;Lorg/aspectj/lang/ProceedingJoinPoint;)Ljava/lang/Object; (302 bytes) @ 0x00007f14d6bfd1dc [0x00007f14d6bfb140+0x209c] J 12888 C1 com.em.api.face.FaceServer$FaceInfoImpl$2.onNext(Lcom/em/api/face/MotionRequest;)V (262 bytes) @ 0x00007f14d6cf82ac [0x00007f14d6cf3860+0x4a4c] J 12887 C1 com.em.api.face.FaceServer$FaceInfoImpl$2.onNext(Ljava/lang/Object;)V (9 bytes) @ 0x00007f14d625962c [0x00007f14d62594c0+0x16c] J 12886 C1 io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(Ljava/lang/Object;)V (29 bytes) @ 0x00007f14d68facb4 [0x00007f14d68faba0+0x114] J 12814 C2 io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(Lio/grpc/internal/StreamListener$MessageProducer;)V (48 bytes) @ 0x00007f14d6c7b368 [0x00007f14d6c7ad00+0x668] J 9527 C2 io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext()V (77 bytes) @ 0x00007f14d62834fc [0x00007f14d62834a0+0x5c] J 6881 C2 io.grpc.internal.ContextRunnable.run()V (35 bytes) @ 0x00007f14d5999568 [0x00007f14d5999360+0x208] J 11054 C2 io.grpc.internal.SerializingExecutor.run()V (99 bytes) @ 0x00007f14d6867678 [0x00007f14d68673e0+0x298] J 9738 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00007f14d57536fc [0x00007f14d5753560+0x19c] J 11295 C1 java.util.concurrent.ThreadPoolExecutor$Worker.run()V (9 bytes) @ 0x00007f14d68ef484 [0x00007f14d68ef380+0x104] J 10257 C1 java.lang.Thread.run()V (17 bytes) @ 0x00007f14d61f5544 [0x00007f14d61f5400+0x144] v ~StubRoutines::call_stub V [libjvm.so+0x690206] JavaCalls::call_helper(JavaValue, methodHandle, JavaCallArguments, Thread)+0x1056 V [libjvm.so+0x690711] JavaCalls::call_virtual(JavaValue, KlassHandle, Symbol, Symbol, JavaCallArguments, Thread)+0x321 V [libjvm.so+0x690bb7] JavaCalls::call_virtual(JavaValue, Handle, KlassHandle, Symbol, Symbol, Thread)+0x47 V [libjvm.so+0x7278f0] thread_entry(JavaThread, Thread)+0xa0 V [libjvm.so+0xa742b3] JavaThread::thread_main_inner()+0x103 V [libjvm.so+0xa743fc] JavaThread::run()+0x11c V [libjvm.so+0x924ea8] java_start(Thread)+0x108 C [libpthread.so.0+0x7ea5] start_thread+0xc5 2.code:
log.info("{}:no fatal error !",motionRequest.getSid());
PredictRequest requestLandmark = PredictRequest.newBuilder()
.setModelSpec(MODELSPEC_LANDMARK)
.putInputs("images", lanMarkProto.build())
.build();
PredictResponse responseLandMark;
try {
responseLandMark = faceModelService.getDataFromLandMarkModel(requestLandmark);
requestLandmark = null;
} catch (StatusRuntimeException e) {
if (e.getStatus().getCode() == Status.Code.DEADLINE_EXCEEDED) {
log.warn("detectMotion landmark model timeout", e);
return landmarkResultInfo;
} else {
throw e;
}
}
log.info("receive landmark serving response: {}", motionRequest.getSid());I see IplImage in the log, but I don't see where it's created, how it's referenced, and how you're making sure it's not getting deallocated. You'll need to provide the actual code that you're using if you wish to receive some help.
whalefa1I
commented
3 years ago I see
IplImagein the log, but I don't see where it's created, how it's referenced, and how you're making sure it's not getting deallocated. You'll need to provide the actual code that you're using if you wish to receive some help.
`private List
// int angle = motionRequest.getAngle(); // switch (angle) { // case 0: // finalHeight = motionRequest.getHeight(); // finalWidth = motionRequest.getWidth(); // break; // case 90: { // finalHeight = motionRequest.getWidth(); // finalWidth = motionRequest.getHeight(); // oriImagCurrent = rotateImage(oriImagCurrent, finalWidth, finalHeight, 0); // break; // } // case 180: { // finalHeight = motionRequest.getHeight(); // finalWidth = motionRequest.getWidth(); // cvFlip(oriImagCurrent, oriImagCurrent, -1); // break; // } // case 270: { // finalHeight = motionRequest.getWidth(); // finalWidth = motionRequest.getHeight(); // oriImagCurrent = rotateImage(oriImagCurrent, finalWidth, finalHeight, 1); // break; // } // default: { // break; // } // }
// get some base info for the current IplImage
float oriRatio = ((float) FACEDET_SIZE_NEW / Math.max(oriImagCurrent.width(), oriImagCurrent.height()));
int faceDetWidth = (int) (oriImagCurrent.width() * oriRatio);
int faceDetHeight = (int) (oriImagCurrent.height() * oriRatio);
// preprocess the image
// 1.resize the current IplImage
IplImage resizedImg = IplImage.create(faceDetWidth, faceDetHeight, oriImagCurrent.depth(), oriImagCurrent.nChannels());
cvResize(oriImagCurrent, resizedImg);
// 2.padding the current IplImage
IplImage paddingImg = IplImage.create(FACEDET_SIZE_NEW, FACEDET_SIZE_NEW, resizedImg.depth(), resizedImg.nChannels());
int dx = (FACEDET_SIZE_NEW - faceDetWidth) / 2;
int dy = (FACEDET_SIZE_NEW - faceDetHeight) / 2;
cvCopyMakeBorder(resizedImg, paddingImg, cvPoint(dx, dy), IPL_BORDER_CONSTANT);
TensorProto.Builder faceDetProto = TensorProto.newBuilder().setDtype(DT_FLOAT).setTensorShape(DETECT_PROTO_NEW);
// 3.normalize the image to [-1, 1]
try ( PointerScope scope = new PointerScope()){
Mat dst = new Mat();
Mat src = new Mat(paddingImg);
normalize(src, dst, -1, 1, NORM_MINMAX, CV_32FC1, null);
// build detect protobuilder
ByteBuffer buffer = dst.getByteBuffer();
ByteString content = UnsafeByteOperations.unsafeWrap(buffer);
faceDetProto.setTensorContent(content);
}
log.info("{}:faceDetProto no fatal error !",motionRequest.getSid());// String outputPath = "/data/data8/sunzheng/project/em_project/blazeFaceFolder/faceDetbp"+motionRequest.getSid()+".bin"; // OutputStream outputStream = new ObjectOutputStream(new FileOutputStream(outputPath)); // faceDetProto.build().writeTo(outputStream); // outputStream.close(); // TensorProto tensorProto = TensorProto.parseFrom(new FileInputStream("/data/data8/sunzheng/project/em_project/blazeFaceFolder/faceDet_bp_1.bin")); // build & send detect request PredictRequest requestFaceDet = PredictRequest.newBuilder() .setModelSpec(MODELSPEC_DETECT_NEW) .putInputs("input_image", faceDetProto.build()) .build(); log.info("{}:requestFaceDet no fatal error !",motionRequest.getSid()); PredictResponse responseFaceDet; try { responseFaceDet = faceModelService.getDataFromFaceModel(requestFaceDet); } catch (StatusRuntimeException e) { if (e.getStatus().getCode() == Status.Code.DEADLINE_EXCEEDED) { log.warn("detectFace face model timeout", e); return new ArrayList<>(); } else { throw e; } } log.info("receive detect serving response: {}", motionRequest.getSid()); String concat = "concatenate_4";
// get tensorflow serving response
List<Float> servingResultList = responseFaceDet.getOutputsMap().get(concat).getFloatValList();
Float[] servingResultArray = servingResultList.toArray(new Float[15232]);
FloatDataBuffer floatDataBuffer = NioDataBufferFactory.create(FloatBuffer.allocate(15232));
floatDataBuffer.write(servingResultArray);
FloatNdArray servingResultNdArray = NdArrays.wrap(Shape.of(896, 17), floatDataBuffer);
float[][] servingBoxResultArray = StdArrays.array2dCopyOf(servingResultNdArray);
// prepare data for non max suppression
FloatNdArray scores1dArray = servingResultNdArray.slice(all(), at(16));
FloatNdArray a = servingResultNdArray.slice(all(), seq(1, 0));
FloatNdArray b = servingResultNdArray.slice(all(), seq(3, 2));
//generate constant max filled 2
FloatDataBuffer constantArrayBuffer = NioDataBufferFactory.create(FloatBuffer.allocate(1796));
Float[] fConst2 = new Float[1796];
Arrays.fill(fConst2, 2.f);
constantArrayBuffer.write(fConst2);
FloatNdArray constantArray = NdArrays.wrap(Shape.of(896, 2), constantArrayBuffer);
//generate constant max filled 0
FloatDataBuffer constantArrayBuffer0 = NioDataBufferFactory.create(FloatBuffer.allocate(1796));
Float[] fConst0 = new Float[1796];
Arrays.fill(fConst0, 0.f);
constantArrayBuffer0.write(fConst0);
FloatNdArray constantArray0 = NdArrays.wrap(Shape.of(896, 2), constantArrayBuffer0);
//generate constant max filled max value
FloatDataBuffer constantArrayBufferNull = NioDataBufferFactory.create(FloatBuffer.allocate(1796));
Float[] fConstNull = new Float[1796];
Arrays.fill(fConstNull, 1230f);
constantArrayBufferNull.write(fConstNull);
FloatNdArray constantArrayNull = NdArrays.wrap(Shape.of(896, 2), constantArrayBufferNull);
try (EagerSession session = EagerSession.create()) {
Ops tf = Ops.create(session);
Div<TFloat32> div = tf.math.div(tf.constant(b), tf.constant(constantArray));
Neg<TFloat32> neg = tf.math.neg(div);
Add<TFloat32> add = tf.math.add(neg, tf.constant(a));
Add<TFloat32> add2 = tf.math.add(div, tf.constant(a));
ClipByValue<TFloat32> clipByValue = tf.clipByValue(add, tf.constant(constantArray0), tf.constant(constantArrayNull));
Concat<TFloat32> finalBoxes = tf.concat(Arrays.asList(clipByValue, add2), tf.constant(1));
NonMaxSuppression nonMaxSuppression = tf.image.nonMaxSuppression(finalBoxes, tf.constant(scores1dArray), tf.constant(5), tf.constant(0.5f), tf.constant(FACE_THRESHOLD), tf.constant(0.01f));
Output<TInt32> indiceOutput = nonMaxSuppression.selectedIndices();
int[] indexNMS = StdArrays.array1dCopyOf(indiceOutput.data());
Output<TFloat32> scoreOutput = nonMaxSuppression.selectedScores();
float[] scoreNMS = StdArrays.array1dCopyOf(scoreOutput.data());
List scoreList = new ArrayList();
float[] boxArray;
if (scoreNMS.length > 0) {
boxArray = servingBoxResultArray[indexNMS[0]];
scoreList.add(scoreNMS[0]);
log.info("scoreList index 0 = {}", scoreNMS[0]);
} else {
boxArray = new float[4];
}
float offsetXOri = (float) dx / (float) FACEDET_SIZE_NEW;
float offsetYOri = (float) dy / (float) FACEDET_SIZE_NEW;
float scaleXOri = (float) FACEDET_SIZE_NEW / (float) faceDetWidth;
float scaleYOri = (float) FACEDET_SIZE_NEW / (float) faceDetHeight;
// get Top1 highest score Box
int centX = (int) ((boxArray[0] - offsetXOri) * scaleYOri * finalHeight);
int centY = (int) ((boxArray[1] - offsetYOri) * scaleXOri * finalWidth);
int responseWidth = (int) (boxArray[2] * scaleYOri * finalHeight);
int responseHeight = (int) (boxArray[3] * scaleXOri * finalWidth);
CvRect cvRect2 = new CvRect();
cvRect2.x((centX - responseWidth / 2));
cvRect2.y((centY - responseHeight / 2));
cvRect2.width(responseWidth);
cvRect2.height(responseHeight);
Position topLeft = Position.newBuilder().setX((centX - responseWidth / 2)).setY((centY - responseHeight / 2)).build();
Position bottomRight = Position.newBuilder().setX((centX + responseWidth / 2)).setY((centY + responseHeight / 2)).build();
List<Position> facePosition = new ArrayList<>();
facePosition.add(topLeft);
facePosition.add(bottomRight);
return facePosition;
}
}`
saudet
commented
3 years ago try ( PointerScope scope = new PointerScope()){ Mat dst = new Mat(); Mat src = new Mat(paddingImg); normalize(src, dst, -1, 1, NORM_MINMAX, CV_32FC1, null); // build detect protobuilder ByteBuffer buffer = dst.getByteBuffer(); ByteString content = UnsafeByteOperations.unsafeWrap(buffer); faceDetProto.setTensorContent(content); }
You're trying to access the data from the Mat after deallocating it. That won't work.
whalefa1I
commented
3 years ago try ( PointerScope scope = new PointerScope()){ Mat dst = new Mat(); Mat src = new Mat(paddingImg); normalize(src, dst, -1, 1, NORM_MINMAX, CV_32FC1, null); // build detect protobuilder ByteBuffer buffer = dst.getByteBuffer(); ByteString content = UnsafeByteOperations.unsafeWrap(buffer); faceDetProto.setTensorContent(content); }You're trying to access the data from the Mat after deallocating it. That won't work.
Which code deallocate the dst Mat? And what it should be?thanks!
saudet
commented
3 years ago Which code deallocate the dst Mat? And what it should be?thanks!
That's what PointerScope does: http://bytedeco.org/news/2018/07/17/bytedeco-as-distribution/
You need to either extend the life of that Mat or copy the data somewhere else.
whalefa1I
commented
3 years ago Which code deallocate the dst Mat? And what it should be?thanks!
That's what PointerScope does: http://bytedeco.org/news/2018/07/17/bytedeco-as-distribution/
You need to either extend the life of that Mat or copy the data somewhere else.
Do you mean something like the following code?Ending the try part after sending the request to TF serving? ` PredictResponse responseFaceDet;
try ( PointerScope scope = new PointerScope()){
Mat dst = new Mat();
Mat src = new Mat(paddingImg);
normalize(src, dst, -1, 1, NORM_MINMAX, CV_32FC1, null);
ByteBuffer buffer = dst.getByteBuffer();
ByteString content = UnsafeByteOperations.unsafeWrap(buffer);
faceDetProto.setTensorContent(content);
PredictRequest requestFaceDet = PredictRequest.newBuilder()
.setModelSpec(MODELSPEC_DETECT_NEW)
.putInputs("input_image", faceDetProto.build())
.build();
log.info("{}:requestFaceDet no fatal error !",motionRequest.getSid());
responseFaceDet = faceModelService.getDataFromFaceModel(requestFaceDet);
}
log.info("{}:faceDetProto no fatal error !",motionRequest.getSid());`
karllessard
commented
3 years ago @whalefa1I , it looks like you are still using version 0.2.0, some memory leak issues have been fixed in 0.3.1, can you give try the original code with this version? Here the migrating guide if you need to adapt some part of your code.
System information
You can collect some of this information using our environment capture script You can also obtain the TensorFlow version with python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
Describe the current behavior some fatal error happend occasionaly when I test the code in multiple threads:
Describe the expected behavior How to avoid this situation? This fatal error killed the entire Java service Code to reproduce the issue Provide a reproducible test case that is the bare minimum necessary to generate the problem.
1.code: `// get tensorflow serving response List servingResultList = responseFaceDet.getOutputsMap().get(concat).getFloatValList();
Float[] servingResultArray = servingResultList.toArray(new Float[15232]);
FloatDataBuffer floatDataBuffer = NioDataBufferFactory.create(FloatBuffer.allocate(15232));
floatDataBuffer.write(servingResultArray);
FloatNdArray servingResultNdArray = NdArrays.wrap(Shape.of(896, 17), floatDataBuffer);
float[][] servingBoxResultArray = StdArrays.array2dCopyOf(servingResultNdArray);
Other info / logs Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
2.hs_err_pid24993.log A fatal error has been detected by the Java Runtime Environment:
SIGSEGV (0xb) at pc=0x00007f0d1d772691, pid=13893, tid=0x00007f0c76afa700
JRE version: Java(TM) SE Runtime Environment (8.0_111-b14) (build 1.8.0_111-b14) Java VM: Java HotSpot(TM) 64-Bit Server VM (25.111-b14 mixed mode linux-amd64 compressed oops) Problematic frame: C [libc.so.6+0x16f691] __strlen_sse2_pminub+0x11
Core dump written. Default location: /opt/app/logs/bd-pica-api/core or core.13893
If you would like to submit a bug report, please visit: http://bugreport.java.com/bugreport/crash.jsp The crash happened outside the Java Virtual Machine in native code. See problematic frame for where to report the bug. --------------- T H R E A D ---------------
Current thread (0x00007f0c88389800): JavaThread "grpc-executor-48" daemon [_thread_in_native, id=22083, stack(0x00007f0c769fa000,0x00007f0c76afb000)]
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00007f0b502e32e0
Registers: RAX=0x0000000000000000, RBX=0x00007f0c76af9198, RCX=0x0000000000000020, RDX=0x00007f0b57ff8900 RSP=0x00007f0c76af9168, RBP=0x00007f0c76af91e0, RSI=0x0000000000000000, RDI=0x00007f0b502e32e0 R8 =0x00007f0c8c63c290, R9 =0x0000000000000002, R10=0x0000000000000000, R11=0x00007f0d1d5d6f88 R12=0x00007f0c76af91f8, R13=0x00007f0b502e32e0, R14=0x00007f0c8c63c110, R15=0x00007f0bd3c49510 RIP=0x00007f0d1d772691, EFLAGS=0x0000000000010287, CSGSFS=0x0000000000000033, ERR=0x0000000000000004 TRAPNO=0x000000000000000e
Top of Stack: (sp=0x00007f0c76af9168) 0x00007f0c76af9168: 00007f0c4ecc0153 0000000000000002 0x00007f0c76af9178: 00007f0c8c63c290 00007f0b57ff8900 0x00007f0c76af9188: 00007f0c8ffbb540 00000007c02be468 0x00007f0c76af9198: 0000000000000000 0000000000000000 0x00007f0c76af91a8: 0000000000000000 00000007bf985ad8 0x00007f0c76af91b8: 00000007bf985b00 00007f0c8c683c00 0x00007f0c76af91c8: 00000007bf985b5c 00007f0c76af9250 0x00007f0c76af91d8: 00007f0c88389800 00007f0c76af9210 0x00007f0c76af91e8: 00007f0c4ec38138 00007f0c76af9260 0x00007f0c76af91f8: 0000000000000000 00000007bf985b00 0x00007f0c76af9208: 0000000000000000 00007f0c76af9260 0x00007f0c76af9218: 00007f0d09122d4d 00000007bf985720 0x00007f0c76af9228: 00000007bf985c10 00000007bf985b68 0x00007f0c76af9238: 00000007bf985cc0 0000001a48257a79 0x00007f0c76af9248: 0000000000000003 00000006c0baf2e8 0x00007f0c76af9258: 00007f0c5593e4b7 00000007bf985cf0 0x00007f0c76af9268: 00007f0d097f3964 00000007bf985cc0 0x00007f0c76af9278: 00000007bf9789e8 00000007bf985ad8 0x00007f0c76af9288: 00000007bf985720 00000007bf985b68 0x00007f0c76af9298: 00000007bf985c10 00000007bf985cc0 0x00007f0c76af92a8: 00007f0c4ece5f7f 00000007bf9859f8 0x00007f0c76af92b8: 00000007bf985990 00000006c0baf2e8 0x00007f0c76af92c8: 00000007bf9859f4 00000000f7f30b45 0x00007f0c76af92d8: 00007f0d08ec31a0 00000007bf985a28 0x00007f0c76af92e8: 0000000000000000 00000007bf985ac0 0x00007f0c76af92f8: 00007f0d09b2d4ac 00000000f7f30b45 0x00007f0c76af9308: 00007f0d091e8278 00000007bf985968 0x00007f0c76af9318: 00000007bf985570 00000007bf985a58 0x00007f0c76af9328: 00000007bf985980 0000001a48257a79 0x00007f0c76af9338: 0000000000000003 00000006c0baf2e8 0x00007f0c76af9348: 00000007bf96dd28 00000007bf985ac0 0x00007f0c76af9358: 00007f0d0933aa6c 00000007bf985ac0
Instructions: (pc=0x00007f0d1d772691) 0x00007f0d1d772671: c0 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 48 0x00007f0d1d772681: 31 c0 89 f9 83 e1 3f 66 0f ef c0 83 f9 30 77 1d 0x00007f0d1d772691: f3 0f 6f 0f 66 0f 74 c1 66 0f d7 d0 85 d2 0f 85 0x00007f0d1d7726a1: 4e 02 00 00 48 89 f8 48 83 e0 f0 eb 24 48 89 f8
Register to memory mapping:
RAX=0x0000000000000000 is an unknown value RBX=0x00007f0c76af9198 is pointing into the stack for thread: 0x00007f0c88389800 RCX=0x0000000000000020 is an unknown value RDX=0x00007f0b57ff8900 is an unknown value RSP=0x00007f0c76af9168 is pointing into the stack for thread: 0x00007f0c88389800 RBP=0x00007f0c76af91e0 is pointing into the stack for thread: 0x00007f0c88389800 RSI=0x0000000000000000 is an unknown value RDI=0x00007f0b502e32e0 is an unknown value R8 =0x00007f0c8c63c290 is an unknown value R9 =0x0000000000000002 is an unknown value R10=0x0000000000000000 is an unknown value R11=0x00007f0d1d5d6f88: <offset 0xfc3f88> in /opt/programs/jdk1.8.0_111/jre/lib/amd64/server/libjvm.so at 0x00007f0d1c613000 R12=0x00007f0c76af91f8 is pointing into the stack for thread: 0x00007f0c88389800 R13=0x00007f0b502e32e0 is an unknown value R14=0x00007f0c8c63c110 is an unknown value R15=0x00007f0bd3c49510 is an unknown value
Stack: [0x00007f0c769fa000,0x00007f0c76afb000], sp=0x00007f0c76af9168, free space=1020k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) C [libc.so.6+0x16f691] __strlen_sse2_pminub+0x11 C [libtensorflow_cc.so.2+0x2385138] TFE_Execute+0x28 J 3768 org.tensorflow.internal.c_api.global.tensorflow.TFE_Execute(Lorg/tensorflow/internal/c_api/TFE_Op;Lorg/bytedeco/javacpp/PointerPointer;Lorg/bytedeco/javacpp/IntPointer;Lorg/tensorflow/internal/c_api/TF_Status;)V (0 bytes) @ 0x00007f0d09122d4d [0x00007f0d09122c40+0x10d] J 5826 C2 org.tensorflow.EagerOperationBuilder.execute(Lorg/tensorflow/internal/c_api/TFE_Op;Lorg/tensorflow/EagerSession;)[Lorg/tensorflow/internal/c_api/TFE_TensorHandle; (210 bytes) @ 0x00007f0d097f3964 [0x00007f0d097f36e0+0x284] J 6675 C2 org.tensorflow.EagerOperationBuilder.build()Lorg/tensorflow/EagerOperation; (57 bytes) @ 0x00007f0d09b2d4ac [0x00007f0d09b2d460+0x4c] J 8643 C2 org.tensorflow.op.MathOps.div(Lorg/tensorflow/Operand;Lorg/tensorflow/Operand;)Lorg/tensorflow/op/math/Div; (10 bytes) @ 0x00007f0d0933aa6c [0x00007f0d09339e80+0xbec] J 7481 C2 com.em.api.algo.impl.FaceProcessorSerivceImpl.getModelInfo(Lcom/em/api/face/ModelCheckRequest;)Lcom/em/api/face/ModelCheckResponse; (2376 bytes) @ 0x00007f0d09f98068 [0x00007f0d09f94140+0x3f28] J 8623 C2 com.em.api.face.ModelCheckServiceGrpc$MethodHandlers.invoke(Ljava/lang/Object;Lio/grpc/stub/StreamObserver;)V (48 bytes) @ 0x00007f0d0987d630 [0x00007f0d0987d000+0x630] J 8652 C2 io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext()V (73 bytes) @ 0x00007f0d08afc0f4 [0x00007f0d08afc020+0xd4] J 7467 C2 io.grpc.internal.ContextRunnable.run()V (35 bytes) @ 0x00007f0d09e77828 [0x00007f0d09e77620+0x208] J 7931 C2 io.grpc.internal.SerializingExecutor.run()V (99 bytes) @ 0x00007f0d0904b930 [0x00007f0d0904b6a0+0x290] J 9207 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00007f0d096d3994 [0x00007f0d096d36e0+0x2b4] j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 j java.lang.Thread.run()V+11 v ~StubRoutines::call_stub V [libjvm.so+0x690206] JavaCalls::call_helper(JavaValue, methodHandle, JavaCallArguments, Thread)+0x1056 V [libjvm.so+0x690711] JavaCalls::call_virtual(JavaValue, KlassHandle, Symbol, Symbol, JavaCallArguments, Thread)+0x321 V [libjvm.so+0x690bb7] JavaCalls::call_virtual(JavaValue, Handle, KlassHandle, Symbol, Symbol, Thread)+0x47 V [libjvm.so+0x7278f0] thread_entry(JavaThread, Thread)+0xa0 V [libjvm.so+0xa742b3] JavaThread::thread_main_inner()+0x103 V [libjvm.so+0xa743fc] JavaThread::run()+0x11c V [libjvm.so+0x924ea8] java_start(Thread)+0x108 C [libpthread.so.0+0x7ea5] start_thread+0xc5
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) J 3768 org.tensorflow.internal.c_api.global.tensorflow.TFE_Execute(Lorg/tensorflow/internal/c_api/TFE_Op;Lorg/bytedeco/javacpp/PointerPointer;Lorg/bytedeco/javacpp/IntPointer;Lorg/tensorflow/internal/c_api/TF_Status;)V (0 bytes) @ 0x00007f0d09122ccb [0x00007f0d09122c40+0x8b] J 5826 C2 org.tensorflow.EagerOperationBuilder.execute(Lorg/tensorflow/internal/c_api/TFE_Op;Lorg/tensorflow/EagerSession;)[Lorg/tensorflow/internal/c_api/TFE_TensorHandle; (210 bytes) @ 0x00007f0d097f3964 [0x00007f0d097f36e0+0x284] J 6675 C2 org.tensorflow.EagerOperationBuilder.build()Lorg/tensorflow/EagerOperation; (57 bytes) @ 0x00007f0d09b2d4ac [0x00007f0d09b2d460+0x4c] J 8643 C2 org.tensorflow.op.MathOps.div(Lorg/tensorflow/Operand;Lorg/tensorflow/Operand;)Lorg/tensorflow/op/math/Div; (10 bytes) @ 0x00007f0d0933aa6c [0x00007f0d09339e80+0xbec] J 7481 C2 com.em.api.algo.impl.FaceProcessorSerivceImpl.getModelInfo(Lcom/em/api/face/ModelCheckRequest;)Lcom/em/api/face/ModelCheckResponse; (2376 bytes) @ 0x00007f0d09f98068 [0x00007f0d09f94140+0x3f28] J 8623 C2 com.em.api.face.ModelCheckServiceGrpc$MethodHandlers.invoke(Ljava/lang/Object;Lio/grpc/stub/StreamObserver;)V (48 bytes) @ 0x00007f0d0987d630 [0x00007f0d0987d000+0x630] J 8652 C2 io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext()V (73 bytes) @ 0x00007f0d08afc0f4 [0x00007f0d08afc020+0xd4] J 7467 C2 io.grpc.internal.ContextRunnable.run()V (35 bytes) @ 0x00007f0d09e77828 [0x00007f0d09e77620+0x208] J 7931 C2 io.grpc.internal.SerializingExecutor.run()V (99 bytes) @ 0x00007f0d0904b930 [0x00007f0d0904b6a0+0x290] J 9207 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00007f0d096d3994 [0x00007f0d096d36e0+0x2b4] j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 j java.lang.Thread.run()V+11 v ~StubRoutines::call_stub