390057892
commented
2 years ago Can anyone help me？please
Closed 390057892 closed 2 years ago
390057892
commented
2 years ago Can anyone help me？please
sureshdagooglecom
commented
2 years ago Hi @390057892 , Can you share any reference video and code changes to investigate further on this issue.
390057892
commented
2 years ago Camera@b30f6fb[id=0] UNKNOWN
Camera@962a3ad[id=1] OPENING
-------------------------------------------------------------------
Open count: 1 (Max allowed: 1)Camera@b30f6fb[id=0] UNKNOWN
Camera@962a3ad[id=1] OPEN
-------------------------------------------------------------------
Open count: 1 (Max allowed: 1)2022-06-01 16:53:42.620 5564-5728/com.lite.holistic_tracking D/UseCaseAttachState: All use case: [androidx.camera.core.Preview-45bc1178-c3f5-4e07-9474-fa005ce0fa8e26017363] for camera: 1 2022-06-01 16:53:42.623 5564-5728/com.lite.holistic_tracking D/DeferrableSurface: New surface in use[total_surfaces=1, used_surfaces=1](androidx.camera.core.SurfaceRequest$2@8ffec8e} 2022-06-01 16:53:42.623 5564-5728/com.lite.holistic_tracking D/DeferrableSurface: use count+1, useCount=1 androidx.camera.core.SurfaceRequest$2@8ffec8e 2022-06-01 16:53:42.623 5564-5728/com.lite.holistic_tracking D/CaptureSession: Opening capture session. 2022-06-01 16:53:42.870 5564-5728/com.lite.holistic_tracking W/BpBinder: Slow Binder: BpBinder transact took 243 ms, interface=android.hardware.camera2.ICameraDeviceUser, code=6 oneway=false 2022-06-01 16:53:42.872 5564-5728/com.lite.holistic_tracking D/CaptureSession: Attempting to send capture request onConfigured 2022-06-01 16:53:42.872 5564-5728/com.lite.holistic_tracking D/CaptureSession: Issuing request for session. 2022-06-01 16:53:42.875 5564-5728/com.lite.holistic_tracking D/CaptureSession: CameraCaptureSession.onConfigured() mState=OPENED 2022-06-01 16:53:42.875 5564-5728/com.lite.holistic_tracking D/CaptureSession: CameraCaptureSession.onReady() OPENED 2022-06-01 16:53:42.875 5564-5590/com.lite.holistic_tracking W/Gralloc3: allocator 3.x is not supported 2022-06-01 16:53:43.025 5564-5721/com.lite.holistic_tracking D/ExternalTextureConv: Created output texture: 3 width: 720 height: 1280 2022-06-01 16:53:43.062 5564-5721/com.lite.holistic_tracking I/native: I20220601 16:53:43.062569 5721 graph.cc:478] Start running the graph, waiting for inputs. 2022-06-01 16:53:43.062 5564-5721/com.lite.holistic_tracking I/native: I20220601 16:53:43.062855 5721 gl_context_egl.cc:84] Successfully initialized EGL. Major : 1 Minor: 5 2022-06-01 16:53:43.064 5564-5766/com.lite.holistic_tracking I/native: I20220601 16:53:43.063995 5766 gl_context.cc:335] GL version: 3.2 (OpenGL ES 3.2 V@415.0 (GIT@e541a88, I20154638fb, 1600173890) (Date:09/15/20)) 2022-06-01 16:53:43.066 5564-5721/com.lite.holistic_tracking D/ExternalTextureConv: Created output texture: 4 width: 720 height: 1280 2022-06-01 16:53:43.080 5564-5717/com.lite.holistic_tracking I/native: I20220601 16:53:43.080389 5717 resource_util_android.cc:89] Successfully loaded: pose_detection.tflite 2022-06-01 16:53:43.082 5564-5717/com.lite.holistic_tracking I/tflite: Initialized TensorFlow Lite runtime. 2022-06-01 16:53:43.083 5564-5759/com.lite.holistic_tracking I/native: I20220601 16:53:43.083304 5759 resource_util_android.cc:89] Successfully loaded: face_landmark.tflite 2022-06-01 16:53:43.086 5564-5717/com.lite.holistic_tracking I/tflite: Replacing 291 node(s) with delegate (unknown) node, yielding 1 partitions. 2022-06-01 16:53:43.096 5564-5721/com.lite.holistic_tracking D/ExternalTextureConv: Created output texture: 5 width: 720 height: 1280 2022-06-01 16:53:43.099 5564-5763/com.lite.holistic_tracking I/native: I20220601 16:53:43.099145 5763 resource_util_android.cc:89] Successfully loaded: hand_landmark_full.tflite 2022-06-01 16:53:43.100 5564-5764/com.lite.holistic_tracking I/native: I20220601 16:53:43.100013 5764 jni_util.cc:41] GetEnv: not attached 2022-06-01 16:53:43.102 5564-5761/com.lite.holistic_tracking I/native: I20220601 16:53:43.102221 5761 resource_util_android.cc:89] Successfully loaded: hand_landmark_full.tflite 2022-06-01 16:53:43.103 5564-5760/com.lite.holistic_tracking I/native: I20220601 16:53:43.103941 5760 resource_util_android.cc:89] Successfully loaded: pose_landmark_full.tflite 2022-06-01 16:53:43.127 5564-5758/com.lite.holistic_tracking I/native: I20220601 16:53:43.127578 5758 jni_util.cc:41] GetEnv: not attached 2022-06-01 16:53:43.159 5564-5762/com.lite.holistic_tracking I/native: I20220601 16:53:43.159376 5762 jni_util.cc:41] GetEnv: not attached 2022-06-01 16:53:43.217 5564-5717/com.lite.holistic_tracking I/tflite: Replacing 291 node(s) with delegate (unknown) node, yielding 1 partitions. 2022-06-01 16:53:43.226 5564-5765/com.lite.holistic_tracking I/native: I20220601 16:53:43.226831 5765 jni_util.cc:41] GetEnv: not attached 2022-06-01 16:53:43.295 5564-5763/com.lite.holistic_tracking I/native: I20220601 16:53:43.295612 5763 jni_util.cc:41] GetEnv: not attached 2022-06-01 16:53:43.361 5564-5759/com.lite.holistic_tracking I/native: I20220601 16:53:43.361868 5759 jni_util.cc:41] GetEnv: not attached 2022-06-01 16:53:43.395 5564-5760/com.lite.holistic_tracking I/native: I20220601 16:53:43.395515 5760 jni_util.cc:41] GetEnv: not attached 2022-06-01 16:53:43.901 5564-5761/com.lite.holistic_tracking I/native: I20220601 16:53:43.901232 5761 jni_util.cc:41] GetEnv: not attached 2022-06-01 16:53:46.555 5564-5717/com.lite.holistic_tracking I/tflite: Created TensorFlow Lite delegate for GPU. 2022-06-01 16:53:46.556 5564-5717/com.lite.holistic_tracking I/tflite: Replacing 332 node(s) with delegate (TfLiteGpuDelegate) node, yielding 1 partitions. 2022-06-01 16:53:47.614 5564-5717/com.lite.holistic_tracking I/native: I20220601 16:53:47.614264 5717 resource_util_android.cc:89] Successfully loaded: face_detection_short_range.tflite 2022-06-01 16:53:47.615 5564-5717/com.lite.holistic_tracking I/tflite: Replacing 164 node(s) with delegate (TfLiteGpuDelegate) node, yielding 1 partitions. 2022-06-01 16:53:48.116 5564-5717/com.lite.holistic_tracking I/native: I20220601 16:53:48.116806 5717 resource_util_android.cc:89] Successfully loaded: hand_recrop.tflite 2022-06-01 16:53:48.117 5564-5717/com.lite.holistic_tracking I/tflite: Replacing 63 node(s) with delegate (TfLiteGpuDelegate) node, yielding 1 partitions. 2022-06-01 16:53:48.523 5564-5717/com.lite.holistic_tracking I/native: I20220601 16:53:48.523654 5717 resource_util_android.cc:89] Successfully loaded: hand_recrop.tflite 2022-06-01 16:53:48.524 5564-5717/com.lite.holistic_tracking I/tflite: Replacing 63 node(s) with delegate (TfLiteGpuDelegate) node, yielding 1 partitions. 2022-06-01 16:53:48.586 5564-5717/com.lite.holistic_tracking I/tflite: Replacing 210 node(s) with delegate (TfLiteGpuDelegate) node, yielding 1 partitions. 2022-06-01 16:53:49.027 5564-5717/com.lite.holistic_tracking I/tflite: Replacing 165 node(s) with delegate (TfLiteGpuDelegate) node, yielding 1 partitions. 2022-06-01 16:53:49.610 5564-5763/com.lite.holistic_tracking I/native: I20220601 16:53:49.610852 5763 resource_util_android.cc:89] Successfully loaded: handedness.txt 2022-06-01 16:53:49.611 5564-5717/com.lite.holistic_tracking I/tflite: Replacing 165 node(s) with delegate (TfLiteGpuDelegate) node, yielding 1 partitions. 2022-06-01 16:53:49.799 5564-5762/com.lite.holistic_tracking I/native: I20220601 16:53:49.799099 5762 resource_util_android.cc:89] Successfully loaded: handedness.txt 2022-06-01 16:53:49.831 5564-5717/com.lite.holistic_tracking I/native: I20220601 16:53:49.831265 5717 jni_util.cc:41] GetEnv: not attached 2022-06-01 16:53:49.874 5564-5764/com.lite.holistic_tracking D/MEDIAPIPE: pose_landmarks timestamp=1654073629874 2022-06-01 16:53:49.891 5564-5764/com.lite.holistic_tracking W/listic_trackin: Accessing hidden method Lsun/misc/Unsafe;->allocateInstance(Ljava/lang/Class;)Ljava/lang/Object; (greylist, linking, allowed) 2022-06-01 16:53:49.893 5564-5764/com.lite.holistic_tracking W/listic_trackin: Accessing hidden method Llibcore/io/Memory;->pokeByte(JB)V (greylist, reflection, allowed) 2022-06-01 16:53:49.893 5564-5764/com.lite.holistic_tracking W/listic_trackin: Accessing hidden method Llibcore/io/Memory;->peekByte(J)B (greylist, reflection, allowed) 2022-06-01 16:53:49.893 5564-5764/com.lite.holistic_tracking W/listic_trackin: Accessing hidden method Lsun/misc/Unsafe;->getInt(Ljava/lang/Object;J)I (greylist, linking, allowed) 2022-06-01 16:53:49.893 5564-5764/com.lite.holistic_tracking W/listic_trackin: Accessing hidden method Lsun/misc/Unsafe;->getLong(Ljava/lang/Object;J)J (greylist,core-platform-api, linking, allowed) 2022-06-01 16:53:49.893 5564-5764/com.lite.holistic_tracking W/listic_trackin: Accessing hidden method Lsun/misc/Unsafe;->getObject(Ljava/lang/Object;J)Ljava/lang/Object; (greylist, linking, allowed) 2022-06-01 16:53:49.893 5564-5764/com.lite.holistic_tracking W/listic_trackin: Accessing hidden method Lsun/misc/Unsafe;->putInt(Ljava/lang/Object;JI)V (greylist, linking, allowed) 2022-06-01 16:53:49.893 5564-5764/com.lite.holistic_tracking W/listic_trackin: Accessing hidden method Lsun/misc/Unsafe;->putLong(Ljava/lang/Object;JJ)V (greylist, linking, allowed) 2022-06-01 16:53:49.893 5564-5764/com.lite.holistic_tracking W/listic_trackin: Accessing hidden method Lsun/misc/Unsafe;->putObject(Ljava/lang/Object;JLjava/lang/Object;)V (greylist, linking, allowed) 2022-06-01 16:53:49.893 5564-5764/com.lite.holistic_tracking W/listic_trackin: Accessing hidden method Lsun/misc/Unsafe;->getLong(Ljava/lang/Object;J)J (greylist,core-platform-api, reflection, allowed) 2022-06-01 16:53:49.893 5564-5764/com.lite.holistic_tracking W/listic_trackin: Accessing hidden method Lsun/misc/Unsafe;->getInt(Ljava/lang/Object;J)I (greylist, reflection, allowed) 2022-06-01 16:53:49.893 5564-5764/com.lite.holistic_tracking W/listic_trackin: Accessing hidden method Lsun/misc/Unsafe;->putInt(Ljava/lang/Object;JI)V (greylist, reflection, allowed) 2022-06-01 16:53:49.894 5564-5764/com.lite.holistic_tracking W/listic_trackin: Accessing hidden method Lsun/misc/Unsafe;->getLong(Ljava/lang/Object;J)J (greylist,core-platform-api, reflection, allowed) 2022-06-01 16:53:49.894 5564-5764/com.lite.holistic_tracking W/listic_trackin: Accessing hidden method Lsun/misc/Unsafe;->putLong(Ljava/lang/Object;JJ)V (greylist, reflection, allowed) 2022-06-01 16:53:49.894 5564-5764/com.lite.holistic_tracking W/listic_trackin: Accessing hidden method Lsun/misc/Unsafe;->getObject(Ljava/lang/Object;J)Ljava/lang/Object; (greylist, reflection, allowed) 2022-06-01 16:53:49.894 5564-5764/com.lite.holistic_tracking W/listic_trackin: Accessing hidden method Lsun/misc/Unsafe;->putObject(Ljava/lang/Object;JLjava/lang/Object;)V (greylist, reflection, allowed) 2022-06-01 16:53:49.902 5564-5764/com.lite.holistic_tracking W/listic_trackin: Accessing hidden method Lsun/misc/Unsafe;->getObject(Ljava/lang/Object;J)Ljava/lang/Object; (greylist, linking, allowed)
390057892
commented
2 years ago `public class holistic_activity extends AppCompatActivity {
private static final String TAG = "MainActivity";
private static final boolean FLIP_FRAMES_VERTICALLY = true;
private static final int NUM_BUFFERS = 2;
static {
// Load all native libraries needed by the app.
System.loadLibrary("mediapipe_jni");
try {
System.loadLibrary("opencv_java3");
} catch (UnsatisfiedLinkError e) {
// Some example apps (e.g. template matching) require OpenCV 4.
System.loadLibrary("opencv_java4");
}
}
// Sends camera-preview frames into a MediaPipe graph for processing, and displays the processed
// frames onto a {@link Surface}.
protected FrameProcessor processor;
// Handles camera access via the {@link CameraX} Jetpack support library.
protected CameraXPreviewHelper cameraHelper;
// {@link SurfaceTexture} where the camera-preview frames can be accessed.
private SurfaceTexture previewFrameTexture;
// {@link SurfaceView} that displays the camera-preview frames processed by a MediaPipe graph.
private SurfaceView previewDisplayView;
// Creates and manages an {@link EGLContext}.
private EglManager eglManager;
// Converts the GL_TEXTURE_EXTERNAL_OES texture from Android camera into a regular texture to be
// consumed by {@link FrameProcessor} and the underlying MediaPipe graph.
private ExternalTextureConverter converter;
// ApplicationInfo for retrieving metadata defined in the manifest.
private ApplicationInfo applicationInfo;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_holistic_activity);
try {
applicationInfo =
getPackageManager().getApplicationInfo(getPackageName(), PackageManager.GET_META_DATA);
} catch (PackageManager.NameNotFoundException e) {
Log.e(TAG, "Cannot find application info: " + e);
}
previewDisplayView = new SurfaceView(this);
setupPreviewDisplayView();
// Initialize asset manager so that MediaPipe native libraries can access the app assets, e.g.,
// binary graphs.
AndroidAssetUtil.initializeNativeAssetManager(this);
eglManager = new EglManager(null);
processor =
new FrameProcessor(
this,
eglManager.getNativeContext(),
applicationInfo.metaData.getString("binaryGraphName"),
applicationInfo.metaData.getString("inputVideoStreamName"),
applicationInfo.metaData.getString("outputVideoStreamName")
);
processor.getVideoSurfaceOutput()
.setFlipY(
applicationInfo.metaData.getBoolean("flipFramesVertically", FLIP_FRAMES_VERTICALLY));
PermissionHelper.checkAndRequestCameraPermissions(this);
processor.addPacketCallback("face_landmarks", new PacketCallback() {
@Override
public void process(Packet packet) {
try {
byte[] landmarksRaw = PacketGetter.getProtoBytes(packet);
LandmarkProto.NormalizedLandmarkList landmarks = LandmarkProto.NormalizedLandmarkList.parseFrom(landmarksRaw);
if (landmarks == null) {
Log.d(TAG, "[TS:" + packet.getTimestamp() + "] No hand landmarks.");
return;
}
Log.d(TAG, "[face_landmarks:" + packet.getTimestamp() + "point:" + landmarks.getLandmarkList().size());
} catch (InvalidProtocolBufferException e) {
Log.e(TAG, "Couldn't Exception received - " + e);
}
}
});
processor.addPacketCallback("pose_landmarks", new PacketCallback() {
@Override
public void process(Packet packet) {
Log.d("MEDIAPIPE", "pose_landmarks timestamp=" + System.currentTimeMillis());
try {
byte[] landmarksRaw = PacketGetter.getProtoBytes(packet);
LandmarkProto.NormalizedLandmarkList landmarks = LandmarkProto.NormalizedLandmarkList.parseFrom(landmarksRaw);
if (landmarks == null) {
Log.d(TAG, "[TS:" + packet.getTimestamp() + "] No hand landmarks.");
return;
}
Log.d(TAG, "[pose_landmarks:" + packet.getTimestamp()
+ "] #Landmarks for hand: "
+ landmarks.getLandmarkCount()
+ "point:"
+ landmarks.getLandmarkList().size());
} catch (InvalidProtocolBufferException e) {
Log.e(TAG, "Couldn't Exception received - " + e);
}
}
});
}
@Override
protected void onResume() {
super.onResume();
converter =
new ExternalTextureConverter(
eglManager.getContext(),
applicationInfo.metaData.getInt("converterNumBuffers", NUM_BUFFERS));
converter.setFlipY(
applicationInfo.metaData.getBoolean("flipFramesVertically", FLIP_FRAMES_VERTICALLY));
converter.setConsumer(processor);
if (PermissionHelper.cameraPermissionsGranted(this)) {
startCamera();
}
}
@Override
protected void onPause() {
super.onPause();
converter.close();
previewDisplayView.setVisibility(View.GONE);
}
@Override
public void onRequestPermissionsResult(
int requestCode, String[] permissions, int[] grantResults) {
super.onRequestPermissionsResult(requestCode, permissions, grantResults);
PermissionHelper.onRequestPermissionsResult(requestCode, permissions, grantResults);
}
protected void onCameraStarted(SurfaceTexture surfaceTexture) {
previewFrameTexture = surfaceTexture;
// Make the display view visible to start showing the preview. This triggers the
// SurfaceHolder.Callback added to (the holder of) previewDisplayView.
previewDisplayView.setVisibility(View.VISIBLE);
}
protected Size cameraTargetResolution() {
return null; // No preference and let the camera (helper) decide.
}
public void startCamera() {
cameraHelper = new CameraXPreviewHelper();
cameraHelper.setOnCameraStartedListener(
surfaceTexture -> {
onCameraStarted(surfaceTexture);
});
CameraHelper.CameraFacing cameraFacing =
applicationInfo.metaData.getBoolean("cameraFacingFront", false)
? CameraHelper.CameraFacing.FRONT
: CameraHelper.CameraFacing.BACK;
cameraHelper.startCamera(
this, cameraFacing, null, cameraTargetResolution());
}
protected Size computeViewSize(int width, int height) {
return new Size(width, height);
}
protected void onPreviewDisplaySurfaceChanged(
SurfaceHolder holder, int format, int width, int height) {
// (Re-)Compute the ideal size of the camera-preview display (the area that the
// camera-preview frames get rendered onto, potentially with scaling and rotation)
// based on the size of the SurfaceView that contains the display.
Size viewSize = computeViewSize(width, height);
Size displaySize = cameraHelper.computeDisplaySizeFromViewSize(viewSize);
boolean isCameraRotated = cameraHelper.isCameraRotated();
// Connect the converter to the camera-preview frames as its input (via
// previewFrameTexture), and configure the output width and height as the computed
// display size.
converter.setSurfaceTextureAndAttachToGLContext(
previewFrameTexture,
isCameraRotated ? displaySize.getHeight() : displaySize.getWidth(),
isCameraRotated ? displaySize.getWidth() : displaySize.getHeight());
previewFrameTexture.updateTexImage();// setSurfaceTextureAndAttachToGLContext(previewFrameTexture,width,height); }
public void setSurfaceTextureAndAttachToGLContext(
SurfaceTexture texture, int width, int height) {
converter.setSurfaceTexture(texture, width, height);
int[] textures = new int[1];
GLES20.glGenTextures(1, textures, 0);
texture.attachToGLContext(textures[0]);
// Update texture image
texture.updateTexImage();
}
private void setupPreviewDisplayView() {
previewDisplayView.setVisibility(View.GONE);
ViewGroup viewGroup = findViewById(R.id.preview_display_layout);
viewGroup.addView(previewDisplayView);
previewDisplayView
.getHolder()
.addCallback(
new SurfaceHolder.Callback() {
@Override
public void surfaceCreated(SurfaceHolder holder) {
processor.getVideoSurfaceOutput().setSurface(holder.getSurface());
}
@Override
public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) {
onPreviewDisplaySurfaceChanged(holder, format, width, height);
}
@Override
public void surfaceDestroyed(SurfaceHolder holder) {
processor.getVideoSurfaceOutput().setSurface(null);
}
});
}} `
390057892
commented
2 years ago 你好@390057892， 您能否分享任何参考视频和代码更改以进一步调查此问题。
hello,The above is my code and log。Through the log, my only found that it takes a long time for hollistic to load TF files。
jiuqiant
commented
2 years ago Holistic graph has 5 models to load (pose detection, pose landmark, hand landmarks for both hands, and face landmark) and 5 tflite interpreters to start correspondingly. It will take much longer initialization time than the Hands graph.
390057892
commented
2 years ago Holistic graph 有 5 个模型要加载（姿势检测、姿势界标、双手的手界标和面部界标）和 5 个 tflite 解释器相应地启动。这将花费比手图更长的初始化时间。
I can understand that it takes longer, so 5-10S is reasonable, right?
sureshdagooglecom
commented
2 years ago Hi @390057892 , Models do take significant time to load, up to several seconds. Holistic has 6 models included, this will multiply the time for loading models. Therefore, 5 to 10 seconds may be expected on some devices.
google-ml-butler[bot]
commented
2 years ago This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.
google-ml-butler[bot]
commented
2 years ago Closing as stale. Please reopen if you'd like to work on this further.
I have built hand and holistic models through WSL and run them on Android phones. The hand model starts quickly, but holistic will be blank for 5-10S
Through the log, it is found that it takes a long time for hollistic to load TF files. Is this normal？