search

getty708 / k4a-frame-extractor

Extract color, depth, and IR frames from kinect recording (.mkv).
1 stars 0 forks source link

[Body Tracking API] デバッグ & 抽出 #9

Open getty708 opened 2 years ago

getty708 commented 2 years ago

Todo

getty708 commented 2 years ago

Todo.1: 骨格検出ができないデータの確認

getty708 commented 2 years ago

Todo.2: タイムスタンプの補正

こちらの方が工数が読めるので、こちらを先にトライする。 ==> 既に修正済み (昔の自分偉い！)

https://github.com/getty708/k4a-frame-extractor/blob/1d912f3fbb0955a1fb0d9cd68373e5043b88584a/body_tracking/k4a_body_tracking_helpers.cpp#L121

getty708 commented 2 years ago

Todo.1: (続き)

k4abt_frame_t の中身？

本当に何も格納されていない? 検出数が0の場合だけでなく、関数が失敗した場合も0を返す使用になっている。

Returns the number of detected bodies. 0 if the function fails.

https://microsoft.github.io/Azure-Kinect-Body-Tracking/release/0.9.x/group__btfunctions_ga29ab088b1a0d1a246bdb5542e21aa3c3.html#ga29ab088b1a0d1a246bdb5542e21aa3c3

これどこでやってる？

# https://docs.microsoft.com/ja-jp/azure/kinect-dk/get-body-tracking-results#capture-depth-and-ir-images
k4a_device_get_capture(device, &capture, TIMEOUT_IN_MS)

サンプル: https://github.com/microsoft/Azure-Kinect-Samples/tree/master/body-tracking-samples/offline_processor

流れはほぼ同じに見える。

getty708 commented 2 years ago

Sample: OfflineProcessor

これを移築してみる。

https://github.com/microsoft/Azure-Kinect-Samples/tree/master/body-tracking-samples/offline_processor

jsonライブラリのインストール

git clone https://github.com/nlohmann/json
cd json
git checkout v3.11.2
mkdir -p build && cd build
cmake ..
make
sudo make install
cd ../..

リンクするパッケージ名を nlohmann_json::nlohmann_json にするとうまく行った。

結果は同じ。 U0102-S0200など、抽出できないファイルは全く抽出できない。

getty708 commented 2 years ago

方針!!

抽出できなかった骨格は、別のモデルでトライする。

getty708 commented 2 years ago

実行ログ

U0103

# === U0103 ===
export K4ABT_USER=U0103

SESSIONS=(
    "S0100 2021-11-12_12:12:53.400 1000"
    "S0200 2021-11-12_13:59:04.308 216000"
    "S0300 2021-11-12_15:04:48.999 1000"
    "S0400 2021-11-12_16:00:52.799 1000"
    "S0500 2021-11-12_17:15:09.334 1000"
)

for ((i=0; ${#SESSIONS[*]}>$i; i++))
do
    params=(${SESSIONS[$i]})

    export K4ABT_SESSION=${params[0]}
    export K4ABT_DATE=${params[1]}
    export K4ABT_START=${params[2]}
    echo "====" $K4ABT_USER $K4ABT_SESSION "|" $K4ABT_DATE $K4ABT_START "===="

    export K4ABT_INPUT_PATH=/mnt/ops-ds/open-pack/raw/${K4ABT_USER}/kinect/${K4ABT_SESSION}.mkv 
    export K4ABT_OUTPUT_DIR=/mnt/data0/open-pack/interim/${K4ABT_USER}/kinect/3d-kpt
    export K4ABT_OUTPUT_PATH=${K4ABT_OUTPUT_DIR}/${K4ABT_SESSION}.csv

    ./build/k4a_body_tracking ${K4ABT_DATE} ${K4ABT_START} ${DEBUG}
done
getty708 commented 2 years ago

U0104

# === U0104 ===
export K4ABT_USER=U0104

SESSIONS=(
    "S0100 2021-11-16_11:49:46.200 1000"
    "S0200 2021-11-16_14:17:00.400 1000"
    "S0300 2021-11-16_15:26:20.200 1000"
    "S0400 2021-11-16_16:20:57.266 1000"
)

U0105

# === U0105 ===
export K4ABT_USER=U0105

SESSIONS=(
    "S0100 2021-11-19_11:14:09.400 1000"
    "S0200 2021-11-19_12:14:17.600 1000"
    "S0300 2021-11-19_14:00:12.133 1000"
    "S0400 2021-11-19_14:54:12.133 1000"
    "S0500 2021-11-19_15:46:33.333 1000"
)

U0106

# === U0106 ===
export K4ABT_USER=U0106

SESSIONS=(
    "S0100 2021-11-26_11:06:20.667 1000"
    "S0200 2021-11-26_12:18:38.600 1000"
    "S0300 2021-11-26_14:08:43.266 1000"
    "S0400 2021-11-26_15:01:46.000 1000"
    "S0500 2021-11-26_15:58:00.467 1000"
)

U0107

# === U0107 ===
export K4ABT_USER=U0107

SESSIONS=(
    "S0100 2021-12-03_11:03:39.934 1000"
    "S0200 2021-12-03_12:02:37.333 1000"
    "S0300 2021-12-03_13:48:54.400 1000"
    "S0400 2021-12-03_14:38:12.866 1000"
    "S0500 2021-12-03_15:28:06.866 1000"
)

U0108

# === U0108 ===
export K4ABT_USER=U0108

SESSIONS=(
    "S0100 2021-12-09_09:58:16.461 1000"
    "S0200 2021-12-09_10:58:18.666 1000"
    "S0300 2021-12-09_11:56:07.866 1000"
    "S0400 2021-12-09_13:38:30.266 1000"
    "S0500 2021-12-09_14:37:48.498 1000"
)

U0109

# === U0109 ===
export K4ABT_USER=U0109

SESSIONS=(
    "S0100 2021-12-17_12:27:10.733 1000"
    "S0200 2021-12-17_13:36:48.999 1000"
    "S0300 2021-12-17_14:41:27.999 1000"
    "S0400 2021-12-17_15:56:09.866 1000"
    "S0500 2021-12-17_16:53:53.400 1000"
)

U0110

# === U0110 ===
export K4ABT_USER=U0110

SESSIONS=(
    "S0100 2021-12-22_10:52:44.799 1000"
    "S0200 2021-12-22_11:45:55.400 1000"
    "S0300 2021-12-22_13:13:49.400 1000"
    "S0400 2021-12-22_13:56:44.533 1000"
    "S0500 2021-12-22_14:42:22.066 1000"
)

U0111

# === U0111 ===
export K4ABT_USER=U0111

SESSIONS=(
    "S0100 2021-12-27_11:04:06.533 1000"
    "S0200 2021-12-27_12:13:39.200 1000"
    "S0300 2021-12-27_13:57:48.600 1000"
    "S0400 2021-12-27_14:52:55.133 1000"
    "S0500 2021-12-27_15:45:45.600 1000"
)
getty708 commented 2 years ago

LOG

U0107

==== U0107 S0300 | 2021-12-03_13:48:54.400 1000 ====
Input       : /mnt/ops-ds/open-pack/raw/U0107/kinect/S0300.mkv
Output      : /mnt/data0/open-pack/interim/U0107/kinect/3d-kpt/S0300.csv
FUNC        : playback
BASE TIME   : sec=1638506934,  usec=400000
Rec Length [usec]: 2169533400
Start Point : 1.000000 [sec]
End   Point : 2169.533400 [sec]
Succeeded in initializing the body tracker.
[TIME: 0.000 min] 0 frames finised!(0.012%, 1266877 / 2169533400[us])
[TIME: 0.433 min] 1000 frames finised!(3.087%, 67933566 / 2169533400[us])
[TIME: 0.850 min] 2000 frames finised!(6.161%, 134600200 / 2169533400[us])
[TIME: 1.300 min] 3000 frames finised!(9.235%, 201266866 / 2169533400[us])
[TIME: 1.717 min] 4000 frames finised!(12.309%, 267933544 / 2169533400[us])
[TIME: 2.117 min] 5000 frames finised!(15.384%, 334600200 / 2169533400[us])
[TIME: 2.533 min] 6000 frames finised!(18.458%, 401266866 / 2169533400[us])
[TIME: 2.950 min] 7000 frames finised!(21.532%, 467933533 / 2169533400[us])
[TIME: 3.367 min] 8000 frames finised!(24.607%, 534600211 / 2169533400[us])
[TIME: 3.783 min] 9000 frames finised!(27.681%, 601266877 / 2169533400[us])
[TIME: 4.200 min] 10000 frames finised!(30.755%, 667933533 / 2169533400[us])
[TIME: 4.617 min] 11000 frames finised!(33.829%, 734600222 / 2169533400[us])
[TIME: 5.017 min] 12000 frames finised!(36.904%, 801266866 / 2169533400[us])
[TIME: 5.433 min] 13000 frames finised!(39.978%, 867933533 / 2169533400[us])
[TIME: 5.850 min] 14000 frames finised!(43.052%, 934600200 / 2169533400[us])
[TIME: 6.267 min] 15000 frames finised!(46.126%, 1001266866 / 2169533400[us])
[2022-10-17 04:26:17.399] [error] [t=1495112] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/image/image.c (51): k4a_image_t_get_context(). Invalid k4a_image_t (nil)
[2022-10-17 04:26:17.399] [error] [t=1495112] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/image/image.c (389): Invalid argument to image_get_buffer(). image_handle ((nil)) is not a valid handle of type k4a_image_t
[2022-10-17 04:26:17.399] [error] [t=1495112] [K4ABT] ../src/TrackerHost/DepthFrameBlobK4A.cpp (13): Initialize(). Get depth buffer from the capture handle failed!
[2022-10-17 04:26:17.399] [error] [t=1495112] [K4ABT] ../src/TrackerHost/TrackerHost.cpp (274): EnqueueCapture(). Initialize DepthFrameBlob failed!
Error! Add capture to tracker process queue failed!
Failed to get color image from capture @extract_and_write_color_frame()
[2022-10-17 04:26:18.243] [error] [t=1495112] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/image/image.c (51): k4a_image_t_get_context(). Invalid k4a_image_t (nil)
[2022-10-17 04:26:18.243] [error] [t=1495112] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/image/image.c (363): Invalid argument to image_dec_ref(). image_handle ((nil)) is not a valid handle of type k4a_image_t
Failed to get color image from capture @extract_and_write_color_frame()
[2022-10-17 04:26:18.265] [error] [t=1495112] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/image/image.c (51): k4a_image_t_get_context(). Invalid k4a_image_t (nil)
[2022-10-17 04:26:18.265] [error] [t=1495112] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/image/image.c (363): Invalid argument to image_dec_ref(). image_handle ((nil)) is not a valid handle of type k4a_image_t
Failed to get color image from capture @extract_and_write_color_frame()
[2022-10-17 04:26:18.286] [error] [t=1495112] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/image/image.c (51): k4a_image_t_get_context(). Invalid k4a_image_t (nil)
[2022-10-17 04:26:18.286] [error] [t=1495112] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/image/image.c (363): Invalid argument to image_dec_ref(). image_handle ((nil)) is not a valid handle of type k4a_image_t
Failed to get color image from capture @extract_and_write_color_frame()
[2022-10-17 04:26:18.308] [error] [t=1495112] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/image/image.c (51): k4a_image_t_get_context(). Invalid k4a_image_t (nil)
[2022-10-17 04:26:18.308] [error] [t=1495112] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/image/image.c (363): Invalid argument to image_dec_ref(). image_handle ((nil)) is not a valid handle of type k4a_image_t
Failed to get color image from capture @extract_and_write_color_frame()
[2022-10-17 04:26:18.337] [error] [t=1495112] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/image/image.c (51): k4a_image_t_get_context(). Invalid k4a_image_t (nil)
[2022-10-17 04:26:18.337] [error] [t=1495112] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/image/image.c (363): Invalid argument to image_dec_ref(). image_handle ((nil)) is not a valid handle of type k4a_image_t
Failed to get color image from capture @extract_and_write_color_frame()
getty708 commented 2 years ago

実行ログ (第2段)

# === U0201 ===
export K4ABT_USER=U0201

SESSIONS=( 
    "S0100 2022-03-01_11:40:16.133"
    "S0200 2022-03-01_13:50:20.799"
    "S0300 2022-03-01_15:08:47.400"
    "S0400 2022-03-15_15:16:58.999"
    "S0500 2022-03-22_15:03:47.799"
)

U0202

# === U0202 ===
export K4ABT_USER=U0202

SESSIONS=( 
    "S0100 2022-03-04_10:01:22.333"
    "S0200 2022-03-04_11:10:28.999"
    "S0300 2022-03-04_13:05:10.200"
    "S0400 2022-03-04_14:03:38.866"
    "S0500 2022-03-04_14:57:06.066"
)

U0203

# === U0203 ===
export K4ABT_USER=U0203

SESSIONS=( 
    "S0100 2022-03-09_09:50:45.066"
    "S0200 2022-03-09_10:53:16.799"
    "S0300 2022-03-09_11:54:52.133"
    "S0400 2022-03-09_13:37:16.466"
    "S0500 2022-03-09_14:48:21.799"
)

U0204

# === U0204 ===
export K4ABT_USER=U0204

SESSIONS=(
    "S0100 2022-03-11_09:54:43.533"
    "S0200 2022-03-11_10:39:59.533"
    "S0300 2022-03-11_11:30:24.933"
    "S0400 2022-03-11_13:00:05.600"
    "S0500 2022-03-11_13:48:33.999"
)

U0205

# === U0205 ===
export K4ABT_USER=U0205

SESSIONS=(
    "S0100 2022-03-15_09:56:55.666"
    "S0200 2022-03-15_10:55:55.266"
    "S0300 2022-03-15_11:52:43.866"
    "S0400 2022-03-15_13:34:38.266"
    "S0500 2022-03-15_14:23:55.400"
)

U0206

# === U0206 ===
export K4ABT_USER=U0206

SESSIONS=(
    "S0100 2022-03-18_09:43:03.533 1000"
    "S0200 2022-03-18_10:41:32.999 1000"
    "S0300 2022-03-18_11:34:12.600 1000"
    "S0400 2022-03-18_13:12:54.666 1000"
    "S0500 2022-03-18_14:03:03.200 1000"
)

U0207

# === U0207 ===
export K4ABT_USER=U0207

SESSIONS=(
    "S0100 2022-03-22_10:11:39.933 1000"
    "S0200 2022-03-22_11:03:38.200 1000"
    "S0300 2022-03-22_11:51:52.733 1000"
    "S0400 2022-03-22_13:31:42.400 1000"
    "S0500 2022-03-22_14:20:21.666 1000"
)

U0208

# === U0208 ===
export K4ABT_USER=U0208

SESSIONS=(
    "S0100 2022-03-25_09:49:43.066 1000"
    "S0200 2022-03-25_10:44:43.866 1000"
    "S0300 2022-03-25_11:38:51.466 1000"
    "S0400 2022-03-25_13:27:32.666 1000"
    "S0500 2022-03-25_14:20:21.333 1000"
)

U0209

# === U0209 ===
export K4ABT_USER=U0209

SESSIONS=(
    "S0100 2022-03-29_09:43:15.066 1000"
    "S0200 2022-03-29_10:48:23.600 1000"
    "S0300 2022-03-29_11:42:42.466 1000"
    "S0400 2022-03-29_13:22:07.666 1000"
    "S0500 2022-03-29_14:16:10.733 1000"
)

U0210

# === U0210 ===
export K4ABT_USER=U0210

SESSIONS=(
    "S0100 2022-04-01_09:27:03.999 1000"
    "S0200 2022-04-01_10:20:43.533 1000"
    "S0300 2022-04-01_11:10:30.666 1000"
    "S0400 2022-04-01_12:55:15.666 1000"
    "S0500 2022-04-01_13:45:30.266 1000"
)
getty708 commented 2 years ago

BUG

第一弾はデータの抽出やり直しか？ なぜか全くデータがない (ディレクトリの問題で書き込めていないやつか？)

management@cafe-worker2:~/workspace/k4a-frame-extractor/body_tracking$ ls -la /mnt/data0/open-pack/interim
total 60
drwxrwxr-x 15 management management 4096 Oct 28 06:28 .
drwxr-xr-x  3 management root       4096 Aug 29 10:53 ..
drwxrwxr-x  3 management management 4096 Aug 29 11:05 U0101
drwxrwxr-x  3 management management 4096 Aug 29 11:39 U0102
drwxrwxr-x  3 management management 4096 Sep 21 20:41 U0105
drwxrwxr-x  3 management management 4096 Oct 17 12:12 U0201
drwxrwxr-x  3 management management 4096 Oct 17 13:40 U0202
drwxrwxr-x  3 management management 4096 Oct 17 14:55 U0203
drwxrwxr-x  3 management management 4096 Oct 27 23:53 U0204
drwxrwxr-x  3 management management 4096 Oct 28 00:54 U0205
drwxrwxr-x  3 management management 4096 Oct 28 02:09 U0206
drwxrwxr-x  3 management management 4096 Oct 28 03:04 U0207
drwxrwxr-x  3 management management 4096 Oct 28 04:05 U0208
drwxrwxr-x  3 management management 4096 Oct 28 05:13 U0209
drwxrwxr-x  3 management management 4096 Oct 28 06:28 U0210
management@cafe-worker2:~/workspace/k4a-frame-extractor/body_tracking$ date
Tue 01 Nov 2022 09:57:33 PM JST

U0103--U0111 ==> やり直し開始

U0101, U0102 --> やり直し必要