Need further explain for MJPEG decode

[crab2313]

Can I use the put_packet/get_frame interface to decode MJPEG ? After doing that I found there is no frame from the output side.

[HermanChen]

Not fully verified. You can have a try : )

[tikonen]

Looks like it does not work? No errors are returned from any of the api calls but there is never output frame after calling decode or decode_get_frame.

[HermanChen]

What is your error log?

[tikonen]

Thanks for looking into this. Here is example, I'm trying to decode a raw MJPEG frame (file frame-1.raw in following).

rock64@rockpro64:~/rockchip/mpp/build/linux/aarch64$ xxd frame-1.raw | head -c 2k
00000000: ffd8 ffe0 0010 4a46 4946 0001 0100 0001  ......JFIF......
00000010: 0001 0000 ffdb 0043 0003 0202 0302 0203  .......C........
00000020: 0303 0304 0303 0405 0805 0504 0405 0a07  ................
00000030: 0706 080c 0a0c 0c0b 0a0b 0b0d 0e12 100d  ................
00000040: 0e11 0e0b 0b10 1610 1113 1415 1515 0c0f  ................
00000050: 1718 1614 1812 1415 14ff c000 0b08 0190  ................
00000060: 0500 0101 1100 ffc4 001f 0000 0105 0101  ................
00000070: 0101 0101 0000 0000 0000 0000 0102 0304  ................
00000080: 0506 0708 090a 0bff c400 b510 0002 0103  ................
00000090: 0302 0403 0505 0404 0000 017d 0102 0300  ...........}....
000000a0: 0411 0512 2131 4106 1351 6107 2271 1432  ....!1A..Qa."q.2
000000b0: 8191 a108 2342 b1c1 1552 d1f0 2433 6272  ....#B...R..$3br
000000c0: 8209 0a16 1718 191a 2526 2728 292a 3435  ........%&'()*45
000000d0: 3637 3839 3a43 4445 4647 4849 4a53 5455  6789:CDEFGHIJSTU
000000e0: 5657 5859 5a63 6465 6667 6869 6a73 7475  VWXYZcdefghijstu
000000f0: 7677 7879 7a83 8485 8687 8889 8a92 9394  vwxyz...........
00000100: 9596 9798 999a a2a3 a4a5 a6a7 a8a9 aab2  ................
00000110: b3b4 b5b6 b7b8 b9ba c2c3 c4c5 c6c7 c8c9  ................
00000120: cad2 d3d4 d5d6 d7d8 d9da e1e2 e3e4 e5e6  ................
00000130: e7e8 e9ea f1f2 f3f4 f5f6 f7f8 f9fa ffda  ................
00000140: 0008 0101 0000 3f00 fcdb 18ef 566d 933d  ......?.....Vm.=
00000150: 7f9d 69db 45e6 36e5 03fa 0ad3 4523 8c6e  ..i.E.6.....E#.n
00000160: 0076 ab6a b938 53cf 7c55 8814 9395 1d3a  .v.j.8S.|U.....:
00000170: d6a6 9f3b 5ac8 8541 ce7a a1af d1cf f827  ...;Z..A.z.....'
00000180: ff00 ed3b 6b06 3c03 af5d 0852 76dd 612c  ...;k.<..].Rv.a,
00000190: a701 5cf5 427d ff00 9fd6 befc d474 fb7d  ..\.B}.......t.}
000001a0: 5ac6 7b3b c852 e2d6 7468 a586 45ca ba91  Z.{;.R..th..E...
000001b0: 8208 fa57 e48f ed8b fb34 5d7c 16f1 bbdc  ...W.....4]|....
000001c0: 5842 d378 6f51 7692 ce7f ee7a c6de ebfa  XB.xoQv....z....
000001d0: e73d ebe6 b6b6 2cd9 c145 0783 ed52 6d63  .=....,..E...Rmc

000001e0rock64@rockpro64:~/rockchip/mpp/build/linux/aarch64$ file frame-1.raw
frame-1.raw: JPEG image data, JFIF standard 1.01, aspect ratio, density 1x1, segment length 16, baseline, precision 8, 1280x400, frames 1

At least the data looks valid JPEG frame but decoder complains

rock64@rockpro64:~/rockchip/mpp/build/linux/aarch64$ test/mpi_dec_test -i frame-1.raw -o frame-1.mpeg2 -w 1280 -h 400 -t 8 -f 2 -d 1
mpi_dec_test: cmd parse result:
mpi_dec_test: input  file name: frame-1.raw
mpi_dec_test: output file name: frame-1.mpeg2
mpi_dec_test: config file name: 
mpi_dec_test: width      : 1280
mpi_dec_test: height     :  400
mpi_dec_test: type       : 8
mpi_dec_test: debug flag : 1
mpi_dec_test: max frames : 0
mpi_dec_test: mpi_dec_test start
mpi_dec_test: input file size 78659
mpp_rt: NOT found ion allocator
mpp_rt: found drm allocator
mpi_dec_test: mpi_dec_test decoder test start w 1280 h 400 type 8
mpi: mpp_create enter ctx 0x7fcc235008 mpi 0x7fcc235010
mpi: mpp version: a7115286 author: Johnson Ding [dec]: fix info_changed property setting error
mpi: mpp_create leave ret 0 ctx 0x5590e2e1a0 mpi 0x7f95020520
mpi: mpi_control enter ctx 0x5590e2e1a0 cmd 310005 parm 0x7fcc235000
mpi: mpi_control leave ret 0
mpi: mpp_init enter ctx 0x5590e2e1a0 type 0 coding 8
mpi: mpp_init leave ret 0
mpi: mpi_control enter ctx 0x5590e2e1a0 cmd 31000a parm 0x7fcc23554c
mpi: mpi_control leave ret 0
mpi: mpi_poll enter ctx 0x5590e2e1a0 type 0 timeout -1
mpi: mpi_poll leave ret 0
mpi: mpi_dequeue enter ctx 0x5590e2e1a0 type 0 task 0x7fcc235038
mpi: mpi_dequeue leave ret 0
mpi: mpi_enqueue enter ctx 0x5590e2e1a0 type 0 task 0x5590e31b40
mpi: mpi_enqueue leave ret 0
mpi: mpi_poll enter ctx 0x5590e2e1a0 type 1 timeout -1
jpegd_parser: input img maybe png format,check it
mpi: mpi_poll leave ret 0
mpi: mpi_dequeue enter ctx 0x5590e2e1a0 type 1 task 0x7fcc235038
mpi: mpi_dequeue leave ret 0
mpi_dec_test: decoded frame 1
mpi: mpi_enqueue enter ctx 0x5590e2e1a0 type 1 task 0x5590e31ca0
mpi: mpi_enqueue leave ret 0
mpi_dec_test: found last packet
mpi: mpi_poll enter ctx 0x5590e2e1a0 type 0 timeout -1
mpi: mpi_poll leave ret 0
mpi: mpi_dequeue enter ctx 0x5590e2e1a0 type 0 task 0x7fcc235038
mpi: mpi_dequeue leave ret 0
mpi: mpi_enqueue enter ctx 0x5590e2e1a0 type 0 task 0x5590e31b40
mpi: mpi_enqueue leave ret 0
mpi: mpi_poll enter ctx 0x5590e2e1a0 type 1 timeout -1
mpi: mpi_poll leave ret 0
mpi: mpi_dequeue enter ctx 0x5590e2e1a0 type 1 task 0x7fcc235038
mpi: mpi_dequeue leave ret 0
mpi_dec_test: decoded frame 2
mpi_dec_test: found eos frame
mpi: mpi_enqueue enter ctx 0x5590e2e1a0 type 1 task 0x5590e31ca0
mpi: mpi_enqueue leave ret 0
mpi: mpi_reset enter ctx 0x5590e2e1a0
mpi: mpi_reset leave ret 0
mpi: mpp_destroy enter ctx 0x5590e2e1a0
mpi: mpp_destroy leave ret 0
mpi_dec_test: test success max memory 0.00 MB

Output appears to be garbage.

rock64@rockpro64:~/rockchip/mpp/build/linux/aarch64$ file frame-1.mpeg2 
frame-1.mpeg2: data
rock64@rockpro64:~/rockchip/mpp/build/linux/aarch64$ xxd frame-1.mpeg2 | head -c 2k
00000000: 1212 1212 1212 1212 1314 1415 1616 1717  ................
00000010: 1819 1a1a 191a 1d1f 2023 2525 2424 2423  ........ #%%$$$#
00000020: 2726 2527 292a 2a2a 2828 2829 2a2c 2e2f  '&%')***((()*,./
00000030: 2c2c 2e32 363b 434a 6277 cbf9 ffff fffc  ,,.26;CJbw......
00000040: fdff faff fffd e48c 5a44 4737 403a 3b39  ........ZDG7@:;9
00000050: 373a 3b38 3637 393b 3e3f 3f3e 3c3a 3b3c  7:;8679;>??><:;<
00000060: 3b3c 3c3a 3939 3b3e 3a37 3839 3738 3b3c  ;<<:99;>:78978;<
00000070: 363c 3638 3d3b 3b3d 3839 3b3d 3d3b 3d40  6<68=;;=89;==;=@
00000080: 3838 393a 3c3e 4041 3b3c 3e3f 3f3e 3c3b  889:<>@A;<>??><;
00000090: 3a39 3b40 4240 3f3f 3c3b 393a 3d3e 3c39  :9;@B@??<;9:=><9
000000a0: 3e41 4340 3c40 413c 3d3c 3d42 453f 3d43  >AC@<@A<=<=BE?=C
000000b0: 3f41 4241 4142 4240 3c40 403d 3c3f 3f3d  ?ABAABB@<@@=<??=
000000c0: 3d3b 3d41 4141 3e38 453e 413f 443a 3e40  =;=AAA>8E>A?D:>@
000000d0: 4142 4344 4442 3f3d 3a3c 3e40 4140 3f3d  ABCDDB?=:<>@A@?=
000000e0: 4440 453f 3d41 3c41 423f 3e3e 3e3d 3d3e  D@E?=A<AB?>>>==>
000000f0: 3d3c 3d3f 4040 3e3b 3d3d 3836 393a 3a3d  =<=?@@>;==869::=
00000100: 373c 3c39 3a3e 3d38 3d39 3637 3939 3a3b  7<<9:>=8=96799:;
00000110: 363a 3d3c 3a39 3837 383a 3a37 3739 3834  6:=<:9878::77984
00000120: 3539 3835 3637 383d 3435 3738 3634 3437  5985678=45786447
00000130: 3839 3a3a 3836 3637 3735 3537 3837 3638  89::866775578768
00000140: 3837 3736 3636 3636 3637 3939 3837 3535  8776666667998755
00000150: 3633 393b 3c3b 3537 393b 3b39 3839 3835  639;<;579;;98985
00000160: 3a35 383b 3637 3b39 3737 3838 3938 3737  :58;67;977889877
00000170: 3737 3737 3737 3737 3836 3637 3533 3539  7777777786675359
00000180: 3837 3c38 3d36 3733 3838 3737 3936 363a  87<8=6738877966:
00000190: 373b 3c39 383b 3b37 393b 3b39 393c 3b37  7;<98;;79;;99<;7
000001a0: 3a3d 383a 3841 3d3b 3f3a 3c40 3e3c 3d3e  :=8:8A=;?:<@><=>
000001b0: 3c3a 393b 3c3c 3c3e 3b3a 3c3e 3d3a 393b  <:9;<<<>;:<>=:9;
000001c0: 3b3c 3e3d 3b3a 3b3d 383a 3e42 423e 3b3b  ;<>=;:;=8:>BB>;;
000001d0: 3f3f 403e 3d41 423b 3f44 4344 4341 4442  ??@>=AB;?DCDCADB

Version I'm running is release branch.

rock64@rockpro64:~/rockchip/mpp/build/linux/aarch64$ git log HEAD
commit a711528671a842cf998573ed3eb33cc9575002d4 (HEAD -> release, origin/release, origin/develop, origin/HEAD)
Author: Johnson Ding <johnson.ding@rock-chips.com>
Date:   Wed May 29 16:06:14 2019 +0800
...

I'm not sure if related but I see this on /dev/console every time I run the mpi_dec_test command.

[366980.418129] rk_vcodec: vpu_service_ioctl:1890: error: unknown vpu service ioctl cmd 40086c01

[HermanChen]

Can you provide the file frame-1.raw?

[tikonen]

Can you provide the file frame-1.raw?

Sure, but I can't share it publicly here in the ticket. Can you give me email / upload link for sending the file?

[HermanChen]

Could you please send it to haiqiangding@gmail.com. We will analyze it.

[qvoid]

@tikonen I test it in my RK3328 board. The mpi_dec_test does decode frame-1.raw well and output a right NV12 picture. The frame-1.raw is a grayscale JPEG picture. The frame-1.mpeg2 is a NV12 file but only has Y within, since it is decoded from a grayscale picture. You can open it in this site or with Vooya. As for those abnormal log printed in logcat, I am still analyzing.

[qvoid]

@tikonen I use this command:

mpi_dec_test -i ./frame-1.raw -w 1280 -h 400 -t 8 -d 1 -o ./1280x400.nv12 -f 0

Using -f 0 not -f 2. Although neither of them is correct, but with -f 0, I got the right NV12 file.

[tikonen]

@qvoid I tested with the command (-f 0 flag) and got a valid greyscale NV12 file. Great! This is all I need.

Still see the debug message jpegd_parser: input img maybe png format,check it and the console message [441913.146256] rk_vcodec: vpu_service_ioctl:1890: error: unknown vpu service ioctl cmd 40086c01. I guess they are not real errors?

[qvoid]

@tikonen The debug message:

jpegd_parser: input img maybe png format,check it

is not a error message. And I didn't see rk_vcodec print something like :

unknown vpu service ioctl cmd

Perhaps it is printed by last command. You can confirm this by running dmesg -C before testing with mpi_dec_test. As for mpi_dec_test didn't produce the right yuv file, that's because the Utils.c in MPP, function dump_mpp_frame_to_file didn't handle greyscale output. So try to add this patch to it:

diff --git a/utils/utils.c b/utils/utils.c
index 67c1c0f..a37132d 100644
--- a/utils/utils.c
+++ b/utils/utils.c
@@ -132,6 +132,15 @@ void dump_mpp_frame_to_file(MppFrame frame, FILE *fp)
         fwrite(tmp, 1, width * height * 2, fp);
         mpp_free(tmp);
     } break;
+    case MPP_FMT_YUV400SP: {
+        RK_U32 i;
+        RK_U8 *base_y = base;
+        RK_U8 *tmp = mpp_malloc(RK_U8, h_stride * height);
+        for (i = 0; i < height; i++, base_y += h_stride) {
+            fwrite(base_y, 1, width, fp);
+        }
+        mpp_free(tmp);
+    } break;
     default : {
         mpp_err("not supported format %d\n", fmt);
     } break;

After recompiling and re-push mpp libraries to the correct directory, you can run mpp_dec_test without -f specified and get the right output.

The formal patch will be merged into this repo soon.

[tikonen]

Thanks will try out the patch! I checked with clear ring buffer and looks like the console output is from running the mpi_dec_test.

rock64@rockpro64:~/rockchip/mpp/build/linux/aarch64$ sudo dmesg -C
rock64@rockpro64:~/rockchip/mpp/build/linux/aarch64$ test/mpi_dec_test -i ./frame-1.raw -w 1280 -h 400 -t 8 -o ./1280x400.nv12 -f 0
mpi_dec_test: cmd parse result:
mpi_dec_test: input  file name: ./frame-1.raw
mpi_dec_test: output file name: ./1280x400.nv12
mpi_dec_test: config file name: 
mpi_dec_test: width      : 1280
mpi_dec_test: height     :  400
mpi_dec_test: type       : 8
mpi_dec_test: debug flag : 0
mpi_dec_test: max frames : 0
mpi_dec_test: mpi_dec_test start
mpi_dec_test: input file size 78659
mpp_rt: NOT found ion allocator
mpp_rt: found drm allocator
mpi_dec_test: mpi_dec_test decoder test start w 1280 h 400 type 8
mpi: mpp version: a7115286 author: Johnson Ding [dec]: fix info_changed property setting error
jpegd_parser: input img maybe png format,check it
mpi_dec_test: decoded frame 1
mpi_dec_test: found last packet
mpi_dec_test: decoded frame 2
mpi_dec_test: found eos frame
mpi_dec_test: test success max memory 0.00 MB
rock64@rockpro64:~/rockchip/mpp/build/linux/aarch64$ dmesg 
[533208.233926] rk_vcodec: vpu_service_ioctl:1890: error: unknown vpu service ioctl cmd 40086c01
rock64@rockpro64:~/rockchip/mpp/build/linux/aarch64$ 

My kernel is

rock64@rockpro64:~/rockchip/mpp/build/linux/aarch64$ uname -a
Linux rockpro64 4.4.167-1197-rockchip-ayufan-ga7ca17f30176 #1 SMP Mon Jun 3 22:27:06 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux

[HermanChen]

The kernel vpu_service_ioctl log is our platform detection from mpp. There are some kernel driver versions. Mpp will try different ioctl operation to check detect the version. So this error log is ok for detection stage.