search

starfive-tech / VisionFive2

445 stars 79 forks source link

Monochrome camera sensor support (Y8 Y10) #26

Closed witchcraftsman closed 1 year ago

witchcraftsman commented 1 year ago

Hello, just briefly looked into kernel code and found no out-of-the-box support for:

Those are only two formats my camera sensor can work with.

Is this planned? What are the code changes required to output my camera to /dev/video0 (directly)?

changhuangliang commented 1 year ago

hello, if your sensor use mipi interface, you need to add you sensor formats support into:

stf_csiphy.c ---> csiphy_formats_st7110
stf_csi.c ---> csi_formats_st7110
stf_vin.c ---> vin2_formats_st7110
stf_video.c ---> formats_pix_st7110_wr
witchcraftsman commented 1 year ago

@changhuangliang Does this device tree work without any changes needed? https://github.com/starfive-tech/linux/blob/59cf9af678dbfa3d73f6cb86ed1ae7219da9f5c9/arch/riscv/boot/dts/starfive/jh7110-visionfive-v2.dtsi These two blocks in particular: https://github.com/starfive-tech/linux/blob/59cf9af678dbfa3d73f6cb86ed1ae7219da9f5c9/arch/riscv/boot/dts/starfive/jh7110-visionfive-v2.dtsi#L492 https://github.com/starfive-tech/linux/blob/59cf9af678dbfa3d73f6cb86ed1ae7219da9f5c9/arch/riscv/boot/dts/starfive/jh7110-visionfive-v2.dtsi#L553

So if I have CONFIG_VIDEO_STF_VIN=y and CONFIG_VIN_SENSOR_IMX219=y in my compiled kernel and imx219 camera connected to CSI, I will have the camera successfully probed?

I don't have imx219 ATM but my changes are not so big to adapt my camera (I've also ported my cam driver and rebuilt kernel with it) and I just can't power it on, i2cdetect on i2c6 shows nothing and I've got strange output while scanning: i2c_designware 12060000.i2c: controller timed out i2c_designware 12060000.i2c: timeout in disabling adapter i2cdesignware 12060000.i2c: timeout waiting for bus ready (many times)_ i2c_designware 12060000.i2c: timeout in disabling adapter

vin_sysctl probed successfully though: jh7110-vin 19800000.vin_sysctl: stfcamss probe enter! jh7110-vin 19800000.vin_sysctl: stfcamss probe success!

I know it's a little bit out of issue topic, just don't want to flood. Attaching live device tree and dmesg log, just in case

dtcurrent.txt dmesg.txt

changhuangliang commented 1 year ago

@witchcraftsman you are right, I ignored it.

  1. you need to add you sensor node under the &i2c6
  2. &vin_sysctl->ports->port@1 should add the sensor configure
witchcraftsman commented 1 year ago

@changhuangliang

@witchcraftsman you are right, I ignored it.

  1. you need to add you sensor node under the &i2c6
  2. &vin_sysctl->ports->port@1 should add the sensor configure

I mean of course I've made required changes to bring my camera, take a look at dtcurrent.txt EDIT: adding human readable piece of device tree that overwrites jh7110-visionfive-v2.dtsi jh7110-visionfive-v2-veye-mvcam.dtsi.txt

changhuangliang commented 1 year ago

@witchcraftsman This is a very strange question. try to run: i2cdetect -y -r 6 see the i2c controller can detect you sensor? and run: media-ctl -p can see you sensor info?

witchcraftsman commented 1 year ago

@changhuangliang I've already run i2cdetect (please see my post) and it shows no devices (--). There's no sign the camera is powered on. My camera driver has lots of debug info, but nothing of it is showing in dmesg... I just wanted you to confirm that mentioned dtsi does not require extra lines of code to power imx219 on. Then I will troubleshoot it more thoroughly

changhuangliang commented 1 year ago

@witchcraftsman in out platform, power on imx219 just only use pinctrl: pinctrl-names = "default"; pinctrl-0 = <&csi_pins>;

witchcraftsman commented 1 year ago

@changhuangliang I have the following camera pinout: CSI-2 Interface 1 | GND |   2 | MIPI_L0_N |   3 | MIPI_L0_P |   4 | GND |   5 | MIPI_L1_N |   6 | MIPI_L1_P |   7 | GND |   8 | MIPI_CK_N |   9 | MIPI_CK_P |   10 | GND |   11 | ENABLE | Camera Power Enable 12 | NC |   13 | SCL | 3.3V Level 14 | SDA | 3.3V Level 15 | VCC3V3

So how do I set pin 11 to enable? Browsed through docs and didn't find camera port GPIO bindings...

changhuangliang commented 1 year ago

@witchcraftsman

@changhuangliang I have the following camera pinout: CSI-2 Interface 1 | GND | 2 | MIPI_L0_N | 3 | MIPI_L0_P | 4 | GND | 5 | MIPI_L1_N | 6 | MIPI_L1_P | 7 | GND | 8 | MIPI_CK_N | 9 | MIPI_CK_P | 10 | GND | 11 | ENABLE | Camera Power Enable 12 | NC | 13 | SCL | 3.3V Level 14 | SDA | 3.3V Level 15 | VCC3V3

So how do I set pin 11 to enable? Browsed through docs and didn't find camera port GPIO bindings...

your camera pin is the same as #imx219. So your camera pin 11 is link to gpio18. you just need to operate gpio18.

witchcraftsman commented 1 year ago

@changhuangliang , yeah it was simple in the end, finally it's working. I've just set up the pipeline, and though it looks good, it crashes, please see the logs at the end. I use the following pipeline:

`

set format for mvcam 6-003b

"media-ctl" "-d" "platform:19800000.vin_sysctl" "--set-v4l2" '"mvcam 6-003b":0 [fmt:Y10_1X10/1080x1280]'

set format for stf_csiphy0

"media-ctl" "-d" "platform:19800000.vin_sysctl" "--set-v4l2" '"stf_csiphy0":0 [fmt:Y10_1X10/1080x1280]' "media-ctl" "-d" "platform:19800000.vin_sysctl" "--set-v4l2" '"stf_csiphy0":1 [fmt:Y10_1X10/1080x1280]'

set format for stf_csi0

"media-ctl" "-d" "platform:19800000.vin_sysctl" "--set-v4l2" '"stf_csi0":0 [fmt:Y10_1X10/1080x1280]' "media-ctl" "-d" "platform:19800000.vin_sysctl" "--set-v4l2" '"stf_csi0":1 [fmt:Y10_1X10/1080x1280]'

set link stf_csi0 -> stf_vin0_wr and enable it

"media-ctl" "-d" "platform:19800000.vin_sysctl" "-l" "'stf_csi0':1 -> 'stf_vin0_wr':0 [1]"

set format for stf_vin0_wr

"media-ctl" "-d" "platform:19800000.vin_sysctl" "--set-v4l2" '"stf_vin0_wr":0 [fmt:Y10_1X10/1080x1280]' "media-ctl" "-d" "platform:19800000.vin_sysctl" "--set-v4l2" '"stf_vin0_wr":1 [fmt:Y10_1X10/1080x1280]' `

The topology is in attached file. mvcam_vf2_topology.txt

The error output in dmesg after making a raw shot with the following command is below: sudo v4l2-ctl --device /dev/video0 --set-fmt-video=width=1080,height=1280,pixelformat=YUYV --stream-mmap --stream-to=./frame.raw --stream-count=1

[ 2468.749585] [st_video] error: Failed to media_pipeline_start: -32 [ 2468.755832] ------------[ cut here ]------------ [ 2468.760448] WARNING: CPU: 3 PID: 1481 at drivers/media/common/videobuf2/videobuf2-core.c:1548 vb2_start_streaming+0xb6/0x118 [ 2468.771676] Modules linked in: [ 2468.774732] CPU: 3 PID: 1481 Comm: v4l2-ctl Tainted: G W 5.15.0-dirty #8 [ 2468.782737] Hardware name: StarFive VisionFive V2 (DT) [ 2468.787874] epc : vb2_start_streaming+0xb6/0x118 [ 2468.792492] ra : vb2_start_streaming+0x6c/0x118 [ 2468.797109] epc : ffffffff806cf900 ra : ffffffff806cf8b6 sp : ffffffd02b14bb80 [ 2468.804336] gp : ffffffff81504698 tp : ffffffe0c520e900 t0 : ffffffff8151455f [ 2468.811563] t1 : ffffffff81514550 t2 : 0000000000000000 s0 : ffffffd02b14bbc0 [ 2468.818789] s1 : ffffffffffffffe0 a0 : ffffffffffffffe0 a1 : ffffffe1f89a3670 [ 2468.826016] a2 : ffffffe1f89a6228 a3 : 0000000000000000 a4 : ea341e8ad0684800 [ 2468.833243] a5 : 0000000000000004 a6 : 0000000000017fe8 a7 : c0000000ffffefff [ 2468.840470] s2 : ffffffe0c1238258 s3 : ffffffe0c1238448 s4 : 0000000000000001 [ 2468.847696] s5 : 0000000000000004 s6 : ffffffff815060a8 s7 : 0000000000000000 [ 2468.854922] s8 : 0000000000000000 s9 : ffffffe0c1238d40 s10: ffffffff80ece678 [ 2468.862149] s11: 0000000000000000 t3 : 000000000000005b t4 : ffffffffffffffff [ 2468.869375] t5 : 0000000000000438 t6 : ffffffd02b14b898 [ 2468.874686] status: 0000000200000120 badaddr: 0000000000000000 cause: 0000000000000003 [ 2468.882608] [] vb2_start_streaming+0xb6/0x118 [ 2468.888530] [] vb2_core_streamon+0x68/0x13e [ 2468.894275] [] vb2_streamon+0x14/0x4e [ 2468.899503] [] vb2_ioctl_streamon+0x38/0x44 [ 2468.905246] [] v4l_streamon+0x16/0x1e [ 2468.910476] [] __video_do_ioctl+0x142/0x378 [ 2468.916221] [] video_usercopy+0x10e/0x3d6 [ 2468.921795] [] video_ioctl2+0x14/0x1c [ 2468.927021] [] v4l2_ioctl+0x30/0x44 [ 2468.932071] [] sys_ioctl+0xa8/0x812 [ 2468.937125] [] ret_from_syscall+0x0/0x2 [ 2468.942524] ---[ end trace 8642539a7e1c509a ]---

witchcraftsman commented 1 year ago

Attaching changed driver files, just for reference. Archive.zip

changhuangliang commented 1 year ago

@witchcraftsman

maybe you can try my patch

0001-media-starfive-Add-Y8-Y10-format-support.patch

witchcraftsman commented 1 year ago

@changhuangliang after setting the pipeline and running v4l2-ctl -d /dev/video0 --set-fmt-video=width=1080,height=1280,pixelformat=GREY --stream-mmap --stream-to=./frame.raw --stream-skip=9 --stream-count=1 I got The pixelformat 'GREY' is invalid

Here is my video0 formats: `v4l2-ctl -d /dev/video0 --list-formats-ext ioctl: VIDIOC_ENUM_FMT Type: Video Capture

[0]: 'YUYV' (YUYV 4:2:2)
    Size: Continuous 64x64 - 1920x1080
[1]: 'RGBP' (16-bit RGB 5-6-5)
    Size: Continuous 64x64 - 1920x1080
[2]: 'RGGB' (8-bit Bayer RGRG/GBGB)
    Size: Continuous 64x64 - 1920x1080
[3]: 'GRBG' (8-bit Bayer GRGR/BGBG)
    Size: Continuous 64x64 - 1920x1080
[4]: 'GBRG' (8-bit Bayer GBGB/RGRG)
    Size: Continuous 64x64 - 1920x1080
[5]: 'BA81' (8-bit Bayer BGBG/GRGR)
    Size: Continuous 64x64 - 1920x1080
[6]: 'RG10' (10-bit Bayer RGRG/GBGB)
    Size: Continuous 64x64 - 1920x1080
[7]: 'BA10' (10-bit Bayer GRGR/BGBG)
    Size: Continuous 64x64 - 1920x1080
[8]: 'GB10' (10-bit Bayer GBGB/RGRG)
    Size: Continuous 64x64 - 1920x1080
[9]: 'BG10' (10-bit Bayer BGBG/GRGR)
    Size: Continuous 64x64 - 1920x1080
[10]: 'Y8  ' (Y8   little-endian (0x20203859))
    Size: Continuous 64x64 - 1920x1080
[11]: 'Y10 ' (10-bit Greyscale)
    Size: Continuous 64x64 - 1920x1080`

What bothers me is that video0 device format size is not linked/propagated from stf_vin0_wr

EDIT: replaced V4L2_PIX_FMT_Y8 with V4L2_PIX_FMT_GREY in stf_video.c Now v4l2-ctl -d /dev/video0 --set-fmt-video=width=1080,height=1280,pixelformat=GREY --stream-mmap --stream-to=./frame.raw --stream-skip=9 --stream-count=1 starts, but crashes the same way (please notice the first error message about VIDIOC_S_SELECTION). Crash log: crash_log_2.txt

Strange thing also that starting from stf_csiphy0 down to video0, all pads and sources are Y8_1X8/1080x1080 now, but not Y8_1X8/1080x1280.

Just to make clear, now I'm just enabling this link: stf_csi0 -> stf_vin0_wr and my pipeline looks like: mvcam 6-003b -> stf_csiphy0 -> stf_csi0 -> stf_vin0_wr -> stf_vin0_wr_video0

witchcraftsman commented 1 year ago

Another update: was able to successfully capture an image. The issue was the camera sensor is RAW so I had to set a crop on it with v4l2-ctl -d /dev/v4l-subdev10 --try-subdev-selection pad=0,target=crop,top=256,left=28,width=1024,height=768 and then capture with v4l2-ctl -d /dev/video0 --set-fmt-video=width=1024,height=768,pixelformat=GREY --stream-mmap --stream-to=./frame.raw --stream-skip=9 --stream-count=1.

Now do I need to revert this patch back? writel(CSI2RX_STREAM_DATA_CFG_EN_VC_SELECT | - CSI2RX_STREAM_DATA_CFG_VC_SELECT(i) | - CSI2RX_STREAM_DATA_CFG_EN_DATA_TYPE_0 | dt, + CSI2RX_STREAM_DATA_CFG_VC_SELECT(i), reg_base + CSI2RX_STREAM_DATA_CFG_REG(i));

And is it possible to capture with Y10? Now as you may guess I'm getting The pixelformat 'Y10' is invalid as it was before changing Y8 to GREY

changhuangliang commented 1 year ago

@witchcraftsman

Now do I need to revert this patch back? writel(CSI2RX_STREAM_DATA_CFG_EN_VC_SELECT | - CSI2RX_STREAM_DATA_CFG_VC_SELECT(i) | - CSI2RX_STREAM_DATA_CFG_EN_DATA_TYPE_0 | dt, + CSI2RX_STREAM_DATA_CFG_VC_SELECT(i), reg_base + CSI2RX_STREAM_DATA_CFG_REG(i));

In my personal debug, this patch can match all formats automaticallly for csi tranfer. if you warn to revert this patch. you need to add your sensor format code MEDIA_BUS_FMT_Y8_1X8 MEDIA_BUS_FMT_Y10_1X10 in the stf_csi.c -> code_to_data_type, int this function your also need to return your sensor format data type(see MIPI_Aliance_Specification for camera serial interface 2), but I don't find the match data type in it. so I send you this change.

changhuangliang commented 1 year ago

@witchcraftsman

And is it possible to capture with Y10? Now as you may guess I'm getting The pixelformat 'Y10' is invalid as it was before changing Y8 to GREY

I guess it is enought to capture the ‘’Y10” Image. When invaild, maybe the pipeline format don't match?

witchcraftsman commented 1 year ago

Yeah, v4l2-ctl --device /dev/video0 --set-fmt-video=width=1024,height=768,pixelformat='Y10 ' --stream-mmap --stream-to=./frame.raw --stream-count=1 works so I'm closing the issue. There's a long way ahead of setting up gstreamer with Y10