search

libsdl-org / SDL

Simple Directmedia Layer
https://libsdl.org
zlib License
10.2k stars 1.86k forks source link

Camera(Android): Chroma plane(s) shifted #11114

Open Green-Sky opened 1 month ago

Green-Sky commented 1 month ago

I took a look at the camera code and there is nothing obvious wrong with it, however we are not taking into account the stride of the planes and the crop rect(?).

The color format is yuv NV12, displayed unchanged, but it was copied (SurfaceDuplicated). Both front-facing and back-facing cameras are affected, all resolutions I tried too.

It is also not the upload to texture code that is wrong, as converting it to IYUV and sending it over the wire looks the same.

image

slouken commented 1 month ago

Looking at the camera code it's almost certainly wrong, but it would be helpful to have a repro case to test any fix. What Android device are you using to test this?

I assume you're seeing the same problem with SDL's testcamera test program?

Green-Sky commented 1 month ago

Ok, I hacked the ci to produce the testcamera-apk, since i have neither freespace on my devmachine nor the motivation to bring in the android sdk/ndk. (interesting ci setup)

The output of the test is actually way worse for some reason, but it is also from latest master, so that might have influenced it.

Screenshot_20241018-172927 This is supposed to be a wall. (same wall as in op, but without paper with drawing)

Both front- and backfacing cameras are affected, there is a pattern difference, which I suppose is a result of a resolution difference.

Another thing, when you touch(click) the image, the camera switches to the other, but the text "testcamer: Front-facing camera (android)" stays the same.

slouken commented 1 month ago

What Android device is this?

Green-Sky commented 1 month ago

What Android device is this?

OnePlus Nord CE 5G

I am undusting an old Moto G android 7.1 from my drawer right now.

Green-Sky commented 1 month ago

I am undusting an old Moto G android 7.1 from my drawer right now.

Despite the ndk camera api being introduced 1 API level earlier (7.0), SDL does not see them on this device. (sad)

SDL/APP : Saw 0 camera devices.
SDL/APP : No cameras available?
Green-Sky commented 1 month ago

The output of the test is actually way worse for some reason, but it is also from latest master, so that might have influenced it.

Updated to latest and it still looks the same (as in op).

1bsyl commented 1 month ago

@Green-Sky maybe you can add some debug in SDL code to see what's wrong:

show if there are some unknown format we could handle: https://github.com/libsdl-org/SDL/blob/main/src/camera/android/SDL_camera_android.c#L270

https://github.com/libsdl-org/SDL/blob/main/src/camera/android/SDL_camera_android.c#L325 display the number of planes and for each plane: the row stride ( pAImage_getPlaneRowStride ) and the pixel stride:

Similarly to "pAImage_getPlaneRowStride", you need to add the functions and code to get the pixel stride :

static pfnAImage_getPlanePixelStride pAImage_getPlanePixelStride = NULL;
LOADSYM(libmedia, AImage_getPlanePixelStride);
typedef media_status_t (*pfnAImage_getPlanePixelStride)(const AImage*, int, int32_t*);
Green-Sky commented 1 month ago

I did as @1bsyl said:

Window requested size 640x480, got 1080x2014
CAMERA: MaybeAddDevice('0')
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '37'
Unknown format AIMAGE_FORMAT '32'
Unknown format AIMAGE_FORMAT '36'
CAMERA: Adding device 'Back-facing camera' (back-facing) with 64 specs:
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=4624, h=3472, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=4624, h=2080, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=4608, h=3456, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=4608, h=2592, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=4608, h=2256, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=4608, h=2080, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=4160, h=3120, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=4160, h=1890, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=4160, h=1872, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=4000, h=3000, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=4000, h=2250, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=4000, h=1818, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=4000, h=1800, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=3840, h=2160, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=3840, h=1644, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=3472, h=3472, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=3456, h=3456, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=3456, h=2592, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=3280, h=2464, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=3264, h=2448, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=3264, h=1836, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=3264, h=1472, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=3200, h=2400, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=3168, h=1440, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=3120, h=3120, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=3000, h=3000, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=2880, h=2160, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=2688, h=1512, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=2592, h=1944, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=2592, h=1168, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=2448, h=2448, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=2400, h=1080, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=2376, h=1080, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=2328, h=1748, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=2304, h=1728, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=2160, h=1080, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1944, h=1944, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1920, h=1080, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1920, h=864, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1920, h=822, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1600, h=1200, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1600, h=800, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1600, h=720, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1584, h=720, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1440, h=1080, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1280, h=960, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1280, h=768, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1280, h=720, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1200, h=1200, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1200, h=540, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1188, h=540, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1080, h=1080, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1024, h=768, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=840, h=360, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=800, h=400, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=792, h=360, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=720, h=540, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=720, h=480, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=640, h=640, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=640, h=480, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=640, h=360, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=352, h=288, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=320, h=240, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=176, h=144, numerator=30, denominator=1
CAMERA: MaybeAddDevice('1')
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '34'
Unknown format AIMAGE_FORMAT '256'
Unknown format AIMAGE_FORMAT '37'
Unknown format AIMAGE_FORMAT '32'
Unknown format AIMAGE_FORMAT '36'
CAMERA: Adding device 'Front-facing camera' (front-facing) with 30 specs:
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=2328, h=1748, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=2304, h=1728, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=2160, h=1080, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1920, h=1080, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1920, h=864, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1920, h=822, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1600, h=1200, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1600, h=800, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1600, h=720, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1584, h=720, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1440, h=1080, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1280, h=960, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1280, h=768, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1280, h=720, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1200, h=1200, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1200, h=540, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1188, h=540, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1080, h=1080, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=1024, h=768, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=840, h=360, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=800, h=400, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=792, h=360, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=720, h=540, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=720, h=480, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=640, h=640, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=640, h=480, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=640, h=360, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=352, h=288, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=320, h=240, numerator=30, denominator=1
CAMERA:   - fmt=SDL_PIXELFORMAT_NV12, w=176, h=144, numerator=30, denominator=1
CAMERA: CB onCameraAvailable('0')
CAMERA: MaybeAddDevice('0')
CAMERA: CB onCameraAvailable('1')
CAMERA: MaybeAddDevice('1')
CAMERA: CB onCameraAvailable('2')
CAMERA: MaybeAddDevice('2')
Saw 2 camera devices.
  - Camera #0: [back-facing]  Back-facing camera
  - Camera #1: [front-facing]  Front-facing camera
Available formats:
    2328x1748 30.00 FPS SDL_PIXELFORMAT_NV12
    2304x1728 30.00 FPS SDL_PIXELFORMAT_NV12
    2160x1080 30.00 FPS SDL_PIXELFORMAT_NV12
    1920x1080 30.00 FPS SDL_PIXELFORMAT_NV12
    1920x864 30.00 FPS SDL_PIXELFORMAT_NV12
    1920x822 30.00 FPS SDL_PIXELFORMAT_NV12
    1600x1200 30.00 FPS SDL_PIXELFORMAT_NV12
    1600x800 30.00 FPS SDL_PIXELFORMAT_NV12
    1600x720 30.00 FPS SDL_PIXELFORMAT_NV12
    1584x720 30.00 FPS SDL_PIXELFORMAT_NV12
    1440x1080 30.00 FPS SDL_PIXELFORMAT_NV12
    1280x960 30.00 FPS SDL_PIXELFORMAT_NV12
    1280x768 30.00 FPS SDL_PIXELFORMAT_NV12
    1280x720 30.00 FPS SDL_PIXELFORMAT_NV12
    1200x1200 30.00 FPS SDL_PIXELFORMAT_NV12
    1200x540 30.00 FPS SDL_PIXELFORMAT_NV12
    1188x540 30.00 FPS SDL_PIXELFORMAT_NV12
    1080x1080 30.00 FPS SDL_PIXELFORMAT_NV12
    1024x768 30.00 FPS SDL_PIXELFORMAT_NV12
    840x360 30.00 FPS SDL_PIXELFORMAT_NV12
    800x400 30.00 FPS SDL_PIXELFORMAT_NV12
    792x360 30.00 FPS SDL_PIXELFORMAT_NV12
    720x540 30.00 FPS SDL_PIXELFORMAT_NV12
    720x480 30.00 FPS SDL_PIXELFORMAT_NV12
    640x640 30.00 FPS SDL_PIXELFORMAT_NV12
    640x480 30.00 FPS SDL_PIXELFORMAT_NV12
    640x360 30.00 FPS SDL_PIXELFORMAT_NV12
    352x288 30.00 FPS SDL_PIXELFORMAT_NV12
    320x240 30.00 FPS SDL_PIXELFORMAT_NV12
    176x144 30.00 FPS SDL_PIXELFORMAT_NV12

at starup (some more but redundant info.

with AIMAGE_FORMAT_PRIVATE = 0x22 (34) AIMAGE_FORMAT_JPEG = 0x100 (256) AIMAGE_FORMAT_RAW10 = 0x25 (37) AIMAGE_FORMAT_RAW16 = 0x20 (32) AIMAGE_FORMAT_RAW_PRIVATE = 0x24 (36)

CAMERA: CB onActive
Camera approved!
Camera Spec: 2328x1748 30.00 FPS SDL_PIXELFORMAT_NV12
CAMERA: CB onImageAvailable
CAMERA: NumberOfPlanes: 3
CAMERA: frame plane 0 row stride: 2368
CAMERA: frame plane 0 pixel stride: 2368
CAMERA: frame plane 1 row stride: 2368
CAMERA: frame plane 1 pixel stride: 2368
CAMERA: frame plane 2 row stride: 2368
CAMERA: frame plane 2 pixel stride: 2368
CAMERA: New frame available! pixels=0xb4000071c279d050 pitch=2368
CAMERA: Dropping an initial frame
CAMERA: CB onImageAvailable
CAMERA: NumberOfPlanes: 3
CAMERA: frame plane 0 row stride: 2368
CAMERA: frame plane 0 pixel stride: 2368
CAMERA: frame plane 1 row stride: 2368
CAMERA: frame plane 1 pixel stride: 2368
CAMERA: frame plane 2 row stride: 2368
CAMERA: frame plane 2 pixel stride: 2368
CAMERA: New frame available! pixels=0xb4000071c218b050 pitch=2368
CAMERA: Frame is going through without conversion!
CAMERA: CB onImageAvailable
CAMERA: NumberOfPlanes: 3
CAMERA: frame plane 0 row stride: 2368
CAMERA: frame plane 0 pixel stride: 2368
CAMERA: frame plane 1 row stride: 2368
CAMERA: frame plane 1 pixel stride: 2368
CAMERA: frame plane 2 row stride: 2368
CAMERA: frame plane 2 pixel stride: 2368
CAMERA: New frame available! pixels=0xb4000071c002e050 pitch=2368
CAMERA: Frame is going through without conversion!

...
1bsyl commented 1 month ago

@Green-Sky thanks ! I hope this can help could pixel stride be wrong :/ ? (copy paste of row stride ?)

Green-Sky commented 1 month ago

@Green-Sky thanks ! I hope this can help could pixel stride be wrong :/ ? (copy paste of row stride ?)

Oh yea, sorry, my bad, copy paste error :) Re building now and then retesting later... ( for the future: https://github.com/Green-Sky/SDL/pull/1/files )

Green-Sky commented 1 month ago 
CAMERA: frame plane 0 row stride: 2368
CAMERA: frame plane 0 pixel stride: 1
CAMERA: frame plane 1 row stride: 2368
CAMERA: frame plane 1 pixel stride: 2
CAMERA: frame plane 2 row stride: 2368
CAMERA: frame plane 2 pixel stride: 2

there

Green-Sky commented 1 month ago

So the problem is the row stride. The camera reported width and thus also the SDL_Surface's is 2328. However the row stride of the plane data is 2368. I suspect SDL can't properly handle surfaces with multiple planes and stride != width .

slouken commented 1 month ago

So the problem is the row stride. The camera reported width and thus also the SDL_Surface's is 2328. However the row stride of the plane data is 2368. I suspect SDL can't properly handle surfaces with multiple planes and stride != width .

It should be able to, as long as the Y stride and UV stride match. How are you displaying the image?

Green-Sky commented 1 month ago

the camera test uses SDL_UpdateTexture(texture, NULL, frame_current->pixels, frame_current->pitch);

I first use surface duplicate, then I SDL_LockTexture(), memcpy() and SDL_UnlockTexture()

I create the texture like usual, but the first image is uploaded using SDL_UpdateNVTexture(), with hardcoded stride, so I guess this is wrong.

The camera test creates the texture using this block: https://github.com/libsdl-org/SDL/blob/40070d899d551f4602c641b34fa0cbd1937cbf15/test/testcamera.c#L315-L324

Green-Sky commented 1 month ago

This might be multiple bugs, since I had to use updateNV() to get an output on the opengl es renderer. But not sure.

1bsyl commented 1 month ago

this old pr shows the code I tested before: (dont look at the PR itself but at the old revision) https://github.com/libsdl-org/SDL/pull/8565/files#diff-9859fc8ca0ebc34d849490517ec49ba8f707b0f28e3b6a4034a5309eb146f38f

       if (frame->num_planes == 3) {
            /* plane 2 and 3 are interleaved NV12. SDL only takes two planes for this format */
            int pixelStride = 0;
            AImage_getPlanePixelStride(image, 1, &pixelStride);
            if (pixelStride == 2) {
                frame->num_planes -= 1;
            }
        }

but that's already in. current code should reduce the plane number to 2.

    for (i = 0; i < numPlanes && i < 3; i++) {
            int dataLength = 0;
            int rowStride = 0;
            uint8_t *data = NULL;
            frame->num_planes += 1;
            AImage_getPlaneRowStride(image, i, &rowStride);
            res = AImage_getPlaneData(image, i, &data, &dataLength);
            if (res == AMEDIA_OK) {
                frame->data[i] = data;
                frame->pitch[i] = rowStride;
            }
        }

then either use the current row stride, or memcpy the data and use smaller stride

you can SDL_ConvertSurface() from NV12 to RBG and see how it's displaying

1bsyl commented 1 month ago

to make sure, here https://github.com/libsdl-org/SDL/blob/main/src/camera/android/SDL_camera_android.c#L327 num_planes, should go from 3 to 2

1bsyl commented 1 month ago

here's that probably incorrect: https://github.com/libsdl-org/SDL/blob/main/src/camera/android/SDL_camera_android.c#L353

we should pack the plane and use the smallest stride

edit: no, that's should be ok in fact, SDL should be able to handle width != stride as long as this is coherent between planes.

slouken commented 1 month ago

SDL should be able to handle width != stride as long as this is coherent between planes.

Yep, that's correct.

Is frame_current->pitch == 2368?