Something goes wrong on color conversion

[60-hz]

On the last deken, something goes wrong on the color under MacOS

pix_image-help output:

pix_video-help output

MacOS GEM: ver: 0.94.git v0.94-742-gd8e09cc96

[umlaeute]

i know that i've revamped the color-conversion code last night, so the issues are not entirely unexpected.

however, i cannot reproduce them here at all (that is: i can load images and videos on BigSur/amd64 and Linux/amd64 just fine).

this is on the M1, right?

[umlaeute]

could you try again with d7e61f4c5041354b95e906fb6b2a5db1d97ba582? (currently building, so the deken uploads should appear in a few hours).

it will not fix the problem, but should have enabled some additional debugging to see which color converters are enabled (at "debug" verbosity)

the debugging is only enabled for the macOS deken builds; if you compile yourself you should add -DGEM_DEBUG_PIXCONVERT=1 to preprocessor flags.

[umlaeute]

(un)fortunately i can now confirm the problem with [pix_image] on my Monterey box (my other test was on BigSur)

OS	image format ([pix_info])	converter	status
Monterey/arm64	BGRA/UINT8888	ARGBto*	:red_square:
Monterey/x86_64	BGRA/UINT8888	ARGBto*	:red_square:
BigSur/x86_64	BGRA/UINT8888	ARGBto*	:tada:
Linux/x86_64	RGBA/UBYTE	RGBAto*	:tada:

seems like the Red and the Alpha channel got swapped

[60-hz]

this is on the M1, right?

Yes, Sonoma 14.0

I tried with last deken g2d6ace4dc, but debug is not activated with [pix_convert]

[umlaeute]

thanks for checking.

i do believe that debug is activated, but it since [pix_image] already outputs and "RGBA" image, no conversion is happening. (converting to "YUV" should give some printout though).

anyhow, this needs a bit more thinking on my side :-)

[umlaeute]

hmm, with [pix_image] the problem seems to be the imageSTB loader only. disabling the plugin (that is: moving gem_imageSTB.so out of the way) uses the imageIO loader which appears to work fine.

this doesn't explain why [pix_video] is broken though... (unfortunately i have problems testing [pix_video], as my Ventura machine currently has no webcam)

[umlaeute]

here's an idea: if you can compile Gem yourself, could you try changing https://github.com/umlaeute/Gem/blob/2d6ace4dca677b4779162e1fbbe0dd91dc330354/plugins/videoAVF/AVFVideoGrabber.mm#L366-L371 to read

 case GEM_RAW_BGRA: 
    BGRAtoBGRA(isrc4, pixes.image.data, width, height); // GEM_RAW_BGRA 
    break; 
 case GEM_RAW_RGBA: 
    BGRAtoRGBA(isrc4, pixes.image.data, width, height); // GEM_RAW_RGBA 
    break; 

instead? (that is: the BGRAto... target is reversed to what the comment actually says (ARGB->BGRA resp. ABGR->RGBA)

[60-hz]

his doesn't explain why [pix_video] is broken though... (unfortunately i have problems testing [pix_video], as my Ventura machine currently has no webcam)

It turns out that pix_video was working. The issue was on my side sorry...

here's an idea: if you can compile Gem yourself, could you try changing

Done. The color issue of pix_image is still there. And it make pix_video looks bluish this time.

[umlaeute]

hmm, i finally found a half-working USB webcam and plugged it into our Mac Studio: the video looks good (without the above mentioned hack)

two observations:

• the video output you (@60-hz) get looks like utter garbage with some noise inbetween. so the problem is obviously deeper than a simple swapped channel (and my hack in my previous comment would not do anymore than swap some channels, so it cannot possibly be the correct fix)
• my webcam seems to prefer to output to an RGBA-like format anyhow (so the debugging shows plenty of BGRAtoARGB calls); other webcams might prefer a YUV-like format, which would trigger a different codepath.

i currently do not have much ideas on how to debug this (short of adding loads of print-statements to the various stages of the processing).

in any case:

• which camera (and which backend) do you use?
• does the camera still work correctly in other applications? (sorry for asking the obvious)
• the [pix_video] help patch should really display some debugging lines.
  • no need to change anything in the patch
  • however, you must start Pd with the -verbose flag
  • and you must raise the log-level to "4 - Debug" in the Pd-console (there's still a small chance that the color conversion code is skipped, in which case it will not show anything)

[60-hz]

Sorry if I was not clear in the last post, [pix_video] works ok, so the gargabe on screen was a temporary weird issue on my side that I cannot reproduce now, so all is ok.

After that:

• I made the modification in Gem/plugins/videoAVF/AVFVideoGrabber.mm

• Then re-compiled Gem: the color issue in [pix_image] remains the same and it also added a color swap in [pix_video] which becomes blue.

[umlaeute]

ah sorry.

• the suggested code change in AVFVideoGrabber.mm only effects the [pix_video] and was obviously wrong (as you discovered), so just revert it :-)
• I've pushed a fix for [pix_image] in 551e7b2dc , which will be available via deken once the Windows build finished (in about 1½ hours from now... my builder is just too slow...)

[umlaeute]

i've now tested 551e7b2dc6c9b70c4799f03cc6be99d6ad120703 on Monterey (VM), Ventura (metal) and Catalina (VM), and loading images seem to work correctly on all platforms.

I'm therefore closing this for now (if you still have troubles, we can re-open this ticket)

[60-hz]

Works also here under M1, Sonoma.