Buggy .makeImageSnapshot() on IOS

[sravis02]

Description

When I want to snapshot my Skia canvas (which is not that small), it gets buggy. These bugs don't occur always. If you try it multiple times, you will see that sometimes it works fine.

Screenrecording of the bug: https://www.youtube.com/shorts/mH7VuiT7CuU

What bugs?:

• For a short period of time, the elements of my canvas get transformed in a weird way
• Outputted image doesn't contain all items of canvas sometimes

Does RNSkia somehow rerender the Canvas, when the method makeImageSnapshot gets executed? If yes, I think, the snapshot is taken before the rerender process is completely finished.

It makes no sense to me, since the only code that gets triggered on the button click is this one:

const handleShare = async () => {
    const image = canvasRef.current?.makeImageSnapshot();

    if (image) {
      const base64Image = image.encodeToBase64();
      const options = {
        mimeType: 'image/jpeg',
        dialogTitle: 'Share the image',
        UTI: 'public.jpeg',
      };
      const fileUri = FileSystem.documentDirectory + 'test.jpeg'; // todo ts name name
      await FileSystem.writeAsStringAsync(fileUri, base64Image, {
        encoding: FileSystem.EncodingType.Base64,
      });

      Sharing.shareAsync(fileUri, options)
        .then((data: any) => {
          console.log('SHARED');
        })
        .catch((err: any) => {
          console.log(JSON.stringify(err));
        });
    }
  };

Version

0.1.197 in repro but also appears in 0.1.202

Steps to reproduce

How to reproduce:

npm install

On Mac simulator npx expo run:ios

On Iphone with EAS eas build --profile development --platform ios npx expo start --dev-client

In the App

1) Select background image from gallery

2) Select overlaying design image from gallery

Snack, code example, screenshot, or link to a repository

I couldn't really reproduce this in any test projects sadly. This only happened in my big project, i guess its because it has more heavy operations.

https://github.com/sravis02/flashes-app Branch: issue-react-native-skia I invited you both (chrfalch & wcandillion) to my repository

The Canvas is in PreviewCanvas.tsx The Snapshotfunction in preview.tsx

[grillorafael]

I'm having the same issue. I checked the file and it is always empty

[wcandillon]

I'm struggling to understand the issue based on the video, could you walk me thought it? We know have a test suite for this feature any tests you would like to me add there I can add.

[ludwighen]

Same issue but I believe it only happens in dev builds. Does the issue happen in a production build for you @sravis02?

What happens for me: I have a canvas with various elements. When I call makeImageSnapshot in the local dev builds, the resulting image sometimes doesn't contain all my elements from the canvas. It quickly blinks (like in your screen recording) and it seems as the elements are scaled down and on the top left corner falsely positioned.

In production I cannot reproduce this so far...

[sravis02]

I'm struggling to understand the issue based on the video, could you walk me thought it? We know have a test suite for this feature any tests you would like to me add there I can add.

thats exactly the issue i am having. it occurs on expo development builds, expo prebuild builds and expo internal preview builds for me.

[sravis02]

I'm struggling to understand the issue based on the video, could you walk me thought it? We know have a test suite for this feature any tests you would like to me add there I can add.

i will take a look into it again and try to provide some minimal reproduceable tests

[cesargdm]

I've encountered the same issue. When I call the makeImageSnapshot sometimes there's a small blink in the canvas (image of the frame attached).

https://github.com/Shopify/react-native-skia/assets/10179494/434ea7e5-1c2a-4f18-8d65-ad125a38fa20

This is the code I'm using:

export default function ImagePreviewScreen() {
    const navigation = useNavigation()
    const route = useRoute<any>()
    const { t } = useTranslation()
    const { uri, serviceId, isFinal } = route.params ?? {}

    console.log('ImagePreviewScreen')

    const path = useSharedValue<string>('')

    const [isTransitioning, setIsTransitioning] = useState(true)
    const [initialCanvasSize, setInitialCanvasSize] = useState(null as Dimensions)
    const [currentCanvasSize, setCurrentCanvasSize] = useState(null as Dimensions)
    const [hasPath, setHasPath] = useState(false)
    const [isSaving, setIsSaving] = useState(false)

    useAnimatedReaction(
        () => Boolean(path.value),
        (currentValue, previousValue) => {
            if (currentValue !== previousValue) {
                runOnJS(setHasPath)(currentValue)
            }
        },
    )

    const canvasRef = useCanvasRef()

    const image = useImage(uri)

    const handleUndo = useCallback(() => {
        // Remove last line remove until last M
        path.value = path.value.replace(/ ?M[^M]*$/, '')
    }, [path])

    const handleLayout = useCallback(
        (event: LayoutChangeEvent) => {
            const { width, height } = event.nativeEvent.layout

            if (!initialCanvasSize) {
                setInitialCanvasSize({ width, height })
            }

            setCurrentCanvasSize({ width, height })
        },
        [initialCanvasSize],
    )

    const srcRect = useImageRect({ canvas: initialCanvasSize, uri })
    const dstRct = useImageRect({ canvas: currentCanvasSize, uri })

    const handleSave = useCallback(async () => {
        console.log('handleSave')

        setIsSaving(true)

        // Simply save image if there are no annotations
        if (!hasPath) {
            addServiceAttachment(uri, serviceId)
        } else {
            try {
                console.log('Make image snapshot')
                const canvas = canvasRef.current
                if (!canvas) return

                console.log(canvas.state)

                // TODO: use original image size
                // eslint-disable-next-line @typescript-eslint/no-non-null-assertion
                const snapshot = canvas.makeImageSnapshot(dstRct!)

                const base64Image = snapshot.encodeToBase64(ImageFormat.PNG)

                console.log('Image snapshot done')

                if (!base64Image) return

                const fileName =
                    `${serviceId}-${Date.now()}.${SIGNATURE_EXTENSION}` as const

                const imageUri = documentDirectory + fileName

                console.log('Write image to file')
                await writeAsStringAsync(
                    imageUri,
                    base64Image.replace('data:image/png;base64,', ''),
                    { encoding: 'base64' },
                )

                snapshot.dispose()

                console.log('Add attachment to service')

                addServiceAttachment(imageUri, serviceId)
            } catch {
                // TODO: show error
            } finally {
                setIsSaving(false)
            }
        }

        navigation.goBack()
    }, [canvasRef, dstRct, hasPath, navigation, serviceId, uri])

    useLayoutEffect(() => {
        navigation.addListener('transitionEnd' as any, () => {
            setIsTransitioning(false)
        })

        navigation.setOptions({
            headerTintColor: 'white',
            headerStyle: { backgroundColor: 'black', color: 'white' },
            headerLeft() {
                return (
                    <Button
                        $variant="secondary"
                        onPress={() => navigation.goBack()}
                        $small
                    >
                        {t('cancel')}
                    </Button>
                )
            },
            headerRight() {
                return isFinal ? null : (
                    <Button
                        disabled={isSaving}
                        $variant="primary"
                        onPress={handleSave}
                        $small
                    >
                        {t('screens.image-preview.save')}
                    </Button>
                )
            },
        })
    }, [handleSave, handleUndo, isFinal, isSaving, navigation, path.value, t])

    const panGesture = Gesture.Pan()
        .onStart((event) => {
            if (!srcRect || !dstRct) return

            let { x, y } = event

            if (
                JSON.stringify(initialCanvasSize) !== JSON.stringify(currentCanvasSize)
            ) {
                const [{ translateX }, { translateY }, { scaleX }, { scaleY }] =
                    rect2rect(srcRect, dstRct)

                x = (x - translateX) / scaleX
                y = (y - translateY) / scaleY
            }

            path.value = `${path.value} M ${x},${y}`
        })
        .onUpdate((event) => {
            if (!srcRect || !dstRct) return

            let { x, y } = event

            if (
                JSON.stringify(initialCanvasSize) !== JSON.stringify(currentCanvasSize)
            ) {
                const [{ translateX }, { translateY }, { scaleX }, { scaleY }] =
                    rect2rect(srcRect, dstRct)

                x = (x - translateX) / scaleX
                y = (y - translateY) / scaleY
            }

            path.value = `${path.value} L ${x},${y}`
        })

    const tapGesture = Gesture.Tap()
        .runOnJS(true)
        .onStart(() => {
            Keyboard.dismiss()
        })

    const composed = Gesture.Race(tapGesture, panGesture)

    return (
        <SafeAreaView edges={SAFE_AREA_EDGES} style={styles.container}>
            <EXStatusBar style="light" />
            {!isFinal && (
                <View style={styles.actionsContainer}>
                    <Button
                        disabled={!hasPath}
                        $variant="secondary"
                        onPress={handleUndo}
                        $circle
                    >
                        <Button.Icon as={Ionicons} name="return-up-back-outline" />
                    </Button>
                    <View />
                </View>
            )}
            <KeyboardAvoidingView
                style={styles.container}
                behavior="padding"
                keyboardVerticalOffset={125}
            >
                <GestureDetector gesture={composed}>
                    {isTransitioning ? (
                        <View style={styles.empty} />
                    ) : (
                        <Canvas
                            ref={canvasRef}
                            onLayout={handleLayout}
                            mode="continuous"
                            style={styles.canvas}
                        >
                            {dstRct ? (
                                <Group clip={dstRct}>
                                    <Image
                                        image={image}
                                        x={dstRct.x}
                                        y={dstRct.y}
                                        width={dstRct.width}
                                        height={dstRct.height}
                                    />

                                    {srcRect && dstRct ? (
                                        <FitBox src={srcRect} fit="fill" dst={dstRct}>
                                            <Path
                                                strokeCap="round"
                                                path={path}
                                                strokeWidth={5}
                                                style="stroke"
                                                color="red"
                                            />
                                        </FitBox>
                                    ) : null}
                                </Group>
                            ) : null}
                        </Canvas>
                    )}
                </GestureDetector>

                <View style={styles.textInputContainer}>
                    <TextInput
                        placeholderTextColor="gray"
                        style={{ color: 'white' }}
                        $color="white"
                        placeholder={
                            isFinal ? '' : t('screens.image-preview.description.placeholder')
                        }
                        readOnly={isFinal}
                        hideError
                        hideBorder
                        multiline
                    />
                </View>
            </KeyboardAvoidingView>
        </SafeAreaView>
    )
}

[cesargdm]

This is very similar to the issues @ludwighen had

[cesargdm]

Probably related to https://github.com/Shopify/react-native-skia/issues/2133

[ludwighen]

Following up on this, I think I found a workaround for the issue @cesargdm.

Indeed I have the same issue and I first thought it would only happen in my development builds but it also happened in production, only for less powerful devices. It was probably visible in the development builds because they are a bit less performant, and something obviously tries to render the moment the snapshot is taken. I assume in most attempts it's fast enough to take the image after this drawing process.

I figured what if I just delay the makeImageScreenshot by one render cycle using setTimeout. And it works. I have no idea why 😅, but with over 100 attempts, I now can no longer reproduce the issue, so I believe it fixed it. Let me know if it works for you too.

Here's my code:

  const onSavePress = useCallback(async () => {
   //... some other stuff

    setTimeout(() => {
      const image = ref.current?.makeImageSnapshot();
      if (!image) {
        // handle error
        return;
      }

      setSkiaImage(image);
    }, 0);
  }, [ref]);

[sravis02]

thanks a lot, this workaround works for me :) @ludwighen

[ludwighen]

Update: Unfortunately the hack doesn't work fully reliable after all.. It causes the issue to disappear when I have just a handful of elements in the canvas. But when I add say 20 images to the canvas, I still get the glitching.

Feels to me like some sort of race condition where the snapshot happens just before the drawing on the canvas is finished?

[wcandillon]

this bug looks interesting and I will investigate it further but in the meantime, I recommend to draw the content offscreen and take a snapshot of that: https://shopify.github.io/react-native-skia/docs/animations/textures#usetexture

[ludwighen]

Thanks @wcandillon, indeed that's what I planned to do next... I'll keep you posted

[wcandillon]

Gang, I think that this PR may fix the issue: https://github.com/Shopify/react-native-skia/pull/2254, any chance you could try this patch on your app?

[ludwighen]

nice! Probably dumb question but do I need to build skia locally to do that? If I install from your branch it get's added as react-native-skia-dev-tools.

[wcandillon]

that's a fair question, I was asking just in case, we will publish this bug fix soon so you will be able to test on the publish version. But now I am realizing that these a not related most likely. I will try to investigate this particular issue still

On Mon, Feb 26, 2024 at 10:18 PM ludwighen @.***> wrote:

nice! Probably dumb question but do I need to build skia locally to do that? If I install from your branch it get's added as react-native-skia-dev-tools.

— Reply to this email directly, view it on GitHub or unsubscribe. You are receiving this email because you commented on the thread.

Triage notifications on the go with GitHub Mobile for iOS or Android.

[ludwighen]

So I tested your fix with 0.1.241 beta but the issue is still there.

Next I tried the useTexture approach but I cannot get the basic example working. It throws an error: Value is undefined, expected an Object at root.render(element); in Offscreen.tsx.

  const texture = useTexture(
      <Fill color="cyan" />,
    { width, height }
  );

Any idea why?

While looking into all of this, I was wondering if my current approach is even the "right" way, I think I should switch to the imperative API. I know it's not the right thread for it, but given that it would likely mean a significant refactor for me, maybe you could point in me the right direction since the documentation is a bit limited for the imperative API 🥲.

Basically, I have a canvas that can contain lots of different elements from the user (images, svgs, paths, etc.), think something like the Instagram stories editor / stickers example. I use the declarative API like this:

  <Canvas>
      {elements.map((element, index) => (
        <CanvasElement
          gestureInfo={gestureInfo[element.uid]}
          key={element.uid}
          element={element}
          canvasWidth={canvasWidth}
          canvasHeight={canvasHeight}
          clock={clock}
          color={element.color}
        />
      ))}
 </Canvas>

And then essentially the CanvasElement conditionally renders Skia images, svgs, paths, etc. and applies transforms from GestureHandlers.

The useTexture hook would not allow me to pass the same logic in, would it? Should I instead create all of the conditional drawing with the imperative API and just display the results on screen? Also performance wise, is there a difference between declarative vs imperative? I can have up to 100 elements/images drawn on the canvas in my case. Any pointers would be welcome! I also see quite a few people out there that use Skia for a creative "editor" with a preview and an export stage, it could be cool to have resources on how to handle offscreen vs onscreen.

Anyways, thanks a ton @wcandillon as always 🙏

[ludwighen]

Update: I finally was able to solve it (this time reliably @sravis02 @cesargdm).

Not using the canvas ref's makeImageSnapshot() but instead drawing offscreen indeed fixes the problem and it also revealed why the items glitched small in the top left corner. It has something to do with the scale/pixel density.

What I did is just putting the Canvas children 1 to 1 but it has the caveat that the sizing is not identical to the onscreen canvas:

import { drawAsImage } from '@shopify/react-native-skia';

const { scale } = useWindowDimensions();

const scaledHeight = canvasHeight * scale;
const scaledWidth = canvasWidth * scale;

const image = drawAsImage(
      <>
        <Rect
          x={0}
          y={0}
          height={scaledHeight}
          width={scaledWidth}
          color={color}
          }
        />
        {elements.map((element, index) => (
          <CanvasElement
            gestureInfo={gestureInfo[element.uid]}
            //... I also needed to adjust the scale of width/height of my elements here
         />
        ))}
      </>,
      {
        width: scaledWidth,
        height: scaledHeight,
      },
    );

 const base64 = image.encodeToBase64();

So basically, when I use the normal canvasWidth/canvasHeight (via height/width from useWindowDimensions), the offscreen canvas is very small because it uses them as absolute pixel values whereas with the declarative API the pixels are scaled according to device (i.e. 3x for iPhones).

So the little glitch we see when using the ref's makeImageSnapshot method is basically the non-upscaled dimensions. Does this somehow help you in debugging @wcandillon ?

Hope that helps

[wcandillon]

Thank you for this. I filed a seperate bug report for the offscreen scaling: https://github.com/Shopify/react-native-skia/issues/2336 This is gonna sound silly but I knew that of course these need to be scaled to pixel density but couldn't build an example where I would notice the difference: https://github.com/Shopify/react-native-skia/blob/main/package/src/renderer/Offscreen.tsx#L35

But I will definitely revisit.

For makeImageSnapshot, we know offer makeImageSnapshotAsync that runs on the UI thread so it shares the same Skia context than your onscreen canvas: https://shopify.github.io/react-native-skia/docs/canvas/overview/#getting-a-canvas-snapshot