Working with matrices to accumulate transformations

[raffaeler]

Question

I have to draw a custom map supporting translations, rotations and zooming. The transforms of every gesture is always additive in respect of the previous gestures. From a math perspective this simply means multiplying the new transform and proceed to the next.

The transform style in React Native apparently does not support matrices (and this is very surprising to me). How can I work with matrices in react-native-svg in order to accumulate the transformations?

Thank you!

[msand]

Perhaps this can help? https://twitter.com/wcandillon/status/1244988861635268608 https://www.npmjs.com/package/zoomable-svg

[msand]

And yeah, you can use matrices with transforms in react-native-svg, and matrix multiplication follows the normal linear algebra, so just compute what ever product of matrices you want, and give it as a transform to e.g. a G element.

[raffaeler]

I already watched the video last week and also tried to use your zoomable-svg but the problem is that I need to accumulate all the transformations: scaling, rotation and translation. The video shows that the transform is reset at the end of the gesture: definitely too easy because it never accumulates matrices.

How can I specify a matrix?

• transformMatrix has been deprecated according to the docs
• the transform: {[ matrix: [...] is apparently not working

[msand]

Same way as in any other svg https://developer.mozilla.org/en-US/docs/Web/SVG/Attribute/transform#Matrix

  <rect x="10" y="10" width="30" height="20" fill="red"
        transform="matrix(3 1 -1 3 30 40)" />

I don't see the problem, you know how to multiply two matrices no? https://en.wikipedia.org/wiki/Matrix_multiplication#Definition

[raffaeler]

Of course I know. I did not find the transform="matrix(...)" syntax in the docs. Is this supported for react-native-svg only or is it standard for all the react-native based libraries?

I would prefer to apply the transforms to the element whose child is the SVG. Is this a potential performance hit?

[msand]

This only works on react-native-svg, as it comes from the svg spec. Would have to do profiling to check performance. Quite likely, natively animated/computed logic would be more performant than a plain js version.

[msand]

and style={{transform: [{matrix: something}]}} should certainly work in both react-native and react-native-svg

[raffaeler]

Thanks. I am probably going to use animation only to smooth the movements, but I only need to apply the transforms to change the map view.

and style={{transform: [{matrix: something}]}} should certainly work in both react-native and react-native-svg

I was not successful in using it on a simple View element filled with a color. Will retry ...

Thank you!

[msand]

There's also this happy chap i tried to help once, might be useful for you as well: https://github.com/react-native-community/react-native-svg/issues/1064

[msand]

But, when I think about it, seems you don't need to accumulate any state at all, you just need to keep track of where the pointers were when the number of active gestures change, if it changes from 0 or 2 to one pointer, keep track of where it was when that happened, and on new gesture events, the distance of the pointer to that point is your translate transform. If it changes from 0 or 1 to 2, store the position of the two pointers, on new events, the change in distance between the two new positions and two initial ones, is your scale transform, the change in angle between them is your rotation, and the distance between the midpoints of the two pairs is your translation. Gestures / states in between should not have any effect / non-linear accumulation affecting the outcome, transform only depends on current pointer data and when the number of active pointers changed.

[raffaeler]

hmm, let's make an example:

• you pinch the map, and it zooms at a specific point (the center of the pinch)
• than you rotate it, with a different pinch point
• now you pan it If you now want to zoom or rotate, you need to accumulate those transforms over the previous ones. The transforms style (without the matrix) just apply a fixed sequence of translation, rotation, scale. Either you keep the matrix and then try to extract the translation, rotation, scale from that matrix, or you will never be able to keep the gestures correctly. Since I don't skew, the matrix should be reversable, but never did it as keeping the matrix state is far simpler. Do you agree or am I losing something? I always worked this way in other technologies and it worked.

[msand]

Ah, yes, when the gesture ends you should accumulate the transform for sure, was just thinking about individual gestures on top of the current state. So the initial transform is the identity, lets call it A, and then you multiply a transform that has the translate, scale, rotate and, offset needed for the just finished gesture, B = TSRO (translate, scale, rotate, offset), and make C = BA the new accumulated transform

[raffaeler]

Exactly. The video you linked is about Instagram where the photo goes back to the original position after the gesture... that's too easy and I already can do it. When you go to google maps instead, you accumulate the transformations, so a state is needed (starting, as you said in your last message from the identity matrix) by multiplying every transition (I typically order scale, then rotation and lastly translation). BTW every time you pinch, you either have a primitive centering the rotation/zoom, otherwise you have to manually translate + rotate/zoom + translate back.

FYI I tried using reanimate and discovered (sadly after two days) they do not support matrices, so I can't work with their library.

[msand]

Alternatively you can have four different transforms, one for each primitive, and accumulate those, will need to consider interactions between translations and scale-rotate in the gestures a bit more carefully then.

[raffaeler]

Since I don't need to skew, they should be reversable. In this case I could maintain a matrix and then "extract" the separate t,r,s,o parameters. Never did that but in theory it should be possible.

[msand]

But yeah, to clarify the native aspect, if you add animations, it'll probably just feel more disconnected from the gesture, you want to minimize the number of cycles from the gesture event being registered, to the final rendered output being visible on the screen. So you probably want to use react-native-gesture-handler, as that allows the processing to stay completely native, rather than doing a context switch to javascript, run event handler, change state, run react lifecycle, commit changes back to native and only then start rendering, rather than just computing the matrix transform and invalidating one View or Svg / G element... If it doesn't have the matrix support you seek, I recommend forking it and implementing the support yourself and making a pull request there. I can probably help on the way in case there's any questions on the native side.

[msand]

This one use-case would probably deserve its own tailor-made performance / use-case optimized package, something like react-native-pan-zoom-rotate / react-native-zoomable perhaps, pull requests to zoomable-svg for a native mode would be welcome as well.

[msand]

And yeah, if you have the transform as one matrix, you don't need to split it up, just start with the initial A and B matrix as identity, when the number of active pointers change, store the positions, for each event update B (or update the decomposed matrices / primitive transformations and multiply them together to get B), and when the number of active pointers change, accumulate B into A, and set B to identity again

[msand]

So the structure would be something like this:

<Svg style={{transform: [{matrix: B}, {matrix: A}]}}>
  <Text>some content goes here</Text>
</Svg>

Or, equivalently

<Svg>
  <G style={{transform: [{matrix: B}]}}>
    <G style={{transform: [{matrix: A}]}}>
      <Text>some content goes here</Text>
    </G>
  </G>
</Svg>

[msand]

I assume you're familiar with these, but just in case you want a refresher (they're also in the css / svg specs), here's the equivalent matrices to the primitive transforms: https://github.com/react-native-community/react-native-svg/blob/ffa2e69c17ce02b21f393a5b57cdbef1c039fe3d/src/lib/extract/transform.peg#L1-L105

And the main state changes that would need to be implemented in native logic instead: https://github.com/msand/zoomable-svg/blob/fe724c2652595bb6176731be96fde1151e30f21a/index.js#L420-L511

[msand]

Also, the calculation for the rotation is missing there, so something like this:

const initialAngle = Math.atan2(initial_y1 - initial_y2, initial_x1 - initial_x2);
const rotate = Math.atan2(y1 - y2, x1 - x2) - initialAngle;

[raffaeler]

But yeah, to clarify the native aspect, if you add animations, it'll probably just feel more disconnected from the gesture, you want to minimize the number of cycles from the gesture event being registered

My wish is to use animation only to restore a position after the user makes a search. Probably it doesn't make sense (in my scenario) to use animation while the user is actively making a gesture.

I understood how the react-native-gesture-handler library works and it is a great idea (creating the AST for the desired transformations and generate the native code that uses the refs under the hood), but I don't know if I will have time to implement the fundamental matrix support.

[raffaeler]

perhaps, pull requests to zoomable-svg for a native mode would be welcome as well.

As soon as I come to a solution, I will be more than glad to either publish it or making a pull-request

[msand]

Also, maybe these two can be useful for learning more about react-native transforms: https://snack.expo.io/@msand/new-instagram-stories https://snack.expo.io/@msand/rotate-cube

The instagram stories one has three alternative implementations, Stories2 requires a fork of react-native-reanimated I made https://github.com/software-mansion/react-native-reanimated/pull/538

[msand]

Also, Stories1 in new-instagram-stories has a source code parameter / constant called "alt" with a bit different transformation

[raffaeler]

Talking about the transform using matrix over standard react-native elements and the Svg:

• style={{transformMatrix: [1, 0, 50, 0, 1, 50]}} This doesn't work
• style={{ transform: [ { matrix: [1, 0, 50, 0, 1, 50] } ] }} on a react-native element throws with "Error updating property 'transform' of a view managed by: RCTView"
• style={{ transform: [ { matrix: [1, 0, 50, 0, 1, 50] } ] }} on Svg throws with "Invariant Violation: Matrix transform must have a length of 9 (2d) or 16 (3d)..."
• style={{ transform: [ { matrix: [1, 0, 50, 0, 1, 40, 0, 0, 1] } ] }} on a reac-native element or also Svg throws with "Error updating property 'transform' of a view managed by: RCTView"

I didn't find even a single example showing how to use the matrix on react-native ... astonishing.

Of course I am able to use the SVG notation on inner elements of the Svg. The following works:

<Rect x="0" y="0" width="100" height="100" fill="red"  transform="matrix(1 0 0 1 50 50)" />

What is the syntax for matrix? Do you know any example of that?

P.S. I am still reading/working on the other posts you wrote.

Thank you

[msand]

The error comes from here: https://github.com/facebook/react-native/blob/0b9ea60b4fee8cacc36e7160e31b91fc114dbc0d/Libraries/StyleSheet/processTransform.js#L172-L182

There's some useful helpers in that file as well: https://github.com/facebook/react-native/blob/0b9ea60b4fee8cacc36e7160e31b91fc114dbc0d/Libraries/StyleSheet/processTransform.js#L19-L114

So, to use the normal react-native transform style property (with a list of transforms containing matrices), you need to give the matrices as arrays with either 9 (2d) or 16 (3d) numbers, i.e.

  style={{
    transform: [
      { translateX: tx },
      { translateY: ty },
      { scale: s },
      { rotate: r },
      { translateX: ox },
      { translateY: oy },
      { matrix: [ // 9 numbers doesn't seem to work
          1, 0, 0,
          0, 1, 0,
          0, 0, 1
        ]
      },
      { matrix: [ // seems to work
          1, 0, 0, 0,
          0, 1, 0, 0,
          0, 0, 1, 0,
          0, 0, 0, 1
        ]
      }
    ]
  }}

. For the svg standard syntax, you need to give a transform attribute as string instead (n.b. not a style property, but directly on the element instead, although we support it in the style props as well for simplicity, it's not required by the spec), i.e. transform="matrix(a b c d e f)" i.e. 6 numbers inside the parenthesis, or any sequence of svg transform primitives as a string.

Another supported syntax in react-native-svg elements is: giving an array of 6 numbers i.e. transform={[a, c, e, b, d, f]} as the transform attribute / style property, the same as the output of the svg transform string parser referred to in an earlier comment, instead of an array of react-native transform objects.

[raffaeler]

The only transform/matrix I didn't test was the 16 elements ... and of course it worked. But (as I posted before) the one with 9 elements does not work and this is what cheated me. Thank you

[msand]

Oh, that's quite possible, not sure why, would have to set breakpoint in both the javascript, java and objective-c code of react-native and react-native-svg to double check. Might be that only the 16 works properly with the react-native syntax, to fit together with the other 3d transforms.

[msand]

With the tailor-made module, I'm thinking it wouldn't depend on anything but react-native (at most react-native-svg as well, i.e. no reanimated, no react-native-gesture-handler), and would be a single View, accepting only a single child, and its only concern would be to handle any pan-zoom-rotate gestures on that child in native code, calculate the needed matrix, and set the new transform either on itself or on the child, using the ViewManagers directly: https://github.com/facebook/react-native/blob/d0871d0a9a373e1d3ac35da46c85c0d0e793116d/React/Views/RCTViewManager.m#L169-L174

https://github.com/facebook/react-native/blob/f2d58483c2aec689d7065eb68766a5aec7c96e97/ReactAndroid/src/main/java/com/facebook/react/uimanager/BaseViewManager.java#L76-L84

https://github.com/react-native-community/react-native-svg/blob/ffa2e69c17ce02b21f393a5b57cdbef1c039fe3d/ios/ViewManagers/RNSVGNodeManager.m#L157-L164

https://github.com/react-native-community/react-native-svg/blob/ffa2e69c17ce02b21f393a5b57cdbef1c039fe3d/android/src/main/java/com/horcrux/svg/RenderableViewManager.java#L1211-L1225

And would be a simple wrapper:

import * as React from 'react';
import { View, Text } from 'react-native';
import PZR from 'react-native-pan-zoom-rotate';

export default () => (
  <PZR>
    <View>
      <Text>Gesture This</Text>
    </View>
  </PZR>
);

[msand]

@wcandillon Have you made anything matching this exact api?

[msand]

These can be used to create native modules:

create-react-native-module pan-zoom-rotate --view --generate-example --package-identifier io.seaber --github-account msand --author-name 'Mikael Sand' --author-email 'msand@abo.fi'

https://github.com/brodybits/create-react-native-module

and

npx @react-native-community/bob create react-native-pan-zoom-rotate

https://github.com/react-native-community/bob

bob comes preconfigured using kotlin for the native android code by default, while create-react-native-module uses plain old java

[msand]

Oh, and create-react-native-module has a basic native View example when using the --view flag

[wcandillon]

I haven't followed this thread closely, so far it doesn't look like there is a matrix transformation that you cannot express using the transform API. I'm currently working on a complex zooming example with both a pan and a pinch gesture and I'm not encountering any specific issues.

Let me know if you have any specific question regarding that topic. Until I release more content on this theme, these are the two videos that can serve as a basis: https://www.youtube.com/watch?v=0FVnzuyFNSE https://www.youtube.com/watch?v=FZnhzkXOT0c

[msand]

@wcandillon Ah, cool. You have anything combining pinch and rotate?

@raffaeler There's also decomposition logic already implemented: https://github.com/facebook/react-native/search?q=decomposeMatrix&unscoped_q=decomposeMatrix

js: https://github.com/facebook/react-native/blob/0b9ea60b4fee8cacc36e7160e31b91fc114dbc0d/Libraries/Utilities/MatrixMath.js#L560-L750

java: https://github.com/facebook/react-native/blob/aee88b6843cea63d6aa0b5879ad6ef9da4701846/ReactAndroid/src/main/java/com/facebook/react/uimanager/MatrixMathHelper.java#L101-L215

[wcandillon]

@msand not yet but if there is a cool example to implement, I'd love to do it. What might be the issue with rotate?

[msand]

Well, at least it seems a bit tricky to do well / concisely with reanimated / gesture-handler, I think to add it to zoomable-svg just need something like these two lines and another transform for rotate:

const initialAngle = Math.atan2(initial_y1 - initial_y2, initial_x1 - initial_x2);
const rotate = Math.atan2(y1 - y2, x1 - x2) - initialAngle;

[raffaeler]

Thanks @msand and @wcandillon for your interesting links and information and still digging. The issue I rised is about handling an Svg map. While the Instagram gestures video (which I watched last week, great indeed) shows how to apply transforms that are reset at the end of the gesture, managing a map is quite different. I am willing to create a general purpose viewer that is able to accumulate the gestures so that the user can continue moving, zooming and rotating the map after each gesture (kind of google maps but with Svg custom maps).

After familiarizing a bit with react-native-svg, I tried to get the best performance using reanimate, but I discovered they do not support matrix transformation. Then I asked in this thread which other possibilities I had.

BTW As a side note, during my tests I discovered that the translationX/Y of a pan operation obtained from the pan event is "delayed" (probably because Android needs some time to understand whether it is a pan or a pinch) and after realizing it is a pan, starts providing the translations but they are wrong). I could fix this by keeping trace of the initial point using the pinch values (when state is BEGAN) and then reculaculating manually translationX/Y and ignoring the ones provided by the PAN event.

Thanks!

[wcandillon]

@raffaeler not sure if it's related to you last mention, but this is how I detect if the pinch gesture began:

  const pinchBegan =
    Platform.OS === "ios"
      ? eq(pinchState, State.BEGAN)
      : eq(diff(pinchState), State.ACTIVE - State.BEGAN);

[raffaeler]

@wcandillon never worked on iOS, but I believe it is a different issue. The problem I am referring to is more evident when you are working on an Emulator since you use the mouse that is more precise. When you click on an pan-enabled object and start moving it, there is a small amount of time needed by Android to understand which kind of gesture is. If you just use translationX and translationY to compute the new position of the object, you will see it making a small jump at the beginning of the gesture. If instead you manually compute translationX and translationY by getting pinchX and pinchY in the pan event during the BEGAN state and then calculate their deltas when the pan is ACTIVE, you will get the correct movements.

In order to understand the problem I used to draw a small circle centered a the pinchX, pinchY.

[wcandillon]

Based on the description, it looks like we are referring about the same thing. You should also beware that the pinch gesture could work with one finger (on iOS at least) so you need to filter for that as well using numberOfPointer

[raffaeler]

@msand @wcandillon Precious information :) I love the @msand proposal to build a native control, but I have to go step by step and start from a pure react-native control first. I am still not that familiar to with the core stuff to know where to start from.

From the perf perspective, given I am not going to make heavy animations, transforming the view or the Svg will lead to different perf results?

[msand]

Yes, if you transform the view, it'll reuse the bitmap which contains the vector render output, and thus it'll pixelate when you zoom in, it you transform in the Svg element or a G element (wrapped with Animated.createAnimatedComponent), it'll re-render the svg content and stay pixel perfect. Thus you should either tile bitmaps the way that some map viewers do (i.e. one layer of e.g. 256x256 tiles per power of two of scaling), or have a single bitmap which is at most as large as the native display, and transform the content using a single transform / G element. Transforming a bitmap (or even quite a few) using a matrix is very cheap. Re-rendering a complete svg tree, very much depends on the content.

[msand]

I realised it only seems to get a bit more complex if one tries to keep both the A and B matrix decomposed / as separate primitive transforms, and even then, it just requires to decomposeMatrix(C), where C=BA, to get the new A when you want to set B to identity.

But if you don't decompose C (nor A), you can just substitute A with C and set all the decomposed / primitive transforms of B i.e. TSRO, to identity, whenever the type of gesture changes / ends, i.e. when the number of active touches / pointers change.

I.e. it gets a bit tricky from the fact that matrix multiplication in general (or even affine transforms more specifically) doesn't commute. So if you want to maintain a TSRO decomposition of both A and B, you need to take their product C, to do the change of basis due to two rotations with a translate (and a scale) in between, and decompose C to get a single rotation about a single scaled and translated origin, when you set the parts of B to identity / flatten the pan gesture offset.

[msand]

And yeah, the physical intuition / constraint on state change here I aimed for in zoomable-svg, is that when you go from zero to one active pointers, you expect the point you started touching to stay in contact with the pointer as you move, when you go to two active pointers, you expect both points to stay in contact with their active pointers (thus the need for rotation as well), and if you then go back to one pointer, you expect that to stay in contact (i.e. only translation), and go back to translation+scaling+rotation if you then go back to two active pointers.

[raffaeler]

@msand I am currently building a sample test without using reanimated or react-native-gesture-handle in order to work directly with the matrix transform.

But even with the help of decomposition primitives, I could not work with those libraries because I would have to rewrite the decomposition using their primitives since the values used by those libraries never come back to javascript.

Do you agree or am I missing something?

[msand]

I've barely touched react-native-gesture-handler, so I'm probably not the best person to ask about that. I think @wcandillon probably has several orders of magnitude more experience with that library than me, maybe he can answer this?

At least by forking zoomable-svg, or copying the code from index.js into your own project and modifying there, should allow using either the accumulated or decomposed approach, with plain react-native and react-native-svg. The PanResponder api should provide everything needed: https://github.com/msand/zoomable-svg/blob/fe724c2652595bb6176731be96fde1151e30f21a/index.js#L291-L328

[wcandillon]

@raffaeler I finally understand what you are trying to do, sorry that it took me so much time. I'm also trying to save the transformation matrix when the gesture end. First on the JS thread as a proof of concept (which I expect to be glitchy because there will be some latency between the time the gesture values are reset and the JS thread re-rendered the component).

Using offsets with animation values from gestures are convenient, they simulate what happens if the gesture would be continuous. Because of the pinch focal offset, the gesture cannot be treated as "continuous" anymore since it happens when the touch begin. We agree that the generic solution to this problem is simply to keep a transformation matrix in the state (ideally in the UI thread and have its decomposition done in the UI thread as well). In case that turns out to be too complicated/impossible, I would try to spend some time some ad-hoc solution to the pinch focal continuity. I will keep you posted if I make progress, keep me posted on your side as well

[msand]

I'd recommend to keep the accumulated state as a single matrix, and skip the decomposition of it. It's enough that the current delta to that is defined in decomposed form in the state, that makes it easy to calculate the final delta when a gesture ends / type changes, and then multiply that into the current accumulated matrix, and set the decomposed transforms to their identity elements. So e.g.

<Svg>
  <G style={{transform: [
      { translateX: tx },
      { translateY: ty },
      { translateX: ox },
      { translateY: oy },
      { scale: scale },
      { rotate: radians },
      { translateX: -ox },
      { translateY: -oy },
]}}>
    <G style={{transform: [{matrix: accumulatedMatrix}]}}>
      <Text>some content goes here</Text>
    </G>
  </G>
</Svg>

And set tx, ty, ox, oy, radians to zero, and s to one, when you multiply BA = Tx Ty Ox Oy S R Ox^-1 Oy^-1 accumulatedMatrix = C and set the new accumulatedMatrix to C.

[wcandillon]

That makes sense. Do you have any thoughts on how to calculate the accumulatedMatrix? I tried with setting it in the JS thread but that creates a tiny glitch between the end of the gesture and the time it takes to re-render the component.