search

csabau / BTracking

Other
0 stars 0 forks source link

Understanding the data #1

Open mhamilt opened 2 years ago

mhamilt commented 2 years ago

The 2D view is in the file: ARSUIView2D. The components for tracking the 2D body is found the the Sources folder, BodyTracker2D

You will see there is a RKPath file as well, this is another gitHub code I got to try to see whether it’s useful to implement it in this project as it creates lines between user’s taps on the screen, but I didn’t get to the stage where I understood what points I need on the canvas so you will see parts of the code commented out.

[ARSUIView2D - line 91]() This is where circles are attached to joints. I’ve tried to reverse engineer this to get to a joint’s (x,y), but I cannot figure out how. It uses the [.attach] function found for .bodyTracker which is essentially the BodyTracker2D file. The attach function seems to use the [.trackedViews] based on [TwoDBodyJoint] (which is the array containing the joint names), but I struggle to understand what data actually goes in .trackedViews . It is declared on line 38 in BodyTracker2D file and to me it looks like a dictionary of joint and view? [TwoDBodyJoint : UIView] - I’m really confused as to what that means. To try to understand it I’ve printed it, but it gets me even more confused as it prints

<UIView: 0x102fcd630; frame = (407 587; 20 20); layer = <CALayer: 0x2821900c0>>

…for the first and then

<UIView: 0x102feeac0; frame = (397 577; 40 40); layer = <CALayer: 0x2821912e0>>

…for the rest of the circles

[BodyTracker2D - line 95]() I’ve tried to draw lines using the CGPoints, thinking hat jointScreenPositions array would contain all the x,y values, but the code crashes at that point. My reasoning for this came from the function on line 207 angleFrom2Joints, which calls for a private function angleBetween2Points which uses CGPoints…although how that gets worked out its still a mystery to me. angleFrom2Joints takes in 2 TwoDBodyJoints which are part of the array for the joint names, then joint1Index gets the rawValue of that passed joint, which is a number between 0 and 16 as per the TwoDBodyJoints declaration on line 330 , and then that value is passed to joint1ScreenPosition based on the joint1Index from jointscreenPositions (array which has been populated at line 75). That value is then passed onto the private function of angleBetween2Points and used as a CGPoint. So I’m really struggling to understand if jointscreenPositions[] has been used in one function to calculate angles between CGPoints, how can I wrap my head around using the same technique to dray a line in between the two points.

I’m definitely missing something here, I’ve only been using Swift for a week, but it’s so frustrating!

Most of the comments with //---- are my failed attempts at creating something…

mhamilt commented 2 years ago

Since you've now got a GitHub repo you can track all your issues here. Instead of having disparate collection of notes in your Notes.app, Teams, email, on bits of paper &c... collect together all problems into issues here of which you can then easily keep track.

mhamilt commented 2 years ago

You have a couple of different models to work with (model in this case being a form and structure of data)

In the simplest case you will be using:

ARSkeleton2D.jointLandmarks is your data. This is the thing you want to know everything about, how big it is, what shape it is, what type it is.

One quirk is that accessing jointLandmarks directly appears to yield more data than using ARSkeleton2D.landmark(for:)

Connecting things together is very much "knee bone is connected to the hip bone" stuff. That is to say, there is a fixed order of things and so long as you are sensible, connecting lines (paths) between joints won't be too taxing.

Also, one thing that isn't leveraged is the ARSkeletonDefinition object, which might save you a bit of heavy lifting.

If you can get a circle for every joint, the next stage is adding a single Path

updateJointScreenPositions is where you will want to start looking when it comes to drawing. You'll want to create a new view and add it in the same way as the circles are.

mhamilt commented 2 years ago

[TwoDBodyJoint : UIView] - I’m really confused as to what that means.

If you are uncertain what something is, print it. Print its type and print its contents.

in this case it is simply a dictionary literal

https://developer.apple.com/documentation/swift/dictionary

csabau commented 2 years ago

in the updateJointScreenPosition the jointLandmark is of type CGPoint with x,y coordinates of type CGFloat. These are normalised pixel coordinates (0,0 top-left, 1,1 bottom-right) so the x,y values are always 0. something. In the for loop though, these values are transformed to screen-space coordinates and stored in jointScreenPositions array of CGPoints.

What I struggle to understand is how to Access that jointScreenPositions array. It seems like if I print it out from within the updateJointScreenPosition it constaldnly prints the updated version of every joint, which is great. But if I try to create a function outside of this area, the jointScreenPositions array contains (0,0) values.

Also I don't understand how the bodyTracker.attach function works. It takes in 2 parameters: 1 - thisView of type UIView. I'm not sure if I understand what that actually means in plain English, as in thisView there's a circle being passed. 2 - toThisJoint thisJoint: TwoDBodyJoint - I'm confused about this syntax. The parameter is toThisJoint , the type TwoDBodyJoint, but what's thisJoint doing / meaning? In plain English I understand that essentially is saying - attach this circle to this joint but I do not understand how the code actually finds the location of the joint based on the name alone.

Ideally I should be able to use a similar function and "attach" a line (instead of a circle) between two joints - right? So I would need a function similar to makeCircle but for "makeLine" perhaps that takes in parameters of two joints to determine the distance between them in order to draw it?

First I need to draw a simple line between two joints. I have an array that I don't understand where / how to access values of, and can't seem to reverse engineer the already existing function for applying graphics (circle) to a joint.

mhamilt commented 2 years ago

in the updateJointScreenPosition the jointLandmark is of type CGPoint

jointLandmark is a simd_float2, not a CGPoint, so be careful there.

These are normalised pixel coordinates (0,0 top-left, 1,1 bottom-right) so the x,y values are always 0. something.

correct

What I struggle to understand is how to Access that jointScreenPositions array

Like this

https://github.com/csabau/BTracking/blob/9cba6cd650fe3d50fb30d53a3a791f1e0fcd21ff/Sources/BodyTracking/BodyTracker2D.swift#L153

But if I try to create a function outside of this area, the jointScreenPositions array contains (0,0) values.

It depends on where and how you are doing this, I'd need to see an example.

Also I don't understand how the bodyTracker.attach function works. It takes in 2 parameters: 1 - thisView of type UIView. I'm not sure if I understand what that actually means in plain English, as in thisView there's a circle being passed.(https://github.com/csabau/BTracking/blob/9cba6cd650fe3d50fb30d53a3a791f1e0fcd21ff/Sources/BodyTracking/BodyTracker2D.swift#L86-L92) function works. It takes in 2 parameters: 1 - thisView of type UIView. I'm not sure if I understand what that actually means in plain English, as in thisView there's a circle being passed.

The UIView is the circle. "Animation" is achieved by moving the position of the same circle, not by wiping the screen and redrawing a new circle as you might be used to.

You'd need to the same with your Paths. You would create a bunch of UIViews that are paths and simply update their properties each time you get new data.

2 - toThisJoint thisJoint: TwoDBodyJoint - I'm confused about this syntax.

https://github.com/csabau/BTracking/blob/9cba6cd650fe3d50fb30d53a3a791f1e0fcd21ff/Sources/BodyTracking/BodyTracker2D.swift#L86

This is a little bit of syntactic sugar to make your declaration read nicely, but make the variable names in the function useable.

In long form

 public func attach

create a function called attach which can be used outside the class.

thisView: UIView

The first argument is called thisView and is of type UIView

toThisJoint thisJoint: TwoDBodyJoint

The second argument is toThisJoint but is aliased in the function to simply thisJoint and is of type TwoDBodyJoint

Look at the function

https://github.com/csabau/BTracking/blob/9cba6cd650fe3d50fb30d53a3a791f1e0fcd21ff/Sources/BodyTracking/BodyTracker2D.swift#L86-L92

notice it uses thisJoint and not toThisJoint

Ideally I should be able to use a similar function and "attach" a line (instead of a circle) between two joints

I'd actually edit the current attach function. Your Paths will be based on your joints after all.

So I would need a function similar to makeCircle but for "makeLine" perhaps that takes in parameters of two joints to determine the distance between them in order to draw it?

That's correct, probably using it somewhere around here

https://github.com/csabau/BTracking/blob/9cba6cd650fe3d50fb30d53a3a791f1e0fcd21ff/BodyTracking-Example/BodyTracking-Example/ARSUIView2D.swift#L91-L93

First I need to draw a simple line between two joints.

First you need to just draw a simple line if you haven't already. Then you can look at doing one between joints.

mhamilt commented 2 years ago

Within this there are probably 3 or 4 issues. I'd look to create them so you can segment the discussion a little bit. Otherwise you end up with answers like above where you're going to have to jump around a little

Probably an issue each for

mhamilt commented 2 years ago

This also still doesn't address how you are going to compare one set of data to another

csabau commented 2 years ago

Within this there are probably 3 or 4 issues. I'd look to create them so you can segment the discussion a little bit. Otherwise you end up with answers like above where you're going to have to jump around a little

Probably an issue each for

  • [ ] Draw a Path
  • [ ] Print out jointLandmark reliably
  • [ ] create makePath function
  • [ ] draw Path between two specific joints
  • [ ] draw Path between arbitrary joints
  • [ ] edit attach function to include Paths
  • [ ] remove extraneous comments and failed functions (as they aren't doing you any favours)

I now understand what you mean by having different “issues” I will open them separately later on today and will focus first on just getting a line drawn