search

artoolkitx / jsartoolkit5

Javascript ARToolKit v5.x
https://artoolkitx.github.io/jsartoolkit5/examples/
Other
292 stars 69 forks source link

NFT tracking jittery #97

Open Vasheer26 opened 4 years ago

Vasheer26 commented 4 years ago

Hi all, I have been creating my tests around the examples/nft_threejs_wasm.html. So far I have found this to be the best NFT tracking example since it works on the iPhone, Pixel, and Samsung however, I noticed it is very jittery when I placed a plane/video over the NFT image.

The jittery is also there when the 3D sphere is used but it is not easily noticeable. I would like to spend some time and effort to see if I can help smooth or reduce the jittery.

I assume I would need to look into artoolkit_wasm.js.

Following the instructions- I have started with installing Docker and slowly finding my way through various Youtube videos on how to set up the environments and pulling the Git repo. I would greatly appreciate connecting with anyone here who's more knowledgeable for assistance and direction. I know I will need to work with the c file and eventually use emscripten.

ThorstenBux commented 4 years ago

Hi there,

Help highly appreciated. You don’t necessarily need to touch Emscripten. The build scripts will do that for you. You might not even have to touch the C code. I’ve been in contact with @kalwalt to add a filter to smooth the tracking results in JS. Maybe he can share the latest results.

Get Outlook for iOShttps://aka.ms/o0ukef

From: Vasheer26 notifications@github.com Sent: Thursday, April 30, 2020 9:05:21 PM To: artoolkitx/jsartoolkit5 jsartoolkit5@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: [artoolkitx/jsartoolkit5] NFT tracking jittery (#97)

Hi all, I have been creating my tests around the examples/nft_threejs_wasm.html. So far I have found this to be the best NFT tracking example since it works on the iPhone, Pixel, and Samsung however, I noticed it is very jittery when I placed a plane/video over the NFT image.

The jittery is also there when the 3D sphere is used but it is not easily noticeable. I would like to spend some time and effort to see if I can help smooth or reduce the jittery.

I assume I would need to look into artoolkit_wasm.js.

Following the instructions- I have started with installing Docker and slowly finding my way through various Youtube videos on how to set up the environments and pulling the Git repo. I would greatly appreciate connecting with anyone here who's more knowledgeable for assistance and direction. I know I will need to work with the c file and eventually use emscripten.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fartoolkitx%2Fjsartoolkit5%2Fissues%2F97&data=02%7C01%7C%7Cff42948e8968453c147b08d7ece59cf3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637238343223877651&sdata=Z8wvQbPpTYcvRTh%2FuaIJwBX9txd57k9lqLUe%2BbMr4MM%3D&reserved=0, or unsubscribehttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAD765PA5WL567PGB4TXS7DDRPE5NDANCNFSM4MVKLYQA&data=02%7C01%7C%7Cff42948e8968453c147b08d7ece59cf3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637238343223877651&sdata=2sJVZ768KCw1w0Se4t%2B0rETtCFQbvmpbVm5tIk1Owtk%3D&reserved=0.

kalwalt commented 4 years ago

Hi @Vasheer26, i will share here with you my work. Not on Jsartoolki5 but i'm working on my lighter version JsartoolkitNFT you can try here https://github.com/kalwalt/jsartoolkitNFT/tree/improving-lerp/examples i'm trying to use the one euro filter (see the oneEuroFilter.js) i think you can use the same code for jsartoolkit5. I'm not sure of my code, nor of my implementation. I hope that his will help you...

Vasheer26 commented 4 years ago

@ThorstenBux Thank you for heads up. @kalwalt Thank you. I will spend time on implementing and testing this on a handful of devices and share my results in the coming days/weeks.

kalwalt commented 4 years ago

@ThorstenBux Thank you for heads up. @kalwalt Thank you. I will spend time on implementing and testing this on a handful of devices and share my results in the coming days/weeks.

That's magic! Ask if you need other infos, help, support :slightly_smiling_face:

kalwalt commented 4 years ago

Also, this is the oneEuroFilter that @ThorstenBux point me to

Vasheer26 commented 4 years ago

@ThorstenBux @kalwalt

After spending some times testing the provided codes, these are my findings:-

  1. Visually, the 'interpolationFactor' (IF) approached looked good but it was flawed. The gentle updating of position matrix (interpolationFactor=24) masked the jitterness. The lower the interpolationFaction (e.g 2 or less) makes rendering object follow the marker/image snappier but brought back the jitterness. We want the tracking to be snappier :)

  2. I implemented 3 stages of delta displacement so I could apply different levels of interplationFactor

    • "small" displacement delta <= 0.003 ( using IF = 10 )
    • "medium" displacement where delta > 0.003 ( using IF = 1.5 )
    • "big" displacement where delta > 0.005 ( using IF = 1 )

Again, remember the lower the IF, the snappier for the rendered object to match the marker. When the displacement is small, meaning it is near the correct position, we have to use IF 4 or greater to smooth out the jitterness. This approach was okay but not snappy enough.

  1. I also discovered the root cause of the jitterness is in the data being return in matrixGL_RH
    • With the webcam facing the stationary image, you can see the returning dataset bouncing all over the place. I tried to use every 5th, 10th, 20th and 50th dataset to see if jitterness could be lessen but unfortunately no. If anything you can clearly see the wide range of returning datasets.

I was thinking next

Just sharing my finding and thoughts. Happy to hear any of your thoughts or suggestions.

ThorstenBux commented 4 years ago

Thanks for sharing your findings. As said I don’t think oneeurofilter for smoothing and elimination of off values would be a good approach.

Kind regards Thorsten Bux

Sent from my iPhone

On 4/05/2020, at 6:47 PM, Vasheer26 notifications@github.com wrote:



@ThorstenBuxhttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FThorstenBux&data=02%7C01%7C%7C4ccefaf50a4f4eefd78c08d7eff6f1db%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637241716204129590&sdata=OBBr7ZVUjmD1yYZQNTCC2p8KZveJNfXUfvlB61Ow0vQ%3D&reserved=0 @kalwalthttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkalwalt&data=02%7C01%7C%7C4ccefaf50a4f4eefd78c08d7eff6f1db%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637241716204139584&sdata=0Us0pMZHy0g%2BLLfIZivXHmmBOXR55xtFe6xDJmMgLTc%3D&reserved=0

After spending some times testing the provided codes, these are my findings:-

  1. Visually, the 'interpolationFactor' (IF) approached looked good but it was flawed. The gentle updating of position matrix (interpolationFactor=24) masked the jitterness. The lower the interpolationFaction (e.g 2 or less) makes rendering object follow the marker/image snappier but brought back the jitterness. We want the tracking to be snappier :)

  2. I implemented 3 stages of delta displacement so I could apply different levels of interplationFactor

    • "small" displacement delta <= 0.003 ( using IF = 10 )
    • "medium" displacement where delta > 0.003 ( using IF = 1.5 )
    • "big" displacement where delta > 0.005 ( using IF = 1 )

Again, remember the lower the IF, the snappier for the rendered object to match the marker. When the displacement is small, meaning it is near the correct position, we have to use IF 4 or greater to smooth out the jitterness. This approach was okay but not snappy enough.

  1. I also discovered the root cause of the jitterness is in the data being return in matrixGL_RH

    • With the webcam facing the stationary image, you can see the returning dataset bouncing all over the place. I tried to use every 5th, 10th, 20th and 50th dataset to see if jitterness could be lessen but unfortunately no. If anything you can clearly see the wide range of returning datasets.

I was thinking next

Just sharing my finding and thoughts. Happy to hear any of your thoughts or suggestions.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fartoolkitx%2Fjsartoolkit5%2Fissues%2F97%23issuecomment-623288486&data=02%7C01%7C%7C4ccefaf50a4f4eefd78c08d7eff6f1db%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637241716204139584&sdata=vfHLowatIHCZudeGU7EV5dGJKV7d3%2FwCnXz9KmT2pjE%3D&reserved=0, or unsubscribehttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAD765PBZ7IYGTVPSHO4NXSDRPZQGFANCNFSM4MVKLYQA&data=02%7C01%7C%7C4ccefaf50a4f4eefd78c08d7eff6f1db%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637241716204139584&sdata=FvhZ1YJAr9DJZ92I7JgZFdHP59NrDx%2F8Oj7Io3bgU5g%3D&reserved=0.

kalwalt commented 4 years ago

@Vasheer26 I don't understand the relation betwenn the jtterness and MatrixGL_RH, Do uou think it is a difference between matrix and matrixGL_RH? that is, one with less jtterness than the other? and it's not clear to me what do you mean with: I tried to use every 5th, 10th, 20th and 50th dataset Does we have only 16 elements in the matrix?

Vasheer26 commented 4 years ago

@kalwalt Re: jitterness and MatrixGL_RH

I hope my explanation that make sense. Please let me know if it doesn't.

Anyway, next I will need look deeper into artoolkitNFT.worker.js => markerResult = {type: "found", matrixGL_RH: JSON.stringify(ev.data.matrixGL_RH)

Vasheer26 commented 4 years ago

Also, is there anyway I can review this file build/artoolkitNFT_wasm.js with the formatting in place. Currently, there's no tabbing or line breaks- very difficult to read :)

Vasheer26 commented 4 years ago

My next attempt was to smooth out the data sets by averaging out 5 dataset coming back from matrixGL_RH.

I did find that the jitterness was significantly reduced. (Usable but not 100% stable)

If I continue this approach and take the averaging of 10 data sets, the jitterness decreased again but at the expense of slower mapping of the video over NFT image. For now averaging over 5 data sets is relatively a good number to hold at.

Ideally, I need to get a better understanding why the data coming back from matrixGL_RH varies wildly with each return cycle.

kalwalt commented 4 years ago

Ideally, I need to get a better understanding why the data coming back from matrixGL_RH varies wildly with each return cycle.

Could be the cause is the fact that detection and tracking are not on a separate level as point out @hiukim https://github.com/artoolkitx/jsartoolkit5/issues/95#issuecomment-623059132 ?

ThorstenBux commented 4 years ago

Thanks for sharing your findings. As said I don’t think oneeurofilter for smoothing and elimination of off values would be a good approach. Kind regards Thorsten Bux

@Vasheer26 Hi sorry this was written wrong. I meant oneeurofilter is a good pick. It will eliminate these off values and also handle fast movements of the marker.

The average approach is good but doesn't eliminate values that are far off. OneEuroFilter would.

If you want to understand the values from the matrix you need to dive into the c code.

Vasheer26 commented 4 years ago

@ThorstenBux - cool thank you. Yes, unfortunately I did read you saying oneeurofilter was not good so I did not bother to look at that approach. At least now I will have the oneeurofilter to play with. Thank you for updating that.

Vasheer26 commented 4 years ago

Ideally, I need to get a better understanding why the data coming back from matrixGL_RH varies wildly with each return cycle.

Could be the cause is the fact that detection and tracking are not on a separate level as point out @hiukim #95 (comment) ?

I think no. According to Hiukim, he mentioned there are 2 phases, that being detection and tracking. The cycle that returns the matrixGL_RH, I believe lives with the detection phase.

ThorstenBux commented 4 years ago

No the matrix comes from tracking as far as I remember.

And sorry for the confusion about the filter 😣 my bad.

Kind regards Thorsten Bux

Sent from my iPhone

On 8/05/2020, at 10:59 AM, Vasheer26 notifications@github.com wrote:



Ideally, I need to get a better understanding why the data coming back from matrixGL_RH varies wildly with each return cycle.

Could be the cause is the fact that detection and tracking are not on a separate level as point out @hiukimhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fhiukim&data=02%7C01%7C%7C0bb49b40bd954b141f6208d7f2da56c9%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637244891875686370&sdata=FlQDJ39PCd6ZU%2BqPG6%2Bj6fa2Hc%2FQrM7nyltrkafCuYM%3D&reserved=0 #95 (comment)https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fartoolkitx%2Fjsartoolkit5%2Fissues%2F95%23issuecomment-623059132&data=02%7C01%7C%7C0bb49b40bd954b141f6208d7f2da56c9%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637244891875696350&sdata=vlPvvn6Qp%2BC4gmnr8rrjvIdPzjgSodPOBHy5XCZ5T%2F0%3D&reserved=0 ?

I think no. According to Hiukim, he mentioned there are 2 phases, that being detection and tracking. The cycle that returns the matrixGL_RH, I believe lives with the detection phase.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fartoolkitx%2Fjsartoolkit5%2Fissues%2F97%23issuecomment-625538822&data=02%7C01%7C%7C0bb49b40bd954b141f6208d7f2da56c9%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637244891875706345&sdata=CcLgQbepQe6SRkK3bKEHxU65yIPc0jfJzRmVCV0Y1fw%3D&reserved=0, or unsubscribehttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAD765PAYYM6N5BJUDCAEANLRQM4OBANCNFSM4MVKLYQA&data=02%7C01%7C%7C0bb49b40bd954b141f6208d7f2da56c9%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637244891875716341&sdata=hDuVyMJlkZ3TehtDdbGYZRyipGZX3ekuDi%2BhWvdgzug%3D&reserved=0.

Vasheer26 commented 4 years ago

@ThorstenBux haha.. and it turns out I got my wording wrong also.. You are correct.. The matrix comes from the tracking... not in the detection phase.

evaristoc commented 4 years ago

Hi @kalwalt @ThorstenBux @Vasheer26

Some observations?

  1. It seems you are reaching similar conclusions to the ones I have also reached when studying a script that does more or less the same thing as the artoolkit5 for NFT. I invite you to go through the original demo on which my project was based, or check my changes to the original one, with additional explicative notes. The library used for that was https://inspirit.github.io/jsfeat/, which I found well documented. Any other material regarding this kind of procedures could help. Eg. there are plenty of examples with opencv.

I like this line in the original demo...

  1. I am not an expert, but I am having the impression that there are many changes in this project that are following a naïve approach. Please don't get me wrong (especially you, @kalwalt ): I fully admire you all, as I am also coming from a similar route but you are working with C/C++ (Wauw!). However I personally think the simple trial-and-error route is not the most appropriate. Why?...

  2. As far as I understand, NFT procedures are mature and well established. I don't think the artoolkit5 NFT implementation is far from what is already known. My invitation is to further understand the procedure even before trying to change things. It is possible that there are things that this algorithm is unable to do as it is likely outdated and not maintained. You might be trying to reinvent the wheel.

  3. The people who originally worked on the artoolkit NFT algorithm didn't use a variable naming that helped to reveal what the algorithm is about. I think they kept it general in case of having to change some aspects of the procedure. I think if we can unveil the naming and relate it to the steps of the formal procedures for NFT would help substantially in understanding what can be done and what not.

The route that you follow after this is your choice though. This is a personal suggestion.

evaristoc commented 4 years ago

@Vasheer26 with my humble understanding of the implemented procedure, I think the matrixGL_RH you are working on is what it is known as the homographic matrix?

evaristoc commented 4 years ago

@kalwalt @ThorstenBux @Vasheer26

I will prepare a youtube video with some empirical results of implementing moving averages (what @Vasheer26 appears to implement here: https://github.com/artoolkitx/jsartoolkit5/issues/97#issuecomment-624914372, as well as a linear interpolation for alpha == .5, sample size = 2 ), Kalman filters and what I think is called stacking, by itsefl and combined, all on the javascript side. I made already a demo to show something about the AR.js NFT project to a prospective client. I will work the demo a bit further.

I haven't test the oneeuro filter.

My approach is likely NOT the right one but it might be good for some comparisons.

Vasheer26 commented 4 years ago

@evaristoc - thank you for letting me know the name of the matrix is homographic. This will at least give me a handle to grab onto when I am doing further research. I admit what I know and understand is very surface level and from that level is where I try to tackle what I can see.

While I have not yet had a chance to apply the oneeuro filter, I believe it is a bandage approach as it would be used to patch a problem and not fully understanding the original cause to prevent the root problem. A massive thank you for inviting me to go through the original demo and your changes with the additional notes. Greatly appreciate it.

ThorstenBux commented 4 years ago

Hi, I highly appreciate that you want to look into the depths of artoolkit and find and fix the problem there. For what it is worth, my knowledge about the topic. Nft tracking was implemented quite some time ago by PhD students. I assume thay had quite some knowledge about CV and the algorithms, it can well be that the algorithms improved since then. Now, from my perspective a oneeurofilter approach would show quick, easy and fast results which would improve the project and the motivation. Low hanging fruit 😊. And then I would look into the rabbit hole 🕳. Now as said that is my view and you are free to choose the way you want to go. Simply thought I’d share.

evaristoc commented 4 years ago

@ThorstenBux

Now, from my perspective a oneeurofilter approach would show quick, easy and fast results which would improve the project and the motivation. Low hanging fruit . And then I would look into the rabbit hole . Now as said that is my view and you are free to choose the way you want to go. Simply thought I’d share.

Thanks! No, it is actually more your choice, I think. I was more like helping out :smile: . If we (your target group) are happy with the results, then why not :+1: !

evaristoc commented 4 years ago

@Vasheer26

@evaristoc - thank you for letting me know the name of the matrix is homographic. This will at least give me a handle to grab onto when I am doing further research. I admit what I know and understand is very surface level and from that level is where I try to tackle what I can see.

Yes. I understand your position. It has taken me some time to get the idea of how the whole stuff more or less works. As @ThorstenBux well mentioned, the project needs results soon and it is ok to try at least something.

not fully understanding the original cause to prevent the root problem.

I think the problem with the algorithm is that it might be matching many points that are false positives - they lay away from its corresponding position within the actual target image as seen in the camera. Those false positives, which are outliers, are randomly targeted per frame, so that is the variability.

That is enough to provide wrong measures for point positioning and orientation, which is based on the total points considered as "good" match. One parameter affected would be the homographic matrix, which I think uses all those points for its estimation.

The false positives problem is HARD to solve and it might be inherent to probably all algorithms for image detection.

Because it is hard to evaluate if a point is an outlier or not after detection, one way to solve the issue is post-processing. I believe there are procedures that are very good in matching correct points overall. There are one which I heard is very good but it is licensed.

While I have not yet had a chance to apply the oneeuro filter, I believe it is a bandage approach as it would be used to patch a problem

It is a variation, more advanced filter that uses low pass filter (https://en.wikipedia.org/wiki/Low-pass_filter) at several stages. It samples points at two moments of the signal. For what I can get from the script:

  1. It creates a weighting value alpha, which is based on the freq argument; assumes circular error (gaussian?).
  2. It samples two values for comparison: the current one and another happening at timespan before. Timespan is argument and fixed.
  3. It estimates a expected "distance" between those two values (dx) on frequency (eg. by frame).
  4. It then runs a low pass filter over that difference having also a dcutoff as argument: this part is smoothing the change in the range between two different positions along a fixed time.
  5. It uses that value to adjust the cutoff value eventually used to assign the final new entry point by calculating another low pass, this time over the points, not the range.

It seems powerful: it takes in consideration an adjustment of the estimation of the error change, likely the variance (in the filter that refers to the adjustment of the range).

For the values of the parameter to use, I would guess as first values to try (but NOT sure...):

  1. The freq value calculated by the algo appears to be close to number of frames, perhaps something no more than fps.
  2. If the jumping in values occurs substantially frame by frame (ie. the dx is large frame to frame), I think you want to sample very close values, so timespan should be small but NOT too small (otherwise there will be no sampling at all). I think it would be okey to keep it within a second but depends how far the error occurs.
  3. With a high dCutoff value, I think you are like less concerned by how large the error. However you might want a dCutoff relatively small if the error is high and frequent.
  4. The minCutoff depends on how far you are expected to control for a visible movement from A to B based on your reference system.

A massive thank you for inviting me to go through the original demo and your changes with the additional notes. Greatly appreciate it.

Sorry I can't help more. Please use it cautiously. I am also not a specialist.

evaristoc commented 4 years ago

@kalwalt maybe the message above I just sent to @Vasheer26 can work for you? I wrote to him my impressions about the oneEuroFilter. I am not absolutely sure my impressions are correct but well... it is something.

albjeremias commented 4 years ago

I think the problem with the algorithm is that it might be matching many points that are false positives - they lay away from its corresponding position within the actual target image as seen in the camera. Those false positives, which are outliers, are randomly targeted per frame, so that is the variability.

That is enough to provide wrong measures for point positioning and orientation, which is based on the total points considered as "good" match. One parameter affected would be the homographic matrix, which I think uses all those points for its estimation.

The false positives problem is HARD to solve and it might be inherent to probably all algorithms for image detection.

Hi, I've been looking into how to implement this algorithms. My first approach was using opencvjs with ORB or SIFT. I wonder which algorithm artoolkit5 uses, but the important is that on most of opencv implementations detection and tracking is different algorithms, and already someone mention that happening with artoolkit.. anyway in the most classical approaches i've seen tracking smoothing using a kalman filter, or optical flow estimation.. oneEuroFilter or LowPassFilter does not seem the most adequate at least in most approaches for a computer vision software.

hiukim commented 4 years ago

I think there are two separate algorithms for detection and tracking. The one behind detection is FREAK. I'm still digesting the tracking part.

albjeremias commented 4 years ago

I think there are two separate algorithms for detection and tracking. The one behind detection is FREAK. I'm still digesting the tracking part.

@hiukim can you please post some references on the code lines? id like to digg a bit into it.

hiukim commented 4 years ago

@albjeremias For the detection part, you may start with these entry points:

To generate dataset: https://github.com/artoolkit/ARToolKit5/blob/5bf0b671ff16ead527b9b892e6aeb1a2771f97be/lib/SRC/KPM/kpmRefDataSet.cpp#L287

To do matching: https://github.com/artoolkit/ARToolKit5/blob/5bf0b671ff16ead527b9b892e6aeb1a2771f97be/lib/SRC/KPM/kpmMatching.cpp#L383

These are some good entries point for the feature detection and matching algorithm. Before that there are still a few processing steps.

Also, these parts is only for the detection. Once a marker is detected, there is another algorithm for tracking.

dandingol03 commented 1 year ago

Hi @Vasheer26, i will share here with you my work. Not on Jsartoolki5 but i'm working on my lighter version JsartoolkitNFT you can try here https://github.com/kalwalt/jsartoolkitNFT/tree/improving-lerp/examples i'm trying to use the one euro filter (see the oneEuroFilter.js) i think you can use the same code for jsartoolkit5. I'm not sure of my code, nor of my implementation. I hope that his will help you...

this link https://github.com/kalwalt/jsartoolkitNFT/tree/improving-lerp/examples is invalid now