search

nickw1 / ptam_expts

Experimenting with providing a web interface to PTAM
2 stars 1 forks source link

Testing ptam_expts #1

Open kalwalt opened 3 years ago

kalwalt commented 3 years ago

Hi @nickw1 i'm trying to test your code, assumed that libPTAM.a is compiled from @ThorstenBux test/compiling branch. Inside ptam/wasm folder i run emcamke cmake . and then emmake make, and this is the result:

emmake make
make: make
[ 50%] Linking CXX executable ptam_wasm.js
error: undefined symbol: _ZN2cv3Mat10deallocateEv (referenced by top-level compiled C/C++ code)
warning: Link with `-s LLD_REPORT_UNDEFINED` to get more information on undefined symbols
warning: To disable errors for undefined symbols use `-s ERROR_ON_UNDEFINED_SYMBOLS=0`
warning: __ZN2cv3Mat10deallocateEv may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: _ZN2cv3Mat20updateContinuityFlagEv (referenced by top-level compiled C/C++ code)
warning: __ZN2cv3Mat20updateContinuityFlagEv may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: _ZN2cv5errorEiRKNSt3__212basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEEPKcSA_i (referenced by top-level compiled C/C++ code)
warning: __ZN2cv5errorEiRKNSt3__212basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEEPKcSA_i may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: _ZN2cv8cvtColorERKNS_11_InputArrayERKNS_12_OutputArrayEii (referenced by top-level compiled C/C++ code)
warning: __ZN2cv8cvtColorERKNS_11_InputArrayERKNS_12_OutputArrayEii may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: _ZN2cv8fastFreeEPv (referenced by top-level compiled C/C++ code)
warning: __ZN2cv8fastFreeEPv may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: _ZN3CVD10halfSampleERKNS_10BasicImageIhEERS1_ (referenced by top-level compiled C/C++ code)
warning: __ZN3CVD10halfSampleERKNS_10BasicImageIhEERS1_ may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: _ZN3CVD11fast_nonmaxERKNS_10BasicImageIhEERKNSt3__26vectorINS_8ImageRefENS4_9allocatorIS6_EEEEiRS9_ (referenced by top-level compiled C/C++ code)
warning: __ZN3CVD11fast_nonmaxERKNS_10BasicImageIhEERKNSt3__26vectorINS_8ImageRefENS4_9allocatorIS6_EEEEiRS9_ may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: _ZN3CVD16convolveGaussianERKNS_10BasicImageIfEERS1_dd (referenced by top-level compiled C/C++ code)
warning: __ZN3CVD16convolveGaussianERKNS_10BasicImageIfEERS1_dd may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: _ZN3CVD21fast_corner_detect_10ERKNS_10BasicImageIhEERNSt3__26vectorINS_8ImageRefENS4_9allocatorIS6_EEEEi (referenced by top-level compiled C/C++ code)
warning: __ZN3CVD21fast_corner_detect_10ERKNS_10BasicImageIhEERNSt3__26vectorINS_8ImageRefENS4_9allocatorIS6_EEEEi may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: _ZN3CVD8Internal12aligned_freeEPv (referenced by top-level compiled C/C++ code)
warning: __ZN3CVD8Internal12aligned_freeEPv may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: _ZN3CVD8Internal13aligned_allocEmm (referenced by top-level compiled C/C++ code)
warning: __ZN3CVD8Internal13aligned_allocEmm may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: _ZN3CVD9cvd_timer8get_timeEv (referenced by top-level compiled C/C++ code)
warning: __ZN3CVD9cvd_timer8get_timeEv may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: _ZN6GVars313parse_warningEiNSt3__212basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEES6_S6_ (referenced by top-level compiled C/C++ code)
warning: __ZN6GVars313parse_warningEiNSt3__212basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEES6_S6_ may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: _ZN6GVars33GV311add_typemapEPNS_7BaseMapE (referenced by top-level compiled C/C++ code)
warning: __ZN6GVars33GV311add_typemapEPNS_7BaseMapE may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: _ZN6GVars36GVars26GetIntERKNSt3__212basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEEii (referenced by top-level compiled C/C++ code)
warning: __ZN6GVars36GVars26GetIntERKNSt3__212basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEEii may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: _ZN6GVars36GVars29GetDoubleERKNSt3__212basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEEdi (referenced by top-level compiled C/C++ code)
warning: __ZN6GVars36GVars29GetDoubleERKNSt3__212basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEEdi may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: _ZN6GVars39serialize10FromStreamINSt3__212basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEEE4fromERNS2_13basic_istreamIcS5_EE (referenced by top-level compiled C/C++ code)
warning: __ZN6GVars39serialize10FromStreamINSt3__212basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEEE4fromERNS2_13basic_istreamIcS5_EE may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: _ZN6GVars39serialize12check_streamERNSt3__213basic_istreamIcNS1_11char_traitsIcEEEE (referenced by top-level compiled C/C++ code)
warning: __ZN6GVars39serialize12check_streamERNSt3__213basic_istreamIcNS1_11char_traitsIcEEEE may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: _ZN6GVars39serialize9to_stringERKNSt3__212basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEEb (referenced by top-level compiled C/C++ code)
warning: __ZN6GVars39serialize9to_stringERKNSt3__212basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEEb may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: DISABLE_EXCEPTION_CATCHING was set, which means no C++ exception catching support code is linked in, but such support is required by symbol `__cxa_begin_catch`. Either link with DISABLE_EXCEPTION_CATCHING=0 (if you do want exception catching) or compile all source files with DISABLE_EXCEPTION_CATCHING=1 (the default) (so that no exception catching support code is required).
error: DISABLE_EXCEPTION_CATCHING was set, which means no C++ exception catching support code is linked in, but such support is required by symbol `__cxa_end_catch`. Either link with DISABLE_EXCEPTION_CATCHING=0 (if you do want exception catching) or compile all source files with DISABLE_EXCEPTION_CATCHING=1 (the default) (so that no exception catching support code is required).
Internal compiler error in src/compiler.js!
Please create a bug report at https://github.com/emscripten-core/emscripten/issues/
with a log of the build and the input files used to run. Exception message: "ReferenceError: addCxaCatch is not defined" | ReferenceError: addCxaCatch is not defined
    at functionStubHandler (eval at load (/home/walter/emsdk/upstream/emscripten/src/compiler.js:38:8), <anonymous>:128:7)
    at Array.forEach (<anonymous>)
    at JSify (eval at load (/home/walter/emsdk/upstream/emscripten/src/compiler.js:38:8), <anonymous>:440:19)
    at Object.<anonymous> (/home/walter/emsdk/upstream/emscripten/src/compiler.js:98:3)
    at Module._compile (internal/modules/cjs/loader.js:1063:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
    at Module.load (internal/modules/cjs/loader.js:928:32)
    at Function.Module._load (internal/modules/cjs/loader.js:769:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)
    at internal/main/run_main_module.js:17:47
em++: error: '/home/walter/emsdk/node/14.15.5_64bit/bin/node /home/walter/emsdk/upstream/emscripten/src/compiler.js /tmp/tmp4ae9ecnm.txt' failed (1)
make[2]: *** [CMakeFiles/ptam_wasm.dir/build.make:88: ptam_wasm.js] Errore 1
make[1]: *** [CMakeFiles/Makefile2:76: CMakeFiles/ptam_wasm.dir/all] Errore 2
make: *** [Makefile:84: all] Errore 2
emmake: error: 'make' failed (2)
walter@walter-RC530-RC730:~/kalwalt-github/ptam_expts/ptam/wasm$

quite different from your error reported on slack. The only changes i had to do was in the opencv paths, but i think it'n not the root of the issue. I will report my investigations in this issue.

kalwalt commented 3 years ago

Looking at the errors I think it is required opencv.js lib; does it is right?

nickw1 commented 3 years ago

Hi @kalwalt strange. Yes, opencv.js, or rather OpenCV itself compiled with emscripten, should be there, it was there in an earlier commit and I had taken it out (to try and diagnose what was causing the error) and unintentionally left it missing.

I have now added it back again but I still get the same error. Which commit did you try building? (I ask as I made quite a few commits today).

Odd that you get a different error, maybe it's due to different OpenCV versions? I get errors like that when I change the order of linking .a files; the order I currently have gives no errors of that kind.

Which version of opencv are you using incidentally? The version I am using is from cloning the repo on July 1st.

kalwalt commented 3 years ago

Hi @kalwalt strange. Yes, opencv.js, or rather OpenCV itself compiled with emscripten, should be there, it was there in an earlier commit and I had taken it out (to try and diagnose what was causing the error) and unintentionally left it missing.

I have now added it back again but I still get the same error. Which commit did you try building? (I ask as I made quite a few commits today).

Odd that you get a different error, maybe it's due to different OpenCV versions? I get errors like that when I change the order of linking .a files; the order I currently have gives no errors of that kind.

Which version of opencv are you using incidentally? The version I am using is from cloning the repo on July 1st.

I'am using opencv 4.5.0 - the problem was probably the wrong opencv path for the includes and libs. Infact now i got the same error you encountered, see the error message:

emmake make
make: make
Scanning dependencies of target ptam_wasm
[ 50%] Building CXX object CMakeFiles/ptam_wasm.dir/ptam_wasm.cpp.o
[100%] Linking CXX executable ptam_wasm.js
wasm-ld: warning: function signature mismatch: dgesvd_
>>> defined as (i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32) -> void in /home/walter/kalwalt-github/ptam_plus/ptam/libPTAM.a(homography_init.cc.o)
>>> defined as (i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32) -> i32 in /home/walter/kalwalt-github/ptam_plus/external/clapack-3.2.1/SRC/liblapack.a(dgesvd.c.o)

wasm-ld: warning: function signature mismatch: s_cat
>>> defined as (i32, i32, i32, i32, i32) -> i32 in /home/walter/kalwalt-github/ptam_plus/external/clapack-3.2.1/SRC/liblapack.a(dormbr.c.o)
>>> defined as (i32, i32, i32, i32, i32) -> void in /home/walter/kalwalt-github/ptam_plus/external/clapack-3.2.1/F2CLIBS/libf2c/libf2c.a(s_cat.c.o)
[parse exception: attempted pop from empty stack / beyond block start boundary at 141748 (at 0:141748)]
Fatal: error in parsing input
em++: error: '/home/walter/emsdk/upstream/bin/wasm-emscripten-finalize --minimize-wasm-changes --dyncalls-i64 ptam_wasm.wasm -o ptam_wasm.wasm --detect-features' failed (1)
make[2]: *** [CMakeFiles/ptam_wasm.dir/build.make:93: ptam_wasm.js] Errore 1
make[1]: *** [CMakeFiles/Makefile2:76: CMakeFiles/ptam_wasm.dir/all] Errore 2
make: *** [Makefile:84: all] Errore 2
emmake: error: 'make' failed (2)

to avoid errors and help other developers, if you agree, we could setup a common opencv dependencies. We ( me and @EdwardLu2018) did a script to build opencv or you can download precompiled libs from https://github.com/webarkit/opencv-em/releases. We developed this for wasm-ar (our fork) and for the future of webarkit webarkit-testing. You can see an example in this gist

kalwalt commented 3 years ago

regarding the error: Maybe the mapMaker is not instantiated in the right way?

nickw1 commented 3 years ago

to avoid errors and help other developers, if you agree, we could setup a common opencv dependencies. We ( me and @EdwardLu2018) did a script to build opencv or you can download precompiled libs from https://github.com/webarkit/opencv-em/releases. We developed this for wasm-ar (our fork) and for the future of webarkit webarkit-testing. You can see an example in this gist

Yes, that makes a lot of sense, will use your OpenCV from now on.

nickw1 commented 3 years ago

regarding the error: Maybe the mapMaker is not instantiated in the right way?

I'm fairly sure it is not that, as the code was taken from the original PTAM slam_system.cc, and I would expect a more straightforward "type mismatch" error if the wrong types had been passed to the MapMaker.

kalwalt commented 3 years ago

regarding the error: Maybe the mapMaker is not instantiated in the right way?

I'm fairly sure it is not that, as the code was taken from the original PTAM slam_system.cc, and I would expect a more straightforward "type mismatch" error if the wrong types had been passed to the MapMaker.

yes you are right. I can see that mapMaker class is using std::thread internally see:

mpThread = std::shared_ptr<std::thread>(new std::thread(std::bind(&MapMaker::run, this)));

https://github.com/ThorstenBux/ptam_plus/blob/9722737da073ec6bee0f43be60c53f0edc8fcc4c/ptam/construct/map_maker.cc#L38

Maybe this is causing the issue? We should try to compile the libPTAM.a with -s USE_PTHREADS=1 as described https://emscripten.org/docs/porting/pthreads.html but i'm not sure that is possible. Though this lead to a whole series of consideration: emscripten code compiled with PTHREADS enabled will need the SharedArrayBuffer flag enabled ( ! ). Probably the PTAM code for the web need some modifications.

nickw1 commented 3 years ago

OK thanks for that. Wondering whether a separate thread is so critical for a web front end?

kalwalt commented 3 years ago

OK thanks for that. Wondering whether a separate thread is so critical for a web front end?

We Could try without the thread. But i don' t know how much code Is necessary to change. Maybe Is possibile to use the ptam code without the MapMaker class?

nickw1 commented 3 years ago

Possibly yes, will have a look at the code.

nickw1 commented 3 years ago

OK, I have tried removing all references to threads in both map_maker and tracker - I realise this may have problems but I just tried it to see if it would solve the issue.

Unfortunately it does not, still the same error if the application is linked against a version of PTAM with the thread references removed.

I get the impression this is a difficult one to solve, and may be due to limitations in emscripten itself. Maybe we need to comment out each method, in turn, in map_maker to attempt to identify the source?

kalwalt commented 3 years ago

OK, I have tried removing all references to threads in both map_maker and tracker - I realise this may have problems but I just tried it to see if it would solve the issue.

Unfortunately it does not, still the same error if the application is linked against a version of PTAM with the thread references removed.

I get the impression this is a difficult one to solve, and may be due to limitations in emscripten itself. Maybe we need to comment out each method, in turn, in map_maker to attempt to identify the source?

Thanks for testing, at this point i think we need as you said, to test every little step in mapMaker. For now i have no other ideas.

nickw1 commented 3 years ago

@kalwalt OK had some time this morning to test this.

Have narrowed down the problem to one line, but it's a bit hard to understand why.

Firstly the call to ReprojectPoint() in MapMaker::AddPointEpipolar() causes the problem, when this line is commented out, with the rest of the map_maker.cc source untouched, it links without the above error (there are then other errors to do with order of linking the libraries, but these are what I call 'standard errors' and more easy to deal with).

When I then look at the ReprojectPoint() function, this line seems to be the specific cause of the problem:

TooN::SVD<4, 4> svd(A)

A is just a matrix (specifically of type TooN::Matrix(4))

I have no idea why this would cause the error. Not sure if there's anything we can do now other than to get advice on the Emscripten developers' list?

nickw1 commented 3 years ago

Sorry, to update:

Digging deeper it appears the constructor above

TooN::SVD<4,4> svd(A);

actually performs a calculation. Will investigate further.

Sorry, I'm not so used to a coding style in which constructors perform calculation work! ;-)

kalwalt commented 3 years ago

Good catch Nick! Yes, maybe asking advice to the emscripten devs is a good idea. I can't do anything during this period...

nickw1 commented 3 years ago

Updating, tracked the error down to the gesvd_() call in the above constructor, which then calls dgesvd_() from clapack.

The code in there looks a bit of mess if I am honest: appears to be auto-converted Fortran to C code. Some of the earlier issues were also down to things in clapack. Perhaps not surprising emscripten is getting confused! ;-)

Wondering if there is any way we can replace this with a pure-C, written-from-scratch-in-C alternative. Will have a look.

nickw1 commented 3 years ago

Further gone down into dgesvd_().

I have tracked the error down to the long series of if/else statements in between lines 282 and 829 of dgesvd.c in the CLAPACK library. However, bizarrely, if I comment out ALL the code inside the if/else blocks, leaving just the if conditions themselves unchanged, I still get the error.

Have asked on the Emscripten list just in case anyone can shed any light. (EDIT: not PTAM list, this was a typo!)

nickw1 commented 3 years ago

Rebuilt project and the error seemed to go away, maybe I forgot a build step ;-)

Have now I think made progress on diagnosing the error. A lot of the libf2c functions used by clapack are incorrectly declared in clapack, for example s_cat() and s_copy(). In libf2c these functions are implemented as returning void. In clapack they are declared as returning int. (A variation on this problem came earlier in the form of build warnings, which I corrected in one or two cases, to remove the warnings, but not all)

Correcting the declarations in the clapack files to return void appears to solve the problem. Looks like it will need to be done several times though.

nickw1 commented 3 years ago

See https://github.com/nickw1/ptam_plus/tree/ptam_emscripten_fix for the version of PTAM with the fixes made, this is forked from @ThorstenBux's ptam_plus.

nickw1 commented 3 years ago

Well finally got the thing to compile! :-)

s_cat() declaration fix got rid of the strange 'cannot pop from an empty stack' error, which then left various linker errors relating to the CVD library.

Turned out that the CMakeLists.txt was not including the noarch directory within CVD which meant that various functions were never included in the library. See above link for details.

kalwalt commented 3 years ago

Well finally got the thing to compile! :-)

s_cat() declaration fix got rid of the strange 'cannot pop from an empty stack' error, which then left various linker errors relating to the CVD library.

Turned out that the CMakeLists.txt was not including the noarch directory within CVD which meant that various functions were never included in the library. See above link for details.

That's a good news!! Can't help but i'm following your progresses...

ThorstenBux commented 3 years ago

Hi @nickw1, I might have some time over the next few days. How can I best support you right now?

nickw1 commented 3 years ago

Hi @ThorstenBux many thanks! :-) (sorry for late reply, have only just checked my email)

I have now got PTAM and my test app working with pthreads, which is good. I am still getting a few strange emscripten 'index out of bounds' bugs which I can't quite figure out. Would you be able to help here if I cannot resolve?

nickw1 commented 3 years ago

Hi @ThorstenBux ok some details of the error which is proving hard to resolve.

Stack trace:

RuntimeError: index out of bounds
    invoke_ii http://localhost/ptam_expts/ptam/wasm/ptam_wasm.js:8089
    invoke_vii http://localhost/ptam_expts/ptam/wasm/ptam_wasm.js:8056
    createExportWrapper http://localhost/ptam_expts/ptam/wasm/ptam_wasm.js:1314
    passToWasm http://localhost/ptam_expts/ptam/index.js:84
    sendCanvas http://localhost/ptam_expts/ptam/index.js:75
    processVideo http://localhost/ptam_expts/ptam/index.js:64

This always occurs when tracking begins, i.e. the user has captured two initial key frames successfully. It does find the map points which appear to have valid xyz coordinates.

It is generated from Emscripten auto-generated code, specifically the line highlighted:

function invoke_ii(index, a1) {
 var sp = stackSave();
 try {
  return wasmTable.get(index)(a1); // THIS LINE
 } catch (e) {
  stackRestore(sp);
  if (e !== e + 0 && e !== "longjmp") throw e;
  _setThrew(1, 0);
 }
}

It's not always the same function but always a similar one i.e. invoke_<some_roman_numeral> and the specific call wasmTable.get(index) seems to trigger it, suggesting perhaps that it is trying to find an entry in wasmTable which does not exist. It appears to occur after the C++ MapMaker::BundleAdjustAll() method returns.

Note that these functions seem to be called all the time, and index has different values so most of the time it works.

Have tried asking on the Emscripten forum but no luck. Any ideas about this one? Thanks.

nickw1 commented 3 years ago

.. Sorry to update my previous message after doing a bit more digging.

The wasmTable.get(index) DOES return a native function when the error occurs, so the index passed in as an argument to get() is not the index which is out of bounds, I presume.

I'm wondering if it is stack corruption as it seems to occur at the point at which a C++ function returns. In which case it might be down to bugs in PTAM, e.g. buggy PTAM code using an invalid index into an array somewhere and corrupting the stack? Not sure.

kalwalt commented 3 years ago

Hi @nickw1 maybe It require some additional Memory allocation?

nickw1 commented 3 years ago

@kalwalt thanks. Funnily enough I had just come to the same possible conclusion as I tested it in Chrome (have been using Firefox up to now) and got the slightly more useful error logging:

MapMaker::BundleAdjust() : done.
ptam_wasm.js:2606 BundleAdjust() completed
ptam_wasm.js:2606 Returning from MapMaker::BundleAdjustAll()
index.js:93 RuntimeError: memory access out of bounds
    at <anonymous>:wasm-function[883]:0x2d404
    at <anonymous>:wasm-function[883]:0x2d407
    at <anonymous>:wasm-function[883]:0x2d407
    at <anonymous>:wasm-function[883]:0x2d407
    at <anonymous>:wasm-function[883]:0x2d407
    at <anonymous>:wasm-function[883]:0x2d407
    at <anonymous>:wasm-function[883]:0x2d411
    at <anonymous>:wasm-function[883]:0x2d411
    at <anonymous>:wasm-function[874]:0x2a9fd
    ... etc ..

Might be related to this issue, which gives some suggestions on increasing memory allocation:

https://github.com/emscripten-core/emscripten/issues/14538

Will have a play with various memory options and see what I get.

kalwalt commented 3 years ago

@nickw1 there are also some optional Emscripten flags to get more useful error infos. I can't remember now but you can find in the Emscripten help

nickw1 commented 3 years ago

@kalwalt ok thanks.

ThorstenBux commented 3 years ago

Hi Nick,

Yes I can support with that. However, I've been sucked into a project now so might be slower.

How are you currently building/testing everything? I kind of miss an entrance point šŸ˜Š.

ThorstenBux commented 3 years ago

Emscripten debugging flags https://emscripten.org/docs/porting/Debugging.html

nickw1 commented 3 years ago

Hi Thorsten, thanks. There's an index.html inside the ptam directory within this repo. There's also an index.js which deals with capturing the camera feed, passing that onto the C++ backend, and handling the UI. The C++ is inside the wasm directory, there's also a cmake build file in there.

For a while I have been testing it on a standard laptop, but the last two days I have moved to testing on an Android phone instead, with Chrome, using port forwarding to contact the server running on the laptop.

Increasing the available memory (-s INITIAL_MEMORY=512mb) doesn't seem to resolve the problem but will look into the debugging docs.

Also occasionally PTAM fails at various assertions. Maybe worthwhile rewriting these assertions as simple checks instead so that it aborts the tracking without crashing out.

nickw1 commented 3 years ago

To add to that, just in case it's unclear how to use the test application:

The test app has four buttons, press 'Start Tracking' to start PTAM up, then 'Capture key frame 1' to capture the first key frame, then (as recommended by the PTAM authors) translate the camera to an adjacent location and 'Capture key frame 2'.

It may fail to match up key points in the two frames, in which case it will tell you and you will need to capture both key frames again.

Output is just on the console log at the moment, I don't yet try to draw anything in 3D on the camera feed.

Then 'stop tracking' and press 'Cleanup' when you're finished, this is important to free up the memory.

nickw1 commented 3 years ago

OK added a few debugging flags.

It looks like some code, somewhere, is trying to delete memory which has already been deleted:

ptam_wasm.js:2961 ==42==ERROR: AddressSanitizer: attempting double-free on 0x13400f50 in 
thread T0:
#0 0x20888a in wasm-function[-1]+0x20888a (http://localhost:8080/ptam_expts/ptam/wasm/ptam_wasm.wasm+0x20888a)
put_char    @   ptam_wasm.js:2961
write   @   ptam_wasm.js:2896
write   @   ptam_wasm.js:4551
doWritev    @   ptam_wasm.js:5373
_fd_write   @   ptam_wasm.js:8019

(I have added in debugging options for the C++ but it still isn't giving me any more detailed info) I guess it's a case of trying to track down in the PTAM code if any pointers are being freed twice, but if anyone has any other suggestions do let me know :-)

EDIT: getting somewhere now. The key debug flag appears to be --emit-symbol-map : missed that one previously.

nickw1 commented 3 years ago

OK some progress...

Changing a variable from a pointer (to a Toon Vector) to a plain object in atan_camera.cc seemed to fix the issues I was getting, which occurred on destruction of the object. Not entirely sure what the issue was though.

Good news is, this morning I have managed to carry out about 4 successful tracking sessions (out of perhaps 10-15 attempts). Successful in the sense that not only are map points detected, but the pose matrix is obtained - and updated as you move the phone around. Note that there is still no graphical display - it's all console output, so I don't know if the map points correspond to actual real-world features nor whether they are actually tracked accurately as I move the phone around. This is the next stage...

Still some issues though:

Anyhow, at least the tracking now appears to be working - sometimes :-)

kalwalt commented 3 years ago

A real good progress! To the first point: which part of the code need more memory in your opinion?

nickw1 commented 3 years ago

@kalwalt thanks! It seems to be the bundle adjuster which is using up a lot of memory, from my tests so far. It tends to happen when there are many map points. Maybe the code needs to be changed (or maybe there's an option) to set a limit on the number of map points.

nickw1 commented 3 years ago

Also, to make it clear, I did enable the memory growth option for Emscripten, so looks like the available memory of Chrome has been reached.

nickw1 commented 3 years ago

Just continuing to look at this.

Another observation (no solution yet, just noting for future reference): sometimes BundleAdjustAll() is called in a loop continuously due to lack of bundle convergence. Need to look into exactly what is happening here.

kalwalt commented 3 years ago

Just continuing to look at this.

Another observation (no solution yet, just noting for future reference): sometimes BundleAdjustAll() is called in a loop continuously due to lack of bundle convergence. Need to look into exactly what is happening here.

Hi, @nickw1 i hope to have some free time in September, maybe i can help with this issue! šŸ™‚

nickw1 commented 3 years ago

Hi @kalwalt thanks!

Another issue I am finding is that it is almost impossible to get the tracking started outdoors. It only really works indoors (testing on a table with lots of objects). I am trying to initiate tracking outdoors with a dirt path running through grass, or trees - and having no success. This is the use-case for my own app so it would be nice to get outdoor feature detection and 3D coordinate placement working.

PTAM seemed to be tested by the authors on a desk within a room so maybe the tracking is not quite sensitive enough to work outdoors?

Not sure if it's worth also testing ORB-SLAM? Or whether the sensitivity of PTAM in an outdoor setting can be improved?

nickw1 commented 2 years ago

Now also investigating ORB_SLAM : see https://github.com/nickw1/orb-slam-expts/