WIP: Web keypoint UI

[hietalajulius]

WIP: Web keypoint UI

Currently does not load existing keypoints + saving is not yet implemented.

The frontend is based on https://mui.com/ that has a bunch of ready-to-use components. Styling/theming the components: https://mui.com/customization/theming/

Both the backend/frontend has a README for starting things up.

[kekeblom]

Looks good. I'll do some work on this starting with the styles.

The triangulation doesn't work on my setup. It fails with this error:

Traceback (most recent call last):
  File "/home/ken/miniconda3/envs/web/lib/python3.7/site-packages/flask/app.py", line 2073, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/ken/miniconda3/envs/web/lib/python3.7/site-packages/flask/app.py", line 1518, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/ken/miniconda3/envs/web/lib/python3.7/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/ken/miniconda3/envs/web/lib/python3.7/site-packages/flask/app.py", line 1516, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/ken/miniconda3/envs/web/lib/python3.7/site-packages/flask/app.py", line 1502, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/home/ken/miniconda3/envs/web/lib/python3.7/site-packages/flask/views.py", line 84, in view
    return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs)
  File "/home/ken/miniconda3/envs/web/lib/python3.7/site-packages/flask/views.py", line 158, in dispatch_request
    return current_app.ensure_sync(meth)(*args, **kwargs)
  File "/home/ken/Hack/studio/web/backend/app.py", line 73, in post
    kp_2d = camera.project(np.array(kp_3d["point"][:3]), np.linalg.inv(pose))
  File "/home/ken/miniconda3/envs/web/lib/python3.7/site-packages/straylib/camera.py", line 23, in project
    out, _ = cv2.projectPoints(points, R, T_CW[:3, 3], self.camera_matrix, self.distortion)
cv2.error: OpenCV(3.4.2) /tmp/build/80754af9/opencv-suite_1535558553474/work/modules/calib3d/src/calibration.cpp:3296: error: (-215:Assertion failed) npoints >= 0 && (depth == 5 || depth == 6) in function 
'projectPoints'

Another thing I noticed is that it seems the javascript bundle for this is roughly 1.25mb. Any ideas which parts contribute the most to this? Seems like react + redux should be ~100kb minified. Does the minification really make that big a difference or is there some massive libs being included in there?

It seems it also eager loads every single image, which for my test scene is >100mb. Lazy loading could make sense, though i realize that this leads to a less smooth experience when scrubbing through the images.

[kekeblom]

Added some improvements to the styles. Here is how it now looks. There are still some issues related to the undo/redo logic. When adding points, these don't seem to be added to the undo stack and undoing then undoes setting the images.

[hietalajulius]

Looks good. I'll do some work on this starting with the styles.

The triangulation doesn't work on my setup. It fails with this error:

Traceback (most recent call last):
  File "/home/ken/miniconda3/envs/web/lib/python3.7/site-packages/flask/app.py", line 2073, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/ken/miniconda3/envs/web/lib/python3.7/site-packages/flask/app.py", line 1518, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/ken/miniconda3/envs/web/lib/python3.7/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/ken/miniconda3/envs/web/lib/python3.7/site-packages/flask/app.py", line 1516, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/ken/miniconda3/envs/web/lib/python3.7/site-packages/flask/app.py", line 1502, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/home/ken/miniconda3/envs/web/lib/python3.7/site-packages/flask/views.py", line 84, in view
    return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs)
  File "/home/ken/miniconda3/envs/web/lib/python3.7/site-packages/flask/views.py", line 158, in dispatch_request
    return current_app.ensure_sync(meth)(*args, **kwargs)
  File "/home/ken/Hack/studio/web/backend/app.py", line 73, in post
    kp_2d = camera.project(np.array(kp_3d["point"][:3]), np.linalg.inv(pose))
  File "/home/ken/miniconda3/envs/web/lib/python3.7/site-packages/straylib/camera.py", line 23, in project
    out, _ = cv2.projectPoints(points, R, T_CW[:3, 3], self.camera_matrix, self.distortion)
cv2.error: OpenCV(3.4.2) /tmp/build/80754af9/opencv-suite_1535558553474/work/modules/calib3d/src/calibration.cpp:3296: error: (-215:Assertion failed) npoints >= 0 && (depth == 5 || depth == 6) in function 
'projectPoints'

Which scene is this so I can try to repro?

Another thing I noticed is that it seems the javascript bundle for this is roughly 1.25mb. Any ideas which parts contribute the most to this? Seems like react + redux should be ~100kb minified. Does the minification really make that big a difference or is there some massive libs being included in there?

Hmm I did not look into this at all, not really sure what may contribute to it. I'll see if there's an easy way to see what contributes most.

It seems it also eager loads every single image, which for my test scene is >100mb. Lazy loading could make sense, though i realize that this leads to a less smooth experience when scrubbing through the images.

Yeah I did that on purpose, it was not very smooth when only loading when sliding through the frames. Maybe a "hack" could be to resize the images on the backend. While the app runs only locally, this probably won't be a huge issue but probably sucks if the backend runs somewhere else than the user's local machine, let's figure something out

There are still some issues related to the undo/redo logic. When adding points, these don't seem to be added to the undo stack and undoing then undoes setting the images.

Hmm everything that gets added to the store should be undoable 🤔 But yeah since the images are initially added to the store, undo will remove them from the current state. I'm sure there's a way to circumvent this, let's find out 👍

[kekeblom]

This was on the scoots/0008 scene.

[hietalajulius]

This was on the scoots/0008 scene.

Hmm, for me this works:

https://user-images.githubusercontent.com/4254623/152824890-0016af73-cb04-47a9-8fda-9643aaca3142.mp4

Are you able to repro with a specific pair of frames?

[hietalajulius]

Closing for now, branch still exists