vicalib IR Issue

[artforlife]

We are working on a set up using the ARPG library. After much playing around, we finally got all libraries and dependencies to work properly. However, one issue persists. Namely, when launching the calibration routine with the following command:

sudo ./vicalib_exec -grid_preset small -cam convert:[range=ir2]//freenect2:[rgb=0,ir=1,depth=0]

We see the screen:

Upon opening the cameras.xml file, we see that the RGB camera (index =0) is calibrated for distortion, whereas the IR camera (index=1) is not. The example of cameras.xml is pasted below. Such issue persists for various combinations of [range=] and [rgb=0,ir=1,depth=0]. What is more, it appears that RGB camera calibrates even when its boolean value is set to zero as illustrated by the command above. Does any one know what could be causing this? We have 3 machines that have this issue.

P.S. Having looked into this for some time, we were able to get IR camera to work on one machine. However, on this machine, the vicalib window is not responding to mouse or keyboard. That is, once it is launched, it can only be killed from the terminal window. We are unsure whether these are different sides of the same coin.

<rig>
    <camera>
        <camera_model name="" index="0" serialno="220864634347" type="calibu_fu_fv_u0_v0_k1_k2_k3" version="-189929152">
            <width> 1920 </width>
            <height> 1080 </height>
            <!-- Use RDF matrix, [right down forward], to define the coordinate frame convention -->
            <right> [ 1; 0; 0 ] </right>
            <down> [ 0; 1; 0 ] </down>
            <forward> [ 0; 0; 1 ] </forward>
            <!-- Camera parameters ordered as per type name. -->
            <params> [ 1088.496; 1090.916; 934.7571; 497.7893; 0.03453317; -0.006876086; -0.02275644 ] </params>
        </camera_model>
        <pose>
            <!-- Camera pose. World from Camera point transfer. 3x4 matrix, in the RDF frame convention defined above -->
            <T_wc> [ 1, 0, 0, 0; 0, 1, 0, 0; 0, 0, 1, 0 ] </T_wc>
        </pose>
    </camera>
    <camera>
        <camera_model name="" index="1" serialno="220864634347" type="calibu_fu_fv_u0_v0_k1_k2_k3" version="0">
            <width> 512 </width>
            <height> 424 </height>
            <!-- Use RDF matrix, [right down forward], to define the coordinate frame convention -->
            <right> [ 1; 0; 0 ] </right>
            <down> [ 0; 1; 0 ] </down>
            <forward> [ 0; 0; 1 ] </forward>
            <!-- Camera parameters ordered as per type name. -->
            <params> [ 300; 300; 256; 212; 0; 0; 0 ] </params>
        </camera_model>
        <pose>
            <!-- Camera pose. World from Camera point transfer. 3x4 matrix, in the RDF frame convention defined above -->
            <T_wc> [ 1, 0, 0, 0; 0, 1, 0, 0; 0, 0, 1, 0 ] </T_wc>
        </pose>
    </camera>
</rig>

[Algomorph]

We're currently inspecting what is different between three machines. There seem to be two different and unrelated problems: 1) Symptomatic of the incorrect IR feed as shown above, which makes it impossible for us to calibrate the IR feed. Note that SensorViewer example seems to produce the correct IR image regardless. 2) No user input being registered by the windows for SensorViewer app in HAL or vicalib (probably the rest of the applications as well).

Currently, we have problems 1 & 2 affecting machine A, problem 1 affecting machine B, and problem 2 affecting machine C.

Could you please post your output to "ldd vicalib_exec" on your Jenkins machine, so we can narrow down the issue?

[crheckman]

Thanks for bringing these issues up.

Regarding problem 1, we haven't updated our OpenNI drivers here in a long time (over a year) and it's possible their API, especially how to identify and enable/disable certain cameras, has changed. That would be a good first place to start tracking the issue. Eventually we will update, but we haven't yet.

Regarding problem 2, we haven't seen this problem before. It's possible Pangolin, the library we use to draw windows, is not compatible with the newest GLUT. That said, the upstream developer for Pangolin tends to be very good about this sort of thing so it's possible our forked copy is what's not compatible.

To respond to your query:

$ ldd vicalib_exec
linux-vdso.so.1 =>  (0x00007fff65d94000)
    libgflags.so.2 => /usr/local/lib/libgflags.so.2 (0x00007ffcf5f85000)
    libglog.so.0 => /usr/local/lib/libglog.so.0 (0x00007ffcf5d57000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ffcf5b3d000)
    libvicalib.so => /home/arpg/Workspace/VICalib/Build/libvicalib.so (0x00007ffcf582c000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ffcf5527000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ffcf5311000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffcf4f4b000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ffcf4d2c000)
    /lib64/ld-linux-x86-64.so.2 (0x00007ffcf61a7000)
    libopencv_core.so.2.4 => /usr/local/lib/libopencv_core.so.2.4 (0x00007ffcf486d000)
    libopencv_imgproc.so.2.4 => /usr/local/lib/libopencv_imgproc.so.2.4 (0x00007ffcf437d000)
    libcalibu.so => /home/arpg/Workspace/Calibu/Build/libcalibu.so (0x00007ffcf411f000)
    libhal.so => /home/arpg/Workspace/HAL/Build/HAL/libhal.so (0x00007ffcf3c76000)
    libceres.so.1 => /usr/local/lib/libceres.so.1 (0x00007ffcf37ba000)
    libGL.so.1 => /usr/lib/x86_64-linux-gnu/mesa/libGL.so.1 (0x00007ffcf3554000)
    libGLEW.so.1.10 => /usr/lib/x86_64-linux-gnu/libGLEW.so.1.10 (0x00007ffcf32c8000)
    libpangolin.so => /home/arpg/Workspace/Pangolin/Build/libpangolin.so (0x00007ffcf3022000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ffcf2d1c000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ffcf2b13000)
    libopencv_calib3d.so.2.4 => /usr/local/lib/libopencv_calib3d.so.2.4 (0x00007ffcf2837000)
    libopencv_highgui.so.2.4 => /usr/local/lib/libopencv_highgui.so.2.4 (0x00007ffcf23da000)
    libprotobuf.so.9 => /usr/local/lib/libprotobuf.so.9 (0x00007ffcf20ca000)
    libnode.so => /home/arpg/Workspace/Node/Build/libnode.so (0x00007ffcf1de7000)
    libdc1394.so.22 => /usr/lib/x86_64-linux-gnu/libdc1394.so.22 (0x00007ffcf1b72000)
    liblapack.so.3 => /usr/lib/liblapack.so.3 (0x00007ffcf138c000)
    libf77blas.so.3 => /usr/lib/libf77blas.so.3 (0x00007ffcf116c000)
    libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007ffcf0f5c000)
    libglapi.so.0 => /usr/lib/x86_64-linux-gnu/libglapi.so.0 (0x00007ffcf0d35000)
    libXext.so.6 => /usr/lib/x86_64-linux-gnu/libXext.so.6 (0x00007ffcf0b23000)
    libXdamage.so.1 => /usr/lib/x86_64-linux-gnu/libXdamage.so.1 (0x00007ffcf091f000)
    libXfixes.so.3 => /usr/lib/x86_64-linux-gnu/libXfixes.so.3 (0x00007ffcf0719000)
    libX11-xcb.so.1 => /usr/lib/x86_64-linux-gnu/libX11-xcb.so.1 (0x00007ffcf0517000)
    libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007ffcf01e1000)
    libxcb-glx.so.0 => /usr/lib/x86_64-linux-gnu/libxcb-glx.so.0 (0x00007ffceffca000)
    libxcb-dri2.so.0 => /usr/lib/x86_64-linux-gnu/libxcb-dri2.so.0 (0x00007ffcefdc5000)
    libxcb-dri3.so.0 => /usr/lib/x86_64-linux-gnu/libxcb-dri3.so.0 (0x00007ffcefbc1000)
    libxcb-present.so.0 => /usr/lib/x86_64-linux-gnu/libxcb-present.so.0 (0x00007ffcef9be000)
    libxcb-sync.so.1 => /usr/lib/x86_64-linux-gnu/libxcb-sync.so.1 (0x00007ffcef7b8000)
    libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007ffcef598000)
    libxshmfence.so.1 => /usr/lib/x86_64-linux-gnu/libxshmfence.so.1 (0x00007ffcef396000)
    libXxf86vm.so.1 => /usr/lib/x86_64-linux-gnu/libXxf86vm.so.1 (0x00007ffcef190000)
    libdrm.so.2 => /usr/lib/x86_64-linux-gnu/libdrm.so.2 (0x00007ffceef83000)
    libglut.so.3 => /usr/lib/x86_64-linux-gnu/libglut.so.3 (0x00007ffceed39000)
    libopencv_flann.so.2.4 => /usr/local/lib/libopencv_flann.so.2.4 (0x00007ffceeac4000)
    libopencv_features2d.so.2.4 => /usr/local/lib/libopencv_features2d.so.2.4 (0x00007ffcee815000)
    libzmq.so.4 => /usr/local/lib/libzmq.so.4 (0x00007ffcee5c2000)
    libzmqpp.so => /usr/local/lib/libzmqpp.so (0x00007ffcee3a2000)
    libdns_sd.so.1 => /usr/lib/x86_64-linux-gnu/libdns_sd.so.1 (0x00007ffcee199000)
    libraw1394.so.11 => /usr/lib/x86_64-linux-gnu/libraw1394.so.11 (0x00007ffcedf8a000)
    libusb-1.0.so.0 => /lib/x86_64-linux-gnu/libusb-1.0.so.0 (0x00007ffcedd73000)
    libblas.so.3 => /usr/lib/libblas.so.3 (0x00007ffced7a7000)
    libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007ffced48d000)
    libcblas.so.3 => /usr/lib/libcblas.so.3 (0x00007ffced26c000)
    libatlas.so.3 => /usr/lib/libatlas.so.3 (0x00007ffceccd9000)
    libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007ffcecad4000)
    libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007ffcec8ce000)
    libXi.so.6 => /usr/lib/x86_64-linux-gnu/libXi.so.6 (0x00007ffcec6bd000)
    libavahi-common.so.3 => /usr/lib/x86_64-linux-gnu/libavahi-common.so.3 (0x00007ffcec4b1000)
    libavahi-client.so.3 => /usr/lib/x86_64-linux-gnu/libavahi-client.so.3 (0x00007ffcec29f000)
    libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007ffcec08e000)
    libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007ffcebe52000)
    libdbus-1.so.3 => /lib/x86_64-linux-gnu/libdbus-1.so.3 (0x00007ffcebc0c000)
    libcgmanager.so.0 => /lib/x86_64-linux-gnu/libcgmanager.so.0 (0x00007ffceb9f1000)
    libnih.so.1 => /lib/x86_64-linux-gnu/libnih.so.1 (0x00007ffceb7d8000)
    libnih-dbus.so.1 => /lib/x86_64-linux-gnu/libnih-dbus.so.1 (0x00007ffceb5ce000)

[Algomorph]

For problem 2, it seems like the GLUT library on both machines A & B is 3.9.0, and Pangolin is built from the same commit ac8289ae22. Hence, GLUT does not seem to be the issue.

I have just switched to the latest freenect2 driver on machine A using the depth registration fix for HAL. The depth feed seems much nicer on SensorViewer, but both problems 1 & 2 persist, so it looks like it's not related to that either.

Thank you for the dependency list, we will continue investigating.

[Algomorph]

For problem 1, likewise, all three machines are using the same version of OpenNI and (now again) the same version of HAL, but only machine C has the correct IR feed, so it seems to me that OpenNI is also not the issue.

[Algomorph]

https://github.com/arpg/Pangolin/commit/5c3c7074528e24b643a56ee21d4a27228d643c06

This commit causes problem 2 specified above. To make sure, if you clear your CMake cache on your jenkins machine, then you will also experience problem 2.

[Algomorph]

Problem 2 was resolved. Somehow, we got IR feed to work properly on machine A. At this point, we figured out how to run the calibration on 3-6 Kinect 2s from a single machine (C). I could not trace exactly what fixed problem 1 on machine A and reproduce this on machine B, but seems like we don't need to fix that right now.

18 actually breaks everything again, but that would be a separate issue.

Node/calibration with more than one aux. node also throws an error for us, but that's also a separate issue.

Should I close this for now, even though the IR feed problem remains?

[gsibley]

If you can let me know what exactly is breaking for you on #18 I'll fix it. We can also revert that merge if needed as the functionality is duplicated.

[Algomorph]

It might not be #18 but rather the most recent new update to HAL that breaks it. We currently don't have the time to look into that. We're just using the builds from previous commits. If we uncover what it is exactly, we'll let you know. If "problem 1" poses a barrier for us and we come to some more information about it, I can open a new issue as well. Likewise for the issue with Node. I think this thread is safe to close at this point.

[Algomorph]

Latest HAL changes (maybe, + the newest freenect2 driver?) miraculously fixed everything except the Node problem. We are now not experiencing "problem 1" on any of our machines. I will ask my colleague to close this when he becomes available.