search

microsoft / Azure-Kinect-Sensor-SDK

A cross platform (Linux and Windows) user mode SDK to read data from your Azure Kinect device.
https://Azure.com/Kinect
MIT License
1.47k stars 613 forks source link

[Remote] depth_engine_start_helper reports error 204 ... GPU API failure ... at start of streaming #810

Open StevenButner opened 4 years ago

StevenButner commented 4 years ago

I have a 2 synchronized Azure Kinects and a fully working test setup. When I attempt to integrate the Kinects (on the same hardware system where my test works) within a larger real-time application, the cameras consistently fail at the moment when k4a_device_start_cameras() is called. I get an error message on stdout as below:

[2019-10-01 11:49:11.734] [error] [t=4190] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/dewrapper/dewrapper.c (154): depth_engine_start_helper(). Depth engine create and initialize failed with error code: 204.

The wrapper layer that handles all calls to/from the k4a library routines is the same as the one I use in my test program .... and the test program works perfectly. It seems to me that this issue must be resource-related or possibly a bug.

My host system is an Intel Core-i7 with 16 Gbyte RAM running Ubuntu 18.04LTS The SDK is version 1.2
Kinect device firmware is as below: == Azure Kinect DK Firmware Tool == Device Serial Number: 000261592512 Current Firmware Versions: RGB camera firmware: 1.6.102 Depth camera firmware: 1.6.75 Depth config file: 6109.7 Audio firmware: 1.6.14 Build Config: Production Certificate Type: Microsoft

Device Serial Number: 000441592912 Current Firmware Versions: RGB camera firmware: 1.6.102 Depth camera firmware: 1.6.75 Depth config file: 6109.7 Audio firmware: 1.6.14 Build Config: Production Certificate Type: Microsoft

I don't know if this is useful, but I tried running the program "deversion" from the SDK and it produced the following report: ~/Azure-Kinect-Sensor-SDK/src$ ../build/bin/deversion [2019-10-01 12:59:13.968] [error] [t=5362] ../src/dynlib/dynlib_linux.c (86): dynlib_create(). Failed to load shared object libdepthengine.so.0.0 with error: libdepthengine.so.0.0: cannot open shared object file: No such file or directory [2019-10-01 12:59:13.968] [error] [t=5362] ../src/dynlib/dynlib_linux.c (86): dynlib_create(). Failed to load shared object libdepthengine.so.1.0 with error: libdepthengine.so.1.0: cannot open shared object file: No such file or directory Found Depth Engine Pluging Version: 2.0.0 (current)

Any suggestions for getting past this issue would be greatly appreciated.

   Steve B
wes-b commented 4 years ago

It seems like you are missing libdepthengine.so.1.0, see https://github.com/microsoft/Azure-Kinect-Sensor-SDK/blob/develop/docs/depthengine.md.

RoseFlunder commented 4 years ago

In which environment are you starting the application? Logged on session with a running X-Server or headless/ssh session? If its the latter, you need to export the a DISPLAY variable before starting the application like this:

export DISPLAY:=1

StevenButner commented 4 years ago

My application doesn't have any screens. It's a robotic application that uses the depth and intensity cameras for guidance. No screens, means no X server, hence no DISPLAY variable. I guess it may fit your categorty "headless" but it is not in any ssh session.

RoseFlunder commented 4 years ago

Does it work when you export a display variable with the aforementioned command? Without a DISPLAY variable set "k4a_device_start_cameras()" will fail on Ubuntu.

StevenButner commented 4 years ago

I will have to do some reading to learn how to set an environment variable. The process that owns the cameras is a systemd daemon, so I guess I will have to add some commands in its services file or such. I'll try to figure it out and, if it works, I'll close this ticket. If not, I'll let you know. Thnx

RoseFlunder commented 4 years ago

This could help you I think: https://github.com/microsoft/Azure-Kinect-Sensor-SDK/issues/562#issuecomment-536323723

EDIT: @wes-b I saw this issue popping up already 2 or 3 times now and I had it myself (in my case with a ssh-session). Would it either be possible to have a proper error message for Ubuntu systems where the necessary environment variables are not set or at least provide a hint in the documentation? Maybe here: https://github.com/microsoft/Azure-Kinect-Sensor-SDK/blob/develop/docs/usage.md#linux-device-setup

In robotics applications it will be common to not have a X-Server running.

StevenButner commented 4 years ago

Agree with @wes-b suggestion, which would make it easier for a user to understand what the user is. But I am not (yet) able to solve the problem. I need to set an environment variable within my systemd daemon. I've read through #562 and several related threads. As I understand it, systemd runs its programs, by default, as root. Mine also needs to run as root so, until now, all has worked.

The requirement for an environment variable seems to require a user and a working directory. Once you put in a user, it shouldn't be root according to some of the contributors on the threads I've read .... though apparently it can be root. The additional requirement, however, that we set the particular environment variable XDG_RUNTIME_DIR=/run/user/1000 <<where the "1000" has to be changed to match the real user id>> makes this a bit more challenging. On my system, even though there of course is a "root" user, and even though the root user has the id of 0, there is no directory "/run/user/0" so I cannot do what is being suggested ... sigh.

Additionally, I don't know whether the DISPLAY=:0 or DISPLAY=:1 is required. One person told me DISPLAY:=1 and I think that may have been a typo. Another says to use DISPLAY=:0. I've tried all of these but I don't think they are reason for my continued issue. Systemd doesn't seem to like me to set User=root and even when that goes through, I do not see any directory /run/user/0

Any further suggestions for this situation?

wes-b commented 4 years ago

Would it either be possible to have a proper error message for Ubuntu systems where the necessary environment variables are not set or at least provide a hint in the documentation? Maybe here: https://github.com/microsoft/Azure-Kinect-Sensor-SDK/blob/develop/docs/usage.md#linux-device-setup

We are not planning to fix the root cause as it is a limitation in OpenGL. We are also working to see what we can update documentation to recomend better ways to get past this more quickly.

We will use this issue to at a minimum make the error message clearer. Is there an API that can be called to query for the fact there is no display session?

@rabbitdaxi FYI

RoseFlunder commented 4 years ago

@StevenButner Unfortunately I don't have a device myself to test it right now. But tomorrow I may test to create a systemd service, like in the comment I linked, and report if it works. I will be using an Intel NUC PC with:

EDIT: Couldn't really get it working flawlessly. Might also have to do sth. with using an external GPU on my side.

StevenButner commented 4 years ago

And, what would be most useful (at least for me) would be if you could have the systemd program you write run as root.

StevenButner commented 4 years ago

I'm still struggling with this GPU API error 204 thing. I've had to rearrange things a bit so that my application can run without being root (a good thing in the long term, no doubt). And, I've followed very closely the sample given in #562. I can launch my systemd service and it starts running. It opens the depth cameras and checks their serial numbers, etc. as part of its initialization. I can see a record of that in my log. But the instant it starts either camera streaming, get the same 204 error code and message.

My systemd unit file is here. `[Unit] Description=Navigation Manager (navmgr) After=network.target

[Service] Environment=DISPLAY=:0 Environment=XDG_RUNTIME_DIR=/run/user/990 User=rp9 Group=rp9 Type=simple ExecStart=/opt/bin/navmgr Restart=on-failure WorkingDirectory=/tmp

[Install] WantedBy=multi-user.target ` The user that I created is "rp9" and it has id 990. I logged into it via ssh from another computer so that the /run/user/990 path got created. Note that, I do not yet understand how to do this without an external login (i.e. under the hood via systemd). Perhaps my problem is that the external login owns the DISPLAY :0 ?? I did also try DISPLAY=:1 however, with the same result.

Any suggestions of what to try next would be greatly appreciated.

gkrasin commented 4 years ago

We are not planning to fix the root cause as it is a limitation in OpenGL.

Hi @wes-b!

how important is the dependency on OpenGL? Why is it there in the first place? Do you rely on OpenGL to process depth or it's just for some non-core functionality (like visualization)?

Would it be possible to eliminate this dependency?

RoseFlunder commented 4 years ago

I am thinking auto starting as systemd service might not be possible because systemd services are ment to be run before a users login into a graphical session or am I wrong? I am no Linux expert when it comes to this.

wes-b commented 4 years ago

how important is the dependency on OpenGL? Why is it there in the first place? Do you rely on OpenGL to process depth or it's just for some non-core functionality (like visualization)?

@gkrasin OpenGL is essential to the process of converting the raw depth data into images, which we do on the GPU

gkrasin commented 4 years ago

@wes-b got it, it makes sense now. Thank you for your insightful reply!

wes-b commented 4 years ago

796

haljarrett commented 4 years ago

This is a late comment, but it looks like EGL provides a way to get an OpenGL context in headless environments w/o Xorg running:

https://devblogs.nvidia.com/linking-opengl-server-side-rendering/ https://devblogs.nvidia.com/egl-eye-opengl-visualization-without-x-server/ https://stackoverflow.com/a/3331769/7553580

I am working in robotics and working on integrating the K4A into a headless rover system (Jetson Xavier / ROS). The workaround of allowing the window manager / X to run in the background and just setting the DISPLAY environmental variable is working for me, but ideally we would run completely headless. The articles linked make is seem like making the switch to EGL to grab OpenGL contexts is fairly straight forward, but since libdepthengine is closed source that is hard to know.

z33154 commented 4 years ago

I agree with haljarrett and the many others who have posted in issues #796, #681, #562, #503 and more. We could really benefit from a truly headless system. Our robots shouldn't need to run a windows manager. All of these issues come back to OpenGL while trying to run some form of a headless system.

emepetres commented 3 years ago

Hi!,

I have a tool that extracts color & depth images from a kinect recording that works fine in windows. I just stumbled with this issue, trying to port it to a docker container (Ubuntu18LTS) to run in a kubernetes cluster.

I tried just setting the DISPLAY=:0 or DISPLAY=:1 but no luck.

Is there a way to run k4a into a headless environment / headless docker container? If so, what would be the workaround?

Thanks!

wes-b commented 3 years ago

Is there a way to run k4a into a headless environment / headless docker container? If so, what would be the workaround?

For Docker: https://github.com/microsoft/Azure-Kinect-Sensor-SDK/issues/1258#issuecomment-648943546

emepetres commented 3 years ago

Thanks @wes-b for answering.

I can indeed run k4a in a docker container with gpu by mounting the X11 socket. However this solution is not feasible for a cloud/headless environment, as a display is needed to be running in the host.

I've done a deep research in the past few days to overcome the issue, that I post here for reference and to see if anyone can shed some light on it:

These are all the options I've explored for now with little to none success:

Currently I have these other two options that may work:

In the following days I'll try the azure batch approach, as it seems to be easier than using aks-engine.

However, I understand the issue could be easily solved if libdepthengine created a EGL headless context, as posted by others before. @wes-b , any idea of working in that direction?

StevenButner commented 3 years ago

@emepetres .... Your discussion of a docker-enabled GPU supporting the K4A needs is very interesting. Thanks for sharing that. I want to investigate it and try to get it running on my own system. I may have some questions for you in that regard which I will likely send via email.

The description of your various attempts to get the K4As to work in a headless environment are very familiar to me. I can report that, at this point, I have a working Ubuntu 18.04-based headless robotic system that uses 2 K4A cameras to sense depth in an autonomous navigation application. Many of the items that you list were also tried during development of my system .... with similar frustration and resulting failures.

I believe that the most important factor for getting a headless system up is figuring out how to boot up your system with a logged-in user. That's the area where I struggled .... because the investigation led to a number of complex issues bridging such topics as authentication (i.e. the PAM subsystem), session and 'seat' management, all manner of permission-related issues, and lots of systemd experimentation. In the end, I discovered that all I really needed was to install the "nodm" package. This is a full-featured display manager for headless systems (I.e. for systems with no display!). It has the all-important feature that most other display managers have .... configurable autologin. Nodm has a configuration file in /etc/default/nodm. In that file, you can configure the autologin feature. The comments in the /etc/default/nodm file are clear and helpful. There you can specify what userid to autologin and how to fire up the X11 session, including the X11 server options.

With nodm installed and enabled, I can boot up my system without any terminal attached. Everything boots up and a single special userid is automatically logged in and attached to an X11 server. I created some systemd user-mode units to attach to the X11 server for the special userid that I have configured for autologin. These systemd user-mode units get launched by systemd once that userid is logged in ... and that happens due to nodm (at boot time). Coupled with this, I have some regular systemd units for other daemons used in my system. These daemons are in the same group and I use group permission settings to maintain appropriate access and protection of other sensor subsystems in my application.

Note that I can log into my system over a wireless ssh session at any time. I routinely do my development over such an arrangement. If I give the command "loginctl list-sessions", I can see my own login session and one other one .... the userid that got autologged in. It always shows up with seat0 and has TTY "???".

I believe that the session setup --- in particular, the environment in /run/user/ ---- is the missing link that is needed for running any application that needs to stream the K4A depth camera. Once I managed to have a logged-in userid in place, together with its runtime environment, then all of the K4A library functionality worked for me. It all gets hooked up in the systemd unit that launches the application. This unit has to define at least the items shown below:

`[Service] Environment=DISPLAY=:10 <<<--- this is what I use, but it can be different Environment=XDG_RUNTIME_DIR=/run/user/ <<<--- numeric userid of your autologged in user goes here Environment=MESA_LOADER_DRIVER_OVERRIDE=iris <<<--- this is needed on my system which has Intel GPU

User= <<<--- put name of your autologged in userid here Group= <<<--- put your own defined group number here Type=simple Restart=on-failure WorkingDirectory=/tmp <<<<-- you can change this as needed ExecStart= <<<--- path to your executable daemon code here ` I hope this helps. It works well for me.

emepetres commented 3 years ago

Thank you @StevenButner for sharing that, the nodm setup is very interesting.

However if I understand correctly it is still not possible to use it in a nvidia/cuda docker container (nowadays) in a truly headless host, because nodm still needs an X server running in the background. Currently nvidia/cuda containers does not work with an X server running inside because the nvidia driver is not installed in the container (the host driver is used instead).

FYI looking at nodm I found that the author has deprecated it in favor of a lightdm autologin plugin

SteveButner commented 3 years ago

Thanks for the response, @emepetres. I was aware that nodm was deprecated and, in fact, lightdm would almost certainly work in its place. I like the simplicity and smaller size of nodm. As for the docker container not supporting an X11 server inside, that may not be a show-stopper, since you have already demonstrated that the OpenGL environment is apparently available within a docker container. You have shown a run with the k4aviewer application, which supports depth streaming.

I haven't tried this, but I think it is likely that the nodm auto-login could be configured to work without the X11 server launch (in a manner more like a tty). We really don't need the full set of features that X11 supports anyway. Apparently, the nvida/cuda docker technology has enough built-in capability that the K4A functions can run. In this area, all of the details count (and I don't know enough of them at this time to make any speculations).

Can you describe more about how the docker container offers OpenGL without launching an X11 server? And, for those of us who do not have nvidia GPU hardware but still have a headless system and a desire to run it all in a docker container, do you know whether the nvidia/cuda docker setup can be used with other GPUs? The systems I work with have the integrated Intel graphics controllers, e.g. Intel HD Graphics 530.

Added later: After doing some reading about nodm and about the nvdia/cuda docker setup, I now see that one thing I said above is not correct. Nodm does need to be setup with an X11 server. The attachment is quite flexible, however, due to the flexibility built into the launch methods for X11. But, it does need to have X11 services. I do not know whether or not this could be configured so as to attach the auto-login session to an X11 server running outside the container. That determination will require more research.

lesterlo commented 3 years ago

Some GPU Cards need to connect a display monitor to enable OpenGL. You cannot use any OpenGL program if the GPU card is not outputting display signal. This is a common issue in the headless computer system.

To solve this problem, You can purchase a dummy display plug to simulate a display monitor connection. No additional software needs to be installed.

You can buy something like this: dummy display plug

RoseFlunder commented 3 years ago

We got it working with this workaround:

  1. Enable automatic login for the user you are going to use. After that we could shutdown the pc, disconnect the monitor and power it on again. With the automatic login the x-server session will be created.
  2. After connecting via ssh set the Display env variable with: export DISPLAY=:0
  3. Start the software using the k4a sdk as usual
qm13 commented 3 years ago

@RoseFlunder thanks for this work around. One of our team also suggested using command line to enable automatic login. You can find instructions herehttps://vitux.com/how-to-enable-disable-automatic-login-in-ubuntu-18-04-lts/

christian-rauch commented 3 years ago

I am facing the same issue when trying to run the sensor on a robot.

I don't get why the OpenGL dependency would require a display manager. Headless OpenGL configurations have been demonstrated a couple of times before, e.g.: https://developer.nvidia.com/blog/egl-eye-opengl-visualization-without-x-server/.

Isn't there any way to support headless OpenGL modes via EGL?

lukexi commented 3 years ago

+1, would be very grateful to be able to run the Azure Kinect without X11 running. Vulkan has stable & well supported provisions for X11-less GPU usage, if that's an option.

lukexi commented 3 years ago

Hi, just checking in on this! Is there a plan/possibility to allow headless processing via OpenGL EGL as @christian-rauch suggests, or via Vulkan?

qm13 commented 3 years ago

@lukexi we are actively working on an EGL version of the depth engine. This will be available in the next release of the Sensor SDK.

lukexi commented 3 years ago

Fantastic news! Thanks @qm13 !!

zapaishchykova commented 3 years ago

hello, I encounter the same error with Jetson Nano in headless mode and 4 kinects What I Have tried and what did not work:

  1. Setting export DISPLAY 1 or 0
  2. Updating SDK packages to 1.4.1 from https://packages.microsoft.com/ubuntu/18.04/prod/pool/main/libk/libk4a1.4-dev/
  3. Buying one of the dummy display plugs also did not resolve the issue.

Any help would be really appreciated!

[LOG]: Number of connected devices  3
[2021-05-17 16:17:57.828] [error] [t=7603] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/dewrapper/dewrapper.c (154): depth_engine_start_helper(). Depth engine create and initialize failed with error code: 204.
[2021-05-17 16:17:57.829] [error] [t=7603] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/dewrapper/dewrapper.c (160): deresult == K4A_DEPTH_ENGINE_RESULT_SUCCEEDED returned failure in depth_engine_start_helper()
[2021-05-17 16:17:57.829] [error] [t=7603] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/dewrapper/dewrapper.c (194): depth_engine_start_helper(dewrapper, dewrapper->fps, dewrapper->depth_mode, &depth_engine_max_compute_time_ms, &depth_engine_output_buffer_size) returned failure in depth_engine_thread()
[2021-05-17 16:17:57.829] [error] [t=7599] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/dewrapper/dewrapper.c (552): dewrapper_start(). Depth Engine thread failed to start
[2021-05-17 16:17:57.829] [error] [t=7599] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth/depth.c (398): dewrapper_start(depth->dewrapper, config, depth->calibration_memory, depth->calibration_memory_size) returned failure in depth_start()
[2021-05-17 16:17:57.829] [error] [t=7599] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth_mcu/depth_mcu.c (359): cmd_status == CMD_STATUS_PASS returned failure in depthmcu_depth_stop_streaming()
[2021-05-17 16:17:57.829] [error] [t=7599] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth_mcu/depth_mcu.c (362): depthmcu_depth_stop_streaming(). ERROR: cmd_status=0x00000063
[2021-05-17 16:17:57.829] [error] [t=7599] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/sdk/k4a.c (895): depth_start(device->depth, config) returned failure in k4a_device_start_cameras()
Start K4A cameras failed!
  File "capture_mult_streams.py", line 34, in <module>
    pyK4A.device_start_cameras(device_config)
  File "../pyKinectAzure/pyKinectAzure.py", line 151, in device_start_cameras
    _k4a.VERIFY(self.k4a.k4a_device_start_cameras(self.device_handle,self.config.current_config),"Start K4A cameras failed!")
  File "../pyKinectAzure/_k4a.py", line 570, in VERIFY
    traceback.print_stack()
christian-rauch commented 3 years ago

Any help would be really appreciated!

I see that only 3 of the 4 cameras were detected. Is this maybe a bandwidth issue with the fourth camera?

To verify that this is not your code or the bandwidth limitations, could you start the azure kinect viewer with only a single camera connected?: DISPLAY=:1 k4aviewer

I am using a Kinect in a setup with a connected, but powered-off screen, and an Xserver starting on boot. I noticed that the Xserver will not start automatically if no screen had been detected. I've no dummy connector yet to try this.

I've set DISPLAY in the .bashrc so that when I log in via ssh and without X forwarding, then azure kinect viewer or the ROS driver will just start without issues. It's indeed important to not forward X (i.e. do not use -X) so that the viewer or driver connect to the local Xserver via the DISPLAY first. This way you will not be able to see any GUI, which is ok if you just run the driver. Otherwise you have to setup VNC.

jaygullapalli commented 3 years ago

Hello zapaishchykova,

From the logs it looks like OpenGL context is not available, which is why depth engine initialization fails. Did you enable autologin for the user account that you SSH with? The steps for headless operation are listed here: https://docs.microsoft.com/en-us/azure/Kinect-dk/troubleshooting#using-azure-kinect-on-headless-linux-system

Going through the specifications of Jetson Nano, it looks like all the 4 USB 3.0 Type A ports are mapped on to the same USB controller (which means all the ports share the same underlying controller and its bandwidth). Azure Kinect DK when streaming data in high resolution mode (Depth + Color) use most of the USB 3.0 bandwidth. If you are planning for concurrent operation of all cameras, then you'd have to lower the frame rate down to 5 FPS. I suspect the issue of missing 4th camera is due to HW resources on the controller (and not an SDK issue). You can use lsusb to list the USB devices, see if all 4 cameras enumerate properly and made available by the OS.

zapaishchykova commented 3 years ago

Dear @jaygullapalli and @christian-rauch, Maybe my logs were a bit confusing but I have no problems with detecting all 4 cameras - just in this specific test run i connected 3 of them. I increased Jetson power supply and I am capturing non-real-time data - which means fps lower than 5.

Also, my code works with the display connected and all 4 cameras - so this only happens when I try to run in with a fake screen/ no screen, even with one kinect. My auto ssh login is enabled

When I use my fake display, i can see its there from terminal:

xrandr | grep " connected " | awk '{ print$1 }'
HDMI-0

For reference, this is what happens with one connected camera ( same as with 2 or 3):

[LOG]: Number of connected devices  1
[2021-05-25 09:50:05.322] [error] [t=10647] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/dewrapper/dewrapper.c (154): depth_engine_start_helper(). Depth engine create and initialize failed with error code: 204.
[2021-05-25 09:50:05.322] [error] [t=10647] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/dewrapper/dewrapper.c (160): deresult == K4A_DEPTH_ENGINE_RESULT_SUCCEEDED returned failure in depth_engine_start_helper()
[2021-05-25 09:50:05.322] [error] [t=10647] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/dewrapper/dewrapper.c (194): depth_engine_start_helper(dewrapper, dewrapper->fps, dewrapper->depth_mode, &depth_engine_max_compute_time_ms, &depth_engine_output_buffer_size) returned failure in depth_engine_thread()
[2021-05-25 09:50:05.322] [error] [t=10635] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/dewrapper/dewrapper.c (552): dewrapper_start(). Depth Engine thread failed to start
[2021-05-25 09:50:05.322] [error] [t=10635] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth/depth.c (398): dewrapper_start(depth->dewrapper, config, depth->calibration_memory, depth->calibration_memory_size) returned failure in depth_start()
[2021-05-25 09:50:05.322] [error] [t=10635] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth_mcu/depth_mcu.c (359): cmd_status == CMD_STATUS_PASS returned failure in depthmcu_depth_stop_streaming()
[2021-05-25 09:50:05.322] [error] [t=10635] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth_mcu/depth_mcu.c (362): depthmcu_depth_stop_streaming(). ERROR: cmd_status=0x00000063
[2021-05-25 09:50:05.323] [error] [t=10635] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/sdk/k4a.c (895): depth_start(device->depth, config) returned failure in k4a_device_start_cameras()
Start K4A cameras failed!
  File "capture_mult_streams.py", line 34, in <module>
    pyK4A.device_start_cameras(device_config)
  File "../pyKinectAzure/pyKinectAzure.py", line 151, in device_start_cameras
    _k4a.VERIFY(self.k4a.k4a_device_start_cameras(self.device_handle,self.config.current_config),"Start K4A cameras failed!")
  File "../pyKinectAzure/_k4a.py", line 570, in VERIFY
    traceback.print_stack()
zapaishchykova commented 3 years ago

Huh interesting, when I start the session with a connected display and set export DISPLAY=:0 everything works even when i afterwards disconnect the screen .... I did those steps to set up the autologin https://vitux.com/how-to-enable-disable-automatic-login-in-ubuntu/, is there something that I'm missing?

UPDATES: okay, so here is the thing: I can capture with 3 Kinect via ssh connection with no display connected. But when I connect the 4th one the connection to the Jetson drops. I am using wifi-USB-dongle with this fix: https://github.com/cilynx/rtl88x2BU_WiFi_linux_v5.3.1_27678.20180430_COEX20180427-5959 . I guess you are right, something is wrong with bangwidth of USBs of the Kinekts ...

challenger71498 commented 2 years ago

For those who uses docker image for Azure Kinect, here's how I solved this '204' issue.

This might not be a proper solution for headless system, since I just used docker with a normal desktop configuration. But still docker container is a headless system afaik, there may be a relevance.

TL; DR

Add options below, on running docker image.

--volume /tmp/.X11-unix:/tmp/.X11-unix:ro # mount x11 server.
-e DISPLAY=unix$DISPLAY # set environment value for display.

Or, if you use docker-compose yaml file, add below at your service configuration.

# ...
  volumes:
     - /tmp/.X11-unix:/tmp/.X11-unix:ro
  environment:
    DISPLAY=unix$DISPLAY
# ...

Lastly, on your host computer, open terminal and type below:

$ xhost +local:docker

You might need to execute this bash script on every boot.

More details here.

My Situation

Device

Development Environment

In short,

So, I tried...

1. Run docker image configured by docker-compose, with systemd.

2. Run docker image by Startup Application.

3. What if I open Gnome Terminal, and run scripts on there? (With xhost configuration)

Conclusion

SoylentGraham commented 2 years ago

@lukexi we are actively working on an EGL version of the depth engine. This will be available in the next release of the Sensor SDK.

Has there been any update on an EGL version? Are we still currently on a 2020 release?

rajkundu commented 2 years ago

Same question as above; any progress on this front?

christian-rauch commented 2 years ago

@rajkundu As you can see from the git history, the last commit was on the 20th Oct 2021. That means there has been no public progress on this since October last year. I also don't see a public roadmap either.

The headless EGL feature seems to be of low priority on Microsoft's backlog. You have to chaise @qm13 to get some official statement or rely on the community to implement headless EGL.

SoylentGraham commented 2 years ago

or rely on the community to implement headless EGL

This part is closed source. (If I'm wrong, point me in the right direction, and I'll work on it, I can find the time as the X11 reliance means I can't get solid colour&depth from a nano, and stuck with windows/bigger devices)

rajkundu commented 2 years ago

or rely on the community to implement headless EGL

This part is closed source. (If I'm wrong, point me in the right direction, and I'll work on it, I can find the time as the X11 reliance means I can't get solid colour&depth from a nano, and stuck with windows/bigger devices)

This is also my understanding: that the depth engine contains "proprietary code" and thus the community cannot contribute to or fix it.

rajkundu commented 2 years ago

@qm13 Do you have any rough idea, at all, of when the next release might be? Any information you could provide would be greatly appreciated!

rajkundu commented 2 years ago

Unlike a few others in this thread, in my case, I didn't need to use the Azure devices live with a headless host. Rather, my group was trying to remotely analyze footage that was already recorded into .mkv files. We got so many errors because of OpenGL requiring proper display variables, etc., including this 204 error. In the end, we did get the SDK working with X forwarding over SSH. However, an even simpler solution we found was to start a Jupyter Lab instance locally, then port forward/expose the port so that we could use it remotely via HTTP, yet all of the processing is done locally. I'm happy to help however I can, so if anyone's use case is similar to mine, please feel free to reach out!

christian-rauch commented 2 years ago

Rather, my group was trying to remotely analyze footage that was already recorded into .mkv files. We got so many errors because of OpenGL requiring proper display variables, etc., including this 204 error.

You shouldn't need the driver to read the mkv files. It's a standard format that many video players and libraries can read. You could even stream them with VLC from a headless system over the network.

rajkundu commented 2 years ago

Without the SDK's Depth Engine, while I can easily extract the raw 2D images from the MKV files (and have done so already), they are of little use to me. My understanding is that the proprietary Depth Engine is required to convert/reconstruct this data into 3D. Is this not correct?

christian-rauch commented 2 years ago

My understanding is that the proprietary Depth Engine is required to convert/reconstruct this data into 3D. Is this not correct?

Not exactly. The mkv file stores multiple streams:

$ mediainfo OFFICE_SAMPLE_NFOV_UNBINNED.mkv | grep Title
Title                                    : COLOR
Title                                    : DEPTH
Title                                    : IR
Title                                    : IMU

You already get the depth stream. To get the point cloud, you only have to project the depth using the camera intrinsics. You can do this headless with OpenCV or just implement that yourself.

Edit: To clarify, the Depth Engine is used to compute the depth from the IR. It is not used to compute the 3D points from the depth.

You could simplify this a bit by using the ROS (1 or 2) node and recording the point cloud directly to a ROS bag file. Recording will require the depth engine too, but once you have the ROS bag you can essentially treat it like any other log file and read the point clouds from it via any of the supported languages (Python, C++, ...).

rajkundu commented 2 years ago

Amazing! That makes a lot of sense. Thank you so much!

rajkundu commented 2 years ago

@christian-rauch So, I'm just wondering... with pre-recorded footage, is there a reason the SDK requires OpenGL/X/Depth Engine for retrieving 3D data, but not for accessing 2D data? Is it just a byproduct of the way that it is coded? I.e., in theory, could it also have been made in such a way that it does all of these 3D operations without requiring such dependencies?