Failed to load nodelet

[gui2dev]

Hi !

I'm using robot_upstart to load some nodelet at startup.

I'm getting a weird issue, it seems that it fails to load nodelet as I'm getting this kind of fatal error:

Node: /ueye_cam_nodelet_2
Time: 11:14:39.055841859 (2017-10-13)
Severity: Fatal
Published Topics: /rosout

Failed to load nodelet '/ueye_cam_nodelet_2` of type `ueye_cam/ueye_cam_nodelet` to manager `nodelet_manager'

Location:
/tmp/binarydeb/ros-kinetic-nodelet-1.9.10/src/nodelet.cpp:loadNodelet:228

----------------------------------------------------------------------------------------------------

Lauchingsudo job-start on foreground is working fine (no permission issue).

Here is my launch file:

<launch>

  <node name="check_ueye_api" pkg="ueye_cam" type="check_ueye_api" required="true" />
  <arg name="nodelet_manager_name" value="nodelet_manager" />
  <arg name="camera_name" value="camera" />

  <node pkg="nodelet" type="nodelet" name="$(arg nodelet_manager_name)" args="manager"/>

  <node pkg="nodelet" type="nodelet" name="ueye_cam_nodelet_1" respawn="true"
        args="load ueye_cam/ueye_cam_nodelet $(arg nodelet_manager_name)">
    <param name="camera_name" type="str" value="$(arg camera_name)1" /> <!-- == namespace for topics and services -->
    <param name="camera_topic" type="str" value="image_raw" />
    <param name="camera_id" type="int" value="1" /> <!-- 0 = any camera; 1+: camera ID -->
    <param name="camera_intrinsics_file" type="string" value="" /> <!-- default: ~/.ros/camera_info/<camera_name>.yaml -->
    <param name="camera_parameters_file" type="string" value="" /> <!-- default: ~/.ros/camera_conf/<camera_name>.ini -->

    <param name="ext_trigger_mode" type="bool" value="False" /> 

    <param name="color_mode" type="str" value="rgb8" />
    <param name="image_width" type="int" value="800" />
    <param name="image_height" type="int" value="1200" />
    <param name="image_top" type="int" value="-1" /> <!-- -1: center -->
    <param name="image_left" type="int" value="-1" /> <!-- -1: center -->

    <param name="subsampling" type="int" value="1" /> <!-- supported by only some UEye cameras -->
    <param name="binning" type="int" value="1" /> <!-- supported by only some UEye cameras -->
    <param name="sensor_scaling" type="double" value="1.0" /> <!-- supported by only some UEye cameras -->

    <param name="auto_gain" type="bool" value="True" />
    <param name="master_gain" type="int" value="0" />
    <param name="red_gain" type="int" value="0" />
    <param name="green_gain" type="int" value="1" />
    <param name="blue_gain" type="int" value="16" />
    <param name="gain_boost" type="bool" value="False" />

    <param name="auto_exposure" type="bool" value="True" />
    <param name="exposure" type="int" value="33" /> <!-- in ms -->

    <param name="auto_white_balance" type="bool" value="True" />
    <param name="white_balance_red_offset" type="int" value="0" />
    <param name="white_balance_blue_offset" type="int" value="0" />

    <param name="flash_delay" type="int" value="0" /> <!-- in us -->
    <param name="flash_duration" type="int" value="1000" /> <!-- in us -->

    <param name="auto_frame_rate" type="bool" value="False" />
    <param name="frame_rate" type="double" value="10.0" />
    <param name="output_rate" type="double" value="0.0" /> <!-- >0: throttle rate for publishing frames -->
    <param name="pixel_clock" type="int" value="24" />

    <param name="flip_upd" type="bool" value="True" />
    <param name="flip_lr"  type="bool" value="False" />
  </node>

<node pkg="nodelet" type="nodelet" name="ueye_cam_nodelet_2"  respawn="true"
        args="load ueye_cam/ueye_cam_nodelet $(arg nodelet_manager_name)">
    <param name="camera_name" type="str" value="$(arg camera_name)2" /> <!-- == namespace for topics and services -->
    <param name="camera_topic" type="str" value="image_raw" />
    <param name="camera_id" type="int" value="2" /> <!-- 0 = any camera; 1+: camera ID -->
    <param name="camera_intrinsics_file" type="string" value="" /> <!-- default: ~/.ros/camera_info/<camera_name>.yaml -->
    <param name="camera_parameters_file" type="string" value="" /> <!-- default: ~/.ros/camera_conf/<camera_name>.ini -->

    <param name="ext_trigger_mode" type="bool" value="False" /> 

    <param name="color_mode" type="str" value="rgb8" />
    <param name="image_width" type="int" value="800" />
    <param name="image_height" type="int" value="1200" />
    <param name="image_top" type="int" value="-1" /> <!-- -1: center -->
    <param name="image_left" type="int" value="-1" /> <!-- -1: center -->

    <param name="subsampling" type="int" value="1" /> <!-- supported by only some UEye cameras -->
    <param name="binning" type="int" value="1" /> <!-- supported by only some UEye cameras -->
    <param name="sensor_scaling" type="double" value="1.0" /> <!-- supported by only some UEye cameras -->

    <param name="auto_gain" type="bool" value="True" />
    <param name="master_gain" type="int" value="0" />
    <param name="red_gain" type="int" value="0" />
    <param name="green_gain" type="int" value="1" />
    <param name="blue_gain" type="int" value="16" />
    <param name="gain_boost" type="bool" value="False" />

    <param name="auto_exposure" type="bool" value="True" />
    <param name="exposure" type="int" value="33" /> <!-- in ms -->

    <param name="auto_white_balance" type="bool" value="True" />
    <param name="white_balance_red_offset" type="int" value="0" />
    <param name="white_balance_blue_offset" type="int" value="0" />

    <param name="flash_delay" type="int" value="0" /> <!-- in us -->
    <param name="flash_duration" type="int" value="1000" /> <!-- in us -->

    <param name="auto_frame_rate" type="bool" value="False" />
    <param name="frame_rate" type="double" value="10.0" />
    <param name="output_rate" type="double" value="0.0" /> <!-- >0: throttle rate for publishing frames -->
    <param name="pixel_clock" type="int" value="24" />

    <param name="flip_upd" type="bool" value="True" />
    <param name="flip_lr"  type="bool" value="False" />
  </node>

<node name="web_video" type="web_video_server" pkg="web_video_server" respawn="true">
<param name="type" value="vp8"/>
<param name="address" value="10.42.0.1"/>
</node>
 <node pkg="img_saver" type="img_saver_node" name="img_saver"/>
</launch>

[anqixu]

I will need more information to help you debug. Try to manually roslaunch your launch file, and paste the errors that you get. If you only get the error "Failed to load nodelet...", then I'd suspect that you did not install (apt-get or catkin make and/or source setup.bash) the package properly.

[gui2dev]

@anqixu I got no error while launching manually. It's only when it's launch on the background/ at startup that it does not work properly. So I guess something wrong with init / permission ? I'll try to get some more logs out. EDIT : package was installed using apt-get

[gui2dev]

Here is some logs: link

[anqixu]

I suspect that your upstart script is running as root (or another user than the one you use to manually test), and for this alternative user, ROS's setup.bash / ueye_cam package path are not set properly.

Try to sudo bash to get a fully-rootified shell, and then try roslaunch, followed by job-start. If all these fail, then I'm afraid that I don't have any more ideas on what's wrong (but, perhaps naively, I suspect that it's not due to ueye_cam).

[gui2dev]

well these two are working... It's only when using sudo service job start that it fails.

[gui2dev]

From all that I could investigate, it looks as if somehow libraries where not found when launched as a service.

[gui2dev]

Using cron at @reboot with the appropriate sh files, it's working fine.

[ejalaa12]

Hello @gui2dev , I ran into the same problem when trying to start a nodelet on start-up (using robot_upstart). Can you explain how you solved your problem ? Thank you :)

[gui2dev]

I did not solved it actually. I just used cron and some scripts. Found this in that post. Worked like a charm.

[ejalaa12]

Thanks, that looks great ! I will also use this solution.

[gui2dev]

@anqixu I solved the issue. The upstart script needs to wait for the ueye driver service to be up, which was not the case, hence the bug.

[mikepurvis]

@gui2dev Your use case may vary, but our preference at Clearpath has been to launch everything as early on possible on startup, and make it the responsibility of individual nodes/drivers to manage their own retry loops and publish diagnostics about the status of any external dependencies.

In a case like ours where it's an end-user robot, it's not realistic to have failure modes where the entirety of ROS just doesn't come up (or is delayed coming up) and the only way to figure out what went wrong is going in over SSH and examining logs.

[gui2dev]

I won't disagree with that.

[ejalaa12]

@gui2dev We also found out that the ueye driver service had to be up before the upstart service started. So we created our own services with systemd that made sure the ueye.service was started before.

But I'm curious, are you still using cron or did you find a way to make the upstart service wait for another one (i.e. the ueye.service)?

Also @mikepurvis , also out of curiosity, do you mean you would make every node loop check for the availability of the resources (i/o, serial, udp, service ...) it will use ?

[mikepurvis]

@ejalaa12 Yes, but obviously this mostly just matters for the cases where you actually have external dependencies, so basically, device drivers.

[gui2dev]

@ejalaa12 we use the same solution as you do. Specifying after ueye in the.service file.

[ejalaa12]

I see, thanks guys ! :)