Snapdragon: muORB doesn't properly support orb_exist

[RomanBapst]

This is working now on latest master.

[RomanBapst]

This is not working always.

[RomanBapst]

@julianoes I made an interesting observation. The topic is only logged once I print the topic using the listener tool. And the first print is all zeros (meaning the topic was not updated).

[RomanBapst]

@LorenzMeier @julianoes I did some more investigation. The problem seems to be that some topic do not seem to be in the devmap on the posix side. See below the list of topics on the posix and on the qurt side. I was assuming they should be matching but they obviously don't:

POSIX:

INFO  Devices:
INFO     /obj/_obj_
INFO     /obj/parameter_update0
INFO     /obj/vehicle_status0
INFO     /obj/vehicle_control_mode0
INFO     /obj/vehicle_command_ack0
INFO     /obj/mavlink_log0
INFO     /obj/position_setpoint_triplet0
INFO     /obj/vehicle_command0
INFO     /obj/offboard_mission0
INFO     /obj/mission_result0
INFO     /obj/home_position0
INFO     /obj/sensor_combined0
INFO     /obj/vehicle_attitude0
INFO     /obj/vehicle_global_position0
INFO     /obj/actuator_armed0
INFO     /obj/actuator_controls_00
INFO     /obj/airspeed0
INFO     /obj/vehicle_gps_position0
INFO     /obj/vehicle_local_position0
INFO     /obj/input_rc0
INFO     /obj/actuator_outputs0
INFO     /obj/vehicle_attitude_setpoint0
INFO     /obj/vehicle_rates_setpoint0
INFO     /obj/distance_sensor0
INFO     /obj/optical_flow0
INFO     /obj/vehicle_land_detected0
INFO     /obj/vision_position_estimate0
INFO     /obj/debug_key_value0
INFO     /obj/estimator_status0
INFO     /obj/uavcan_parameter_value0
INFO     /obj/telemetry_status0
INFO     /obj/qshell_req0

QURT: (type qshell list_topics)

Devices:  0581  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/_obj_  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/qshell_req0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/sensor_accel0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/sensor_gyro0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/parameter_update0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/sensor_baro0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/sensor_mag0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/input_rc0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/differential_pressure0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/vehicle_control_mode0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/rc_parameter_map0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/manual_control_setpoint0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/sensor_combined0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/mavlink_log0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/vehicle_status0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/actuator_armed0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/safety0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/mission_result0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/geofence_result0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/offboard_control_mode0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/vehicle_global_position0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/vehicle_local_position0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/vehicle_attitude0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/vehicle_land_detected0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/vehicle_gps_position0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/vehicle_command0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/battery_status0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/subsystem_info0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/position_setpoint_triplet0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/system_power0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/actuator_controls_00  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/vtol_vehicle_status0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/airspeed0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/optical_flow0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/distance_sensor0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/estimator_status0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/ekf2_innovations0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/control_state0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/vehicle_attitude_setpoint0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/vehicle_local_position_setpoint0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/vehicle_global_velocity_setpoint0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/vehicle_rates_setpoint0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/multirotor_motor_limits0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/rc_channels0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/mc_att_ctrl_status0  0587  vdev.cpp
[08500/00]  20:05.086  HAP:35:   /obj/actuator_outputs0  0587  vdev.cpp

[RomanBapst]

Notice that e.g the topic "ekf2_innovations" is not on the POSIX side so subscribing to that topic will not succeed. If you use the listener on the posix side and listen to the topic then you will get an empty topic on the first attempt, e.g.

listener ekf2_innovations 1 //  topic will have no values and zero time stamp

If after this you try again then you will get values and suddenly the topic appears in the topic list on the POSIX side. @mcharleb Can you think of what's the problem?

[LorenzMeier]

I think what fails is the orb_exists() check which returns false (but the listener triggers the topic instantiation as part of the attempt to read it). After that it exists on the POSIX side and gets correct values. So the way to fix this is to make sure orb_exists() returns true for topics which are not used at all on the POSIX side but are used on QuRT.

[RomanBapst]

@LorenzMeier Yeah, I tracked it down and it is the orb_exist() call. The question is how do I know which topics are used on the QURT side?

[LorenzMeier]

That's a question for @jywilson, @mcharleb and maybe Rama.

[julianoes]

Good digging! I ran into it before, and raised it to @jywilson before but we didn't get to further debug it yet.

[jywilson]

I will alert Rama to this. He is more familiar wth muorb.

[RomanBapst]

@jywilson Please let me know if there has been progress on this one.

[LorenzMeier]

@jywilson @mcharleb It would be great to know if this will get some attention this week or if we should start to dive into the FastRPC interface ourselves.

[LorenzMeier]

@mcharleb @julianoes I found this bit in the muORB:

    if (_AdspSubscriberCache[messageName] > 0) {// there are remote subscribers
        t2 = hrt_absolute_time();
        rc = _KraitWrapper.SendData(messageName, length, data);
        t3 = hrt_absolute_time();
        _snd_msg_count++;
        //PX4_DEBUG( "***** SENDING[%s] topic to remote....\n", messageName.c_str() );

    } else {
        //PX4_DEBUG( "******* NO SUBSCRIBER PRESENT ON THE REMOTE FOR topic[%s] \n", messageName.c_str() );
    }

Its essentially not updating the topic if no subscribers are present. If we would change this to always update this might be already entirely fixed.

[LorenzMeier]

Needs to be changed on both sides (ADSP and Krait) and tested though.

[mcharleb]

@LorenzMeier I might have a change to look at this tomorrow. Otherwise it may have to wait until the weekend.

[julianoes]

I'm going to push a workaround for now: https://github.com/PX4/Firmware/pull/4220

However, this should still get fixed properly.

[julianoes]

This would need big changes to muorb that probably won't happen anytime soon. I would kick this out of the milestone into the backlog.

[LorenzMeier]

Done

[mcharleb]

@julianoes There was some initial work done on this but I have not had time to fully test it:

https://github.com/ATLFlight/Firmware/tree/muorb_orb_exists

[julianoes]

@mcharleb A nice! Let me know when you want me to test it.

[mcharleb]

Adding reference to https://github.com/PX4/Firmware/pull/5327

[LorenzMeier]

@mcharleb This is stuck on CI not passing.

[LorenzMeier]

Fixed.