Open junwoo091400 opened 1 year ago
Note, it is interesting that https://github.com/PX4/PX4-Autopilot/pull/21190 passed the SITL Test, and it is indeed somewhat related to the tailsitter. But most likely unrelated coincidence.
As expected, it did fail a SITL Test run afterwards: https://github.com/PX4/PX4-Autopilot/actions/runs/4261478422/jobs/7415849888#step:17:607
Currently troubleshooting via compiling MAVSDK tests:
PX4_CMAKE_BUILD_TYPE=RelWithDebInfo test/mavsdk_tests/mavsdk_test_runner.py --speed-factor 20 --abort-early --model tailsitter test/mavsdk_tests/configs/sitl.json --verbose
_telemetry->health().is_armable
isn't becoming true for 20 seconds (timeout). It means, vehicle isn't armable for the preflight failure reasons
standard_vtol
, I can verify that the preflight errors don't occur, and that MAVSDK test runs successfully. So it is a bug with tailsitter.This leads me to conclude that state estimation for tailsitter is broken. @Jaeyoung-Lim @bresch could you check what went wrong on this?
However, note that when I start the tailsitter sim as standalone: make px4_sitl gazebo-classic_tailsitter
, I don't have this attitude estimation being a weird error.
This means that MAVSDK script is somehow messing up the tailsitter simulation
I can confirm that by following the model spawning workflow defined inside the https://github.com/PX4/PX4-Autopilot/blob/main/test/mavsdk_tests/process_helper.py, the spawned model shows erratic attitude estimation behavior.
Thus, finding the difference in standard SITL process of make px4_sitl gazebo-classic_tailsitter
and the following is the key to finding where the estimation going wrong is coming from.
Steps:
PX4_SIM_MODEL=gazebo-classic_tailsitter /home/junwoo/Coding/PX4-Autopilot/build/px4_sitl_default/bin/px4 /home/junwoo/Coding/PX4-Autopilot/build/px4_sitl_default/etc -s etc/init.d-posix/rcS -t /home/junwoo/Coding/PX4-Autopilot/test_data -d
GAZEBO_PLUGIN_PATH=/home/junwoo/Coding/PX4-Autopilot/build/px4_sitl_default/build_gazebo-classic/ GAZEBO_MODEL_PATH=/home/junwoo/Coding/PX4-Autopilot/Tools/simulation/gazebo-classic/sitl_gazebo-classic/models/ stdbuf -o0 -e0 gzserver --verbose /home/junwoo/Coding/PX4-Autopilot/Tools/simulation/gazebo-classic/sitl_gazebo-classic/worlds/empty.world
GAZEBO_MODEL_PATH=/home/junwoo/Coding/PX4-Autopilot/Tools/simulation/gazebo-classic/sitl_gazebo-classic/models/ stdbuf -o0 -e0 gz model --verbose --spawn-file /home/junwoo/Coding/PX4-Autopilot/Tools/simulation/gazebo-classic/sitl_gazebo-classic/models/tailsitter/tailsitter.sdf --model-name tailsitter -x 1.01 -y 0.98 -z 0.83
Further analysis:
make px4_sitl gazebo-classic_tailsitter
results in weird attitude estimation & vehicle is jumping around. This means that either PX4 instance or Gazebo server is causing this weird behavior (as model spawning was done by the standard script)make px4_sitl gazebo-classic_tailsitter
results also in erradic state estimation (which does settle after a bit). This requires removing this line: https://github.com/PX4/PX4-Autopilot/blob/bde194fb12342e4c115ce5a88be7ba839509442e/Tools/simulation/gazebo-classic/sitl_run.sh#L173. Overall, this means that the gazebo server spawning part of the MAVSDK script isn't causing the problem.PX4_SIM_MODEL=gazebo-classic_tailsitter PX4_SIM_WORLD=none /home/junwoo/Coding/PX4-Autopilot/build/px4_sitl_default/bin/px4 /home/junwoo/Coding/PX4-Autopilot/build/px4_sitl_default/etc
, the erratic state estimation is still there. So this leads to the conclusion that all the steps (px4 instance / gazebo server / gazebo model spawning) are not the ones causing this behavior, but is something else.Probably not related, but the only nitpick difference I could find with MAVSDK test script and native sitl build command was that the tmp_masvdk_tests/rootfs
(used as current working directory for PX4 instantiation for MAVSDK test) and rootfs
(used for standard PX4 SITL build) folder contents differ.
Especially, the parameter set seems to differ for some reason.
However, it can't be said this is the cause, as I wasn't using any of the Resolved in comment below.tmp_mavsdk_tests/rootfs
contents when I was running the px4 instance manually, and was still getting weird behavior. So probably unrelated, but still an interesting difference. @julianoes do you know why the contents of these two rootfs
would differ?
Further decoding of the bson file showed that only 'COM_DL_LOSS_T': 200, 'COM_OBC_LOSS_T': 100.0, 'COM_OF_LOSS_T': 10.0, 'COM_RC_LOSS_T': 10.0
is present in MAVSDK rootfs, but that's the only difference.
And, this is of course (suspected) set via the MAVSDK commands inside the script, so this rootfs difference isn't the reason why the state estimation fails.
3. This leads to the conclusion that the PX4 instantiation is the problem.
I can confirm that PX4 instantiation process is the problem, as when I start manually gazebo server & model spawning (haven't tried in any of the steps above), and only let the make px4_sitl_default gazebo-classic_tailsitter
start the PX4 process, the estimation error doesn't occur.
It is just not clear where the PX4 process starting differs when I do it manually vs via the sitl_run.sh
script.
I was able to narrow down the problem to being the 'order of PX4 instance & Gazebo server being started'.
I was sure that the command PX4_SIM_MODEL=gazebo-classic_tailsitter /home/junwoo/Coding/PX4-Autopilot/build/px4_sitl_default/bin/px4 /home/junwoo/Coding/PX4-Autopilot/build/px4_sitl_default/etc
was the same as used by the https://github.com/PX4/PX4-Autopilot/blob/main/Tools/simulation/gazebo-classic/sitl_run.sh. But then it did occur to me that in order to not have the make command start the px4 instance, I was always starting the manual px4 instance FIRST, before the gazebo server.
So, I just added a line exit
here: https://github.com/PX4/PX4-Autopilot/blob/bde194fb12342e4c115ce5a88be7ba839509442e/Tools/simulation/gazebo-classic/sitl_run.sh#L148, and ran that command make px4_sitl_default gazebo-classic_tailsitter
, then ran the PX4 instance manually, then bam. The weird attitude situation was gone.
Now I am certain that the problem was:
And probably PX4 was freaking out when it had no data stream & when it received long after it got started, the estimation went crazy.
For sure this is not a desirable behavior (we don't want the order of simulator / px4 instance instantiation to affect estimation behavior), but at least the solution is clear: Start the simulation first in MAVSDK TEST!
The order fix correctly allows 3 cases out of 4 tailsitter tests to pass!
For the 'VTOL Mission' case, it seems that Failure detector thinks while the VTOL transitions into FW, the maximum pitch (60 degrees) is reached, whereas in reality I think it's just a matter of body attitude conversion hickup (as tailsitter needs to rotate frame of reference as it transitions by 90 deg) @sfuhrer any analysis on this?
Interestingly, the SITL Test action that succeeded also had the pitch failure detector issue. So I am wondering why in my local MAVSDK Test the VTOL entered failsafe & landed :thinking:
This is a separate issue, and isn't related to the MAVSDK test failure case itself.
Log: https://logs.px4.io/plot_app?log=95fcd187-babd-4a9e-9a0f-748c1865d5e1
And probably PX4 was freaking out when it had no data stream & when it received long after it got started, the estimation went crazy.
That sounds odd. I would suggest we investigate that as well though. It shouldn't depend and freak out otherwise.
And probably PX4 was freaking out when it had no data stream & when it received long after it got started, the estimation went crazy.
That sounds odd. I would suggest we investigate that as well though. It shouldn't depend and freak out otherwise.
I agree! Also it's quite weird that this only happens to tailsitter model more frequently :thinking:
For the 'VTOL Mission' case, it seems that Failure detector thinks while the VTOL transitions into FW, the maximum pitch (60 degrees) is reached
Note, I suspected it may be somehow related to https://github.com/PX4/PX4-Autopilot/pull/20904, but the SITL test failure is resulting from a commit with that PR included, so I don't think that's the cause
As you asked during the FW call @dagar this test from local environment shows that GNSS error as pointed out by Jay. So yes, the local & CI errors are the same reason.
And Silvan suggested testing with multi EKF disabled, but that hasn't been done by me yet 😞
The failure is happening again (e.g. here), so I re-opened the issue.
The failure mode exhibits 2 traits:
[ 4.331|px4 ] WARN [health_and_arming_checks] Preflight Fail: Airspeed selector module down
[ 4.331|px4 ] WARN [health_and_arming_checks] Preflight Fail: ekf2 missing data
...
[ 4.832|px4 ] WARN [health_and_arming_checks] Preflight Fail: vertical velocity unstable
[ 4.832|px4 ] WARN [health_and_arming_checks] Preflight Fail: height estimate error
[ 4.832|px4 ] WARN [health_and_arming_checks] Preflight Fail: Attitude failure (roll)
The failure mode is slightly different than before https://github.com/PX4/PX4-Autopilot/pull/21319 was merged, in a sense that before that PR, the tailsitter would fail during the transition, but now it is failing to even arm.
Seems like there's a huge Z position / velocity estimate error! @bresch any initial guess on what's wrong?
Log is from the failed CI run's output
@junwoo091400 Yes, I'm aware of that bug. It is caused by a bad initial attitude (the tailsitter is like a multirotor in hover mode, roll and pitch should be 0). I cannot reproduce it in replay, even with the CI logs (ekf2 replay was enabled to investigate that bug). It might be due to an initial sample that we don't log but that is used by CI while running the test. The interesting thing is that it only occurs on the tailsitter CI and not every time (if you restart the test, it will most likely not fail again).
It might be due to an initial sample that we don't log but that is used by CI while running the test
Which sample are you referring to here? Sensor data? Or the initial attitude estimate?
Initial sensor data. We might want to be a bit more careful with the measurements we're using while initializing the EKF
Describe the bug
Example failure run: https://github.com/PX4/PX4-Autopilot/actions/runs/4299642479/jobs/7495070876
The tailsitter SITL Test has been failing since around 2 weeks ago.
Diagnosis
It seems to be happening due to preflight failures (vertical velocity, height estimate, etc unstable).
I also noticed this in real drone using
main
branch (often the preflight failures would occur easily, with GPS sensor well positioned outside). So I am suspecting it probably is PX4 estimation issue.Resource