Open lum-supakorn opened 4 years ago
Two questions: 1) What FPS do you have in FlightGear? 2) Are you able to publish PX4 Log from the flight (at least text log from console)?
@slimonslimon The frame rate swings from 50 to 100 FPS. Somehow I cannot collect the flight log from QGroundControl, but here is the text log from the console (not the same flight as that in the question but there was still teleportation):
[0/4] Performing build step for 'flightgear_bridge'
ninja: no work to do.
[3/4] cd /home/lum/Code/PX4/Firmware/b...PX4/Firmware/build/px4_sitl_nolockstep
SITL ARGS
sitl_bin: /home/lum/Code/PX4/Firmware/build/px4_sitl_nolockstep/bin/px4
debugger: none
program: flightgear
model: rascal-electric
world: none
src_path: /home/lum/Code/PX4/Firmware
build_path: /home/lum/Code/PX4/Firmware/build/px4_sitl_nolockstep
FG setup
0.08 [INFO]:gui UI languages: ("en-US")
0.08 [INFO]:gui Loaded translations for locale "en_US"
0.08 [INFO]:general FlightGear: Version 2020.1.1
0.08 [INFO]:general FlightGear: Build Type Dev
0.08 [INFO]:general Built with GNU C++ version 9.3
0.08 [INFO]:general Jenkins number/ID 0:none
0.08 [INFO]:general platform default fg_root = Path "/usr/share/games/flightgear"
0.10 [INFO]:input Reading user settings from Path "/home/lum/.fgfs/autosave_2020_1.xml"
0.10 [INFO]:general init package root at:Path "/home/lum/.fgfs/Aircraft"
0.10 [INFO]:input No user specified aircraft, using default
0.10 [INFO]:general Loading aircraft -set file from:Path "/usr/share/games/flightgear/Aircraft/c172p/c172p-set.xml"
0.13 [INFO]:general Found language resource for: en_US
0.13 [INFO]:general Reading localized strings for 'en' from Path "/usr/share/games/flightgear/Translations/default/sys.xml"
0.13 [INFO]:general Reading localized strings for 'en' from Path "/usr/share/games/flightgear/Translations/default/atc.xml"
0.13 [INFO]:general Reading localized strings for 'en' from Path "/usr/share/games/flightgear/Translations/default/tips.xml"
0.13 [INFO]:general option:version =
0.13 [INFO]:general Using default download dir: Path "/home/lum/.fgfs"
0.13 [INFO]:general Using default texture cache directory: Path "/home/lum/.fgfs/TextureCache"
0.13 [INFO]:general Using TerraSync dir: Path "/home/lum/.fgfs/TerraSync"
0.13 [INFO]:io Saving user settings to Path "/home/lum/.fgfs/autosave_2020_1.xml"
0.14 [WARN]:general SGSubsystemMgr::remove: not found: scenery
None
Rascal110-Electric-YASim
https://github.com/ThunderFly-aerospace/FlightGear-Rascal/archive/master.zip
5 /controls/flight/aileron -1
7 /controls/flight/elevator -1
2 /controls/flight/rudder 1
4 /controls/engines/engine/throttle 1
fgfs --aircraft=Rascal110-Electric-YASim --fg-aircraft=./models --enable-terrasync --timeofday=noon --disable-sound --telnet=15400 --generic=socket,out,100,127.0.0.1,15200,udp,FGtoPX4 --generic=socket,in,100,,15300,udp,PX4toFG --model-hz=120 --disable-random-objects --prop:/sim/rendering/texture-compression=off --prop:/sim/rendering/quality-level=0 --prop:/sim/rendering/shaders/quality-level=0 --disable-ai-traffic --prop:/sim/ai/enabled=0 --prop:/sim/rendering/random-vegetation=0 --prop:/sim/rendering/random-buildings=0 --disable-specular-highlight --disable-ai-models --disable-clouds --disable-clouds3d --fog-fastest --visibility=2000 --disable-distance-attenuation --disable-real-weather-fetch --prop:/sim/rendering/particles=0 --prop:/sim/rendering/multi-sample-buffers=1 --prop:/sim/rendering/multi-samples=2 --prop:/sim/rendering/draw-mask/clouds=false --prop:/sim/rendering/draw-mask/aircraft=true --prop:/sim/rendering/draw-mask/models=true --prop:/sim/rendering/draw-mask/terrain=true --disable-random-vegetation --disable-random-buildings --disable-rembrandt --disable-horizon-effect & echo $! > /tmp/px4fgfspid_0
SITL COMMAND: "/home/lum/Code/PX4/Firmware/build/px4_sitl_nolockstep/bin/px4" "/home/lum/Code/PX4/Firmware"/ROMFS/px4fmu_common -s etc/init.d-posix/rcS -t "/home/lum/Code/PX4/Firmware"/test_data
INFO [px4] Creating symlink /home/lum/Code/PX4/Firmware/ROMFS/px4fmu_common -> /home/lum/Code/PX4/Firmware/build/px4_sitl_nolockstep/tmp/rootfs/etc
______ __ __ ___
| ___ \ \ \ / / / |
| |_/ / \ V / / /| |
| __/ / \ / /_| |
| | / /^\ \ \___ |
\_| \/ \/ |_/
px4 starting.
INFO [px4] Calling startup script: /bin/sh etc/init.d-posix/rcS 0
INFO [param] selected parameter default file eeprom/parameters_1034
[param] Loaded: eeprom/parameters_1034
I'm Mavlink to FlightGear Bridge
Targed Bridge Freq: 200, send data every step: 1
4
5 -1
7 -1
2 1
4 1
INFO [dataman] Unknown restart, data manager file './dataman' size is 11798680 bytes
INFO [simulator] Waiting for simulator to accept connection on TCP port 4560
INFO [simulator] Simulator connected on TCP port 4560.
INFO [commander] LED: open /dev/led0 failed (22)
INFO [init] Mixer: etc/mixers-sitl/plane_sitl.main.mix on /dev/pwm_output0
INFO [mavlink] mode: Normal, data rate: 4000000 B/s on udp port 18570 remote port 14550
INFO [airspeed_selector] No airspeed sensor detected. Switch to non-airspeed mode.
0.07 [INFO]:gui UI languages: ("en-US")
0.07 [INFO]:gui Loaded translations for locale "en_US"
0.07 [INFO]:general FlightGear: Version 2020.1.1
0.07 [INFO]:general FlightGear: Build Type Dev
0.07 [INFO]:general Built with GNU C++ version 9.3
0.07 [INFO]:general Jenkins number/ID 0:none
0.07 [INFO]:general platform default fg_root = Path "/usr/share/games/flightgear"
0.08 [INFO]:input Reading user settings from Path "/home/lum/.fgfs/autosave_2020_1.xml"
0.08 [INFO]:general init package root at:Path "/home/lum/.fgfs/Aircraft"
0.08 [INFO]:input aircraft = Rascal110-Electric-YASim
0.08 [INFO]:general Loading aircraft -set file from:Path "/home/lum/Code/PX4/Firmware/Tools/flightgear_bridge/models/Rascal/Rascal110-Electric-YASim-set.xml"
0.08 [INFO]:general Found language resource for: en_US
0.08 [INFO]:general Reading localized strings for 'en' from Path "/usr/share/games/flightgear/Translations/default/sys.xml"
0.08 [INFO]:general Reading localized strings for 'en' from Path "/usr/share/games/flightgear/Translations/default/atc.xml"
0.08 [INFO]:general Reading localized strings for 'en' from Path "/usr/share/games/flightgear/Translations/default/tips.xml"
0.08 [INFO]:general option:aircraft = Rascal110-Electric-YASim
0.08 [INFO]:general option:fg-aircraft = ./models
0.08 [INFO]:general option:enable-terrasync =
0.08 [INFO]:general option:timeofday = noon
0.08 [INFO]:general option:disable-sound =
0.08 [INFO]:general Channel string = 15400
0.08 [INFO]:general option:telnet = 15400
0.08 [INFO]:general Channel string = socket,out,100,127.0.0.1,15200,udp,FGtoPX4
0.08 [INFO]:general option:generic = socket,out,100,127.0.0.1,15200,udp,FGtoPX4
0.08 [INFO]:general Channel string = socket,in,100,,15300,udp,PX4toFG
0.08 [INFO]:general option:generic = socket,in,100,,15300,udp,PX4toFG
0.08 [INFO]:general option:model-hz = 120
0.08 [INFO]:general option:disable-random-objects =
0.08 [INFO]:general option:prop = /sim/rendering/texture-compression=off
0.08 [INFO]:general option:prop = /sim/rendering/quality-level=0
0.08 [INFO]:general option:prop = /sim/rendering/shaders/quality-level=0
0.08 [INFO]:general option:disable-ai-traffic =
0.08 [INFO]:general option:prop = /sim/ai/enabled=0
0.08 [INFO]:general option:prop = /sim/rendering/random-vegetation=0
0.08 [INFO]:general option:prop = /sim/rendering/random-buildings=0
0.08 [INFO]:general option:disable-specular-highlight =
0.08 [INFO]:general option:disable-ai-models =
0.08 [INFO]:general option:disable-clouds =
0.08 [INFO]:general option:disable-clouds3d =
0.08 [INFO]:general option:fog-fastest =
0.08 [INFO]:general option:visibility = 2000
0.08 [INFO]:general option:disable-distance-attenuation =
0.08 [INFO]:general option:disable-real-weather-fetch =
0.08 [INFO]:general option:prop = /sim/rendering/particles=0
0.08 [INFO]:general option:prop = /sim/rendering/multi-sample-buffers=1
0.08 [INFO]:general option:prop = /sim/rendering/multi-samples=2
0.08 [INFO]:general option:prop = /sim/rendering/draw-mask/clouds=false
0.08 [INFO]:general option:prop = /sim/rendering/draw-mask/aircraft=true
0.08 [INFO]:general option:prop = /sim/rendering/draw-mask/models=true
0.08 [INFO]:general option:prop = /sim/rendering/draw-mask/terrain=true
0.08 [INFO]:general option:disable-random-vegetation =
0.08 [INFO]:general option:disable-random-buildings =
0.08 [INFO]:general option:disable-rembrandt =
0.08 [INFO]:general option:disable-horizon-effect =
0.08 [INFO]:general Using default download dir: Path "/home/lum/.fgfs"
0.08 [INFO]:general Using default texture cache directory: Path "/home/lum/.fgfs/TextureCache"
0.08 [INFO]:general Using TerraSync dir: Path "/home/lum/.fgfs/TerraSync"
0.08 [INFO]:general EmbeddedResourceManager: selected locale 'en_US'
0.08 [INFO]:opengl Using stock OSG implementation of GraphicsWindow
0.08 [INFO]:astro initializing cloud layers
Now checking for plug-in osgPlugins-3.6.4/osgdb_nvtt.so
INFO [mavlink] mode: Onboard, data rate: 4000000 B/s on udp port 14580 remote port 14540
INFO [mavlink] mode: Onboard, data rate: 4000 B/s on udp port 14280 remote port 14030
INFO [logger] logger started (mode=all)
INFO [logger] Start file log (type: full)
INFO [logger] [logger] ./log/2020-06-19/09_09_23.ulg
INFO [logger] Opened full log file: ./log/2020-06-19/09_09_23.ulg
INFO [mavlink] MAVLink only on localhost (set param MAV_BROADCAST = 1 to enable network)
INFO [px4] Startup script returned successfully
pxh> 0.22 [WARN]:general Enabling ATI/AMD viewport hack
0.22 [INFO]:general Configuration State
0.22 [INFO]:general ============= =====
0.22 [INFO]:general aircraft-dir = "/home/lum/Code/PX4/Firmware/Tools/flightgear_bridge/models/Rascal"
0.22 [INFO]:general fghome-dir = "Path "/home/lum/.fgfs""
0.22 [INFO]:general download-dir = "/home/lum/.fgfs"
0.22 [INFO]:general terrasync-dir = "/home/lum/.fgfs/TerraSync"
0.22 [INFO]:general aircraft-search-paths =
/home/lum/Code/PX4/Firmware/Tools/flightgear_bridge/models
0.22 [INFO]:general scenery-search-paths =
/home/lum/.fgfs/TerraSync
/usr/share/games/flightgear/Scenery
INFO [mavlink] partner IP: 127.0.0.1
0.35 [INFO]:general NVIDIA Corporation
0.35 [INFO]:general GeForce RTX 2060/PCIe/SSE2
0.35 [INFO]:general 4.6.0 NVIDIA 440.64
0.35 [INFO]:general 4.60 NVIDIA
0.35 [INFO]:general VertexAttribDivisor supported
0.40 [INFO]:view Splash screen progress loading-aircraft-list
0.40 [INFO]:view FGRenderer:: Maximum texture size 32768
0.41 [ALRT]:terrasync Starting automatic scenery download/synchronization to '/home/lum/.fgfs/TerraSync'.
0.41 [INFO]:view Splash screen progress loading-nav-dat
0.41 [INFO]:navaid NavCache at:Path "/home/lum/.fgfs/navdata_2020_1.cache"
0.41 [INFO]:navaid NavCache read-only flags is:0
0.41 [INFO]:navaid NavCache: no main cache rebuild required
0.41 [INFO]:view Splash screen progress init-scenery
0.42 [INFO]:general General Initialization
0.42 [INFO]:general ======= ==============
0.42 [INFO]:general FG_ROOT = "Path "/usr/share/games/flightgear""
0.43 [INFO]:general Attempting to set starting position for BIKF:29
0.44 [INFO]:event Reading timezone info from: Path "/usr/share/games/flightgear/Timezone/zone.tab"
0.44 [INFO]:terrain Selected scenery is tilecache
0.44 [INFO]:terrain FGStgTerrain::init - init tilemgr
0.44 [INFO]:terrain Initializing Tile Manager subsystem.
0.44 [INFO]:terrain Initializing the tile cache.
0.44 [INFO]:terrain max cache size = 100
0.44 [INFO]:terrain current cache size = 0
0.44 [INFO]:terrain done with init()
0.44 [INFO]:view Splash screen progress finalize-position
0.49 [INFO]:view Splash screen progress creating-subsystems
0.49 [INFO]:general Creating Subsystems
0.49 [INFO]:general ======== ==========
0.49 [INFO]:input Reading materials from Path "/usr/share/games/flightgear/Materials/regions/materials.xml"
0.50 [INFO]:terrasync picking entry # 2, server is https://ukmirror.flightgear.org/fgscenery
0.50 [INFO]:terrasync terrasync scenery provider of the day is 'https://ukmirror.flightgear.org/fgscenery'
0.52 [INFO]:terrasync sync of https://ukmirror.flightgear.org/fgscenery/Models started, queue size is 26
1.06 [INFO]:systems Reading systems from Path "/usr/share/games/flightgear/Aircraft/Generic/generic-systems.xml"
1.06 [INFO]:cockpit Initializing HUD Instrument
1.06 [INFO]:sound Reading sound sound from Path "/usr/share/games/flightgear/Aircraft/Generic/generic-sound.xml"
1.06 [INFO]:general Creating subsystems took:576
1.06 [INFO]:view Splash screen progress binding-subsystems
1.54 [ALRT]:nasal Volcanic activity off.
PX4 Communicator: PX4 Connected.
10.68 [ALRT]:flight This aircraft uses yasim version 'YASIM_VERSION_CURRENT' (3)
YASim warning: possible convergence problem.
FGBridge: Low FlightGear Update Freq: 8.57143
FGBridge: Low FlightGear Update Freq: 15
WARN [airspeed_selector] Airspeed: switched from sensor -1 to 1
11.34 [INFO]:OSG
11.34 [WARN]:OSG PNG lib warning : iCCP: known incorrect sRGB profile
11.35 [WARN]:OSG PNG lib warning : iCCP: known incorrect sRGB profile
INFO [ecl/EKF] 1253380640: GPS checks passed
11.90 [WARN]:OSG PNG lib warning : iCCP: known incorrect sRGB profile
11.92 [WARN]:OSG PNG lib warning : iCCP: known incorrect sRGB profile
11.92 [WARN]:OSG PNG lib warning : iCCP: known incorrect sRGB profile
11.92 [WARN]:OSG PNG lib warning : iCCP: known incorrect sRGB profile
11.93 [WARN]:OSG PNG lib warning : iCCP: known incorrect sRGB profile
11.95 [WARN]:OSG PNG lib warning : iCCP: known incorrect sRGB profile
11.95 [WARN]:OSG PNG lib warning : iCCP: known incorrect sRGB profile
11.96 [WARN]:OSG PNG lib warning : iCCP: known incorrect sRGB profile
11.96 [WARN]:OSG PNG lib warning : iCCP: known incorrect sRGB profile
11.96 [WARN]:OSG PNG lib warning : iCCP: known incorrect sRGB profile
INFO [ecl/EKF] 1254366984: EKF aligned, (baro hgt, IMU buf: 18, OBS buf: 14)
INFO [ecl/EKF] 1254366984: reset position to last known position
INFO [ecl/EKF] 1254366984: reset velocity to zero
FGBridge: Low FlightGear Update Freq: 15
INFO [ecl/EKF] 1257956657: reset position to GPS
INFO [ecl/EKF] 1257956657: reset velocity to GPS
INFO [ecl/EKF] 1257956657: starting GPS fusion
INFO [commander] ARMED by Arm/Disarm component command
WARN [navigator] Using minimum takeoff altitude: 20.00 m
INFO [fw_pos_control_l1] Takeoff on runway
INFO [commander] Takeoff detected
INFO [runway_takeoff] #Takeoff airspeed reached
INFO [runway_takeoff] #Climbout
INFO [runway_takeoff] #Navigating to waypoint
INFO [ecl/EKF] 1329458334: reset velocity to GPS
INFO [ecl/EKF] 1329458334: reset position to GPS
INFO [ecl/EKF] 1329458334: Emergency yaw reset - mag use stopped
WARN [commander] Stopping compass use! Check calibration on landing
95.05 [WARN]:OSG PNG lib warning : iCCP: known incorrect sRGB profile
95.11 [WARN]:OSG PNG lib warning : iCCP: known incorrect sRGB profile
FGBridge: Low FlightGear Update Freq: 17.1429
FGBridge: Low FlightGear Update Freq: 20
INFO [ecl/EKF] 1349342267: reset velocity to GPS
INFO [ecl/EKF] 1349342267: reset position to GPS
INFO [ecl/EKF] 1349342267: Emergency yaw reset - mag use stopped
FGBridge: Low FlightGear Update Freq: 17.1429
INFO [ecl/EKF] 1406762730: reset velocity to GPS
INFO [ecl/EKF] 1406762730: reset position to GPS
INFO [ecl/EKF] 1406762730: Emergency yaw reset - mag use stopped
FGBridge: Low FlightGear Update Freq: 17.1429
INFO [navigator] RTL LAND activated
INFO [navigator] RTL: landing at home position.
INFO [navigator] RTL: climb to 145 m (100 m above destination)
FGBridge: Low FlightGear Update Freq: 20
INFO [navigator] RTL: return at 145 m (100 m above destination)
INFO [navigator] RTL: descend to 136 m (91 m above destination)
INFO [navigator] RTL: completed, loitering
INFO [fw_pos_control_l1] Landing, on slope
INFO [ecl/EKF] 1430538618: reset velocity to GPS
INFO [ecl/EKF] 1430538618: reset position to GPS
WARN [ecl/EKF] 1430538618: GPS fusion timeout - reset to GPS
INFO [commander] Landing detected
INFO [commander] DISARMED by Auto disarm initiated
INFO [logger] closed logfile, bytes written: 13203301
You have multiple occurrences of following lines in the console output:
FGBridge: Low FlightGear Update Freq: 17.1429
It seems there is trouble with the FlightGear frame rate. Are you able to check the frame rate of FG itself by enabling the FPS display? In cases where FG FPS drops, the "teleportation" will probably occur.
@kaklik I did obtain the frame rate of FG through FPS display you mentioned. Do you know what caused the FGBridge: Low FlightGear Update Freq
prompt?
The bridge internally counts how often comes messages from the FG. It can be caused by "some interference" of Frequency of internal pooling loop of bridge: https://github.com/ThunderFly-aerospace/PX4-FlightGear-Bridge/blob/master/src/flightgear_bridge.cpp#L145 and frequency of FlightGear main loop (somehow connected with FG FPS - FG is single thread).
To proof this hypothesis, you can change frequency of bridge here: https://github.com/ThunderFly-aerospace/PX4-FlightGear-Bridge/blob/master/src/flightgear_bridge.cpp#L98
But problems can be at more places and they are hard to reproduce or debug. Try reduce mentioned delay closer to zero - it will increase bridge frequency. For example to 100 (theroreticly 10KHz - but it will be slowed down by poling of sockets) and switch off bad data upsampling - set https://github.com/ThunderFly-aerospace/PX4-FlightGear-Bridge/blob/master/src/flightgear_bridge.cpp#L101 to false.
Then the bridge should pool for messages from FG very often and only resend message to PX4. (I hope that it will be working)
The last think what you can set up better is requested frequency of messages from FlightGear - now it is set to 100 Hz. But FG is single thread aplication, so real Frequency of messages will be somehow connected with FPS - it will not be large than FPS. So if you request 1000 messages per second, It will be sending message every frame. You can set-up this a here: https://github.com/ThunderFly-aerospace/PX4-FlightGear-Bridge/blob/master/FG_run.py#L104 rewrite the number 100 with better number. And do the same substitution on next line.
Thank you for your time that you spend with testing these things.
@slimonslimon Thank you for your input and apologize for late reply. I will look into this further as soon as I can.
I'm not able to reproduce this zigzaging
I think this points to the fact that we need lockstep for flightgear
@Jaeyoung-Lim Agreed! Lockstep also makes debugging models much easier.
@Jaeyoung-Lim @lum-supakorn - It could be very cool - but there are 2 main problems:
@slimonslimon Right, but just pointing out that this part of SITL will never be reliable without it.
If we are not able to support it, the least we should do is provide strict performance requirements on hardware to run the simulation.
@Jaeyoung-Lim Does JSBSim not support something similar to lockstep?
JSBSim does, but the flight gear interfaced used in this repo does not
@Jaeyoung-Lim @lum-supakorn I suggest continuing at the discussion about the lock-step option in different issue: #8 Where I also commented on the actual possibilities.
@lum-supakorn are you able to test modifications that were mentioned above? (Because exactly I have not hardware which has higher FPS than about 50 Hz)
well, after long time I have improved gyroscope readouts which may solve this issue. https://github.com/ThunderFly-aerospace/PX4-FlightGear-Bridge/commit/d7470250d9a45439957b0e5c0e123900e44d1d29
Steps to reproduce:
make px4_sitl_nolockstep flightgear_rascal-electric
ormake px4_sitl_nolockstep flightgear_rascal
From time to time the plane in QGroundControl will teleport and the flight path will be drawn as shown in the figure below:
Note that the plane in FlightGear does not teleport, so I assume that the issue comes from PX4 FlightGear bridge.