vooon
commented
6 years ago Do you use ntpd_driver? Do you reset autopilot during this log (suspicious CON msgs)? Do you use 16S battery?
Closed kevin-george closed 6 years ago
vooon
commented
6 years ago Do you use ntpd_driver? Do you reset autopilot during this log (suspicious CON msgs)? Do you use 16S battery?
kevin-george
commented
6 years ago Thank you for your response.
Do you use ntpd_driver?
We use ntpdate for syncing with a time server.
Do you reset autopilot during this log (suspicious CON msgs)?
We did not the reset autopilot
Do you use 16S battery?
We get this when the pixhawk is powered using a usb. Does powering with a 16S battery help?
vooon
commented
6 years ago kevin-george
commented
6 years ago So you have internet connection onboard?
No we don't. We just run the sync manually before flight.
Then problem may be in wiring or hardware.
We have a simple FTDI cable that connects the onboard computer USB to PX4. This happens on multiple machines, so wiring seems unlikely to be the cause unless we got 3 wires from 2 different companies messed up together? Is there any way to check this?
No, i asked because 65 volt battery is unusual. With USB that should be near zero. Or PX4 send undocumented value, 65535 (-1).
Our batteries are not 16S. Where on pixhawk or mavros is the setting that sets battery voltage to 65V? Perhaps we can change it and try. I am guessing this is not the issue though?
vooon
commented
6 years ago Try to use https://github.com/vooon/ntpd_driver to feed GPS time to ntpd. Hard sync only occurs if dT > 10 ms (look at diagnostics "Time Sync"). Also try to measure latency https://gist.github.com/vooon/e50fd7d38c137aa932b8caf18326d12e (require two uart modules connected).
Also try to reconfigure power module params.
TSC21
commented
6 years ago @kevin-george is this solved to you?
kevin-george
commented
6 years ago @TSC21 I apologize for the delay in response, got pulled away to fix other issues. I wish to actively solve this so I don't waste more of your time :)
@vooon
Try to use https://github.com/vooon/ntpd_driver to feed GPS time to ntpd.
Wouldn't this require the GPS module to be connected to the offboard computer(in our case an intel NUC) instead of directly to pixhawk?
Hard sync only occurs if dT > 10 ms (look at diagnostics "Time Sync").
I saw this in system_time.cpp, the diagnostics key "Timesyncs since startup" also gets reset whenever the hard sync occurred. The CON messages that were showing up before, don't show up anymore.
Also try to reconfigure power module params.
Can the misbehavior of the clock be because of incorrect power configuration?
vooon
commented
6 years ago No, GPS module should be connected to FCU. Then it sends SYSTEM_TIME message, that handled by sys_time and converted to ROS time message, then ntpd_driver send that time to ntpd via shared memory.
Of course that chain has significantly worse precision than regular NTP or gpsd, but do not require additional gps.
kevin-george
commented
6 years ago Thank you for responding @vooon
I tried using the ntpd configuration for the ntpd_driver. I still see Hard syncing of clocks every few seconds. Could the clock on the pixhawk be malfunctioning? I don't understand why it would go out of sync so fast?
Also, when I run ttyping.cc to measure latency, I get a segfault
./ttyping -b 921600 /dev/ttyUSB0
--baud: "921600"
--help: false
--interval: "1000"
--pong: false
<tty>: "/dev/ttyUSB0"
Opening: /dev/ttyUSB0 Mode: PING
io_service running
Segmentation fault (core dumped) Here's the backtrace
(gdb) bt
#0 0x0000000000000000 in ?? ()
#1 0x0000000000400e97 in __gthread_equal (__t1=140737354110720, __t2=0)
at /usr/include/x86_64-linux-gnu/c++/5/bits/gthr-default.h:680
#2 0x0000000000405c97 in std::operator== (__x=..., __y=...) at /usr/include/c++/5/thread:84
#3 0x0000000000405cfc in std::thread::joinable (this=0x7fffffffe310)
at /usr/include/c++/5/thread:170
#4 0x00000000004021cd in main (argc=4, argv=0x7fffffffe568) at ttyping.cc:243The pixhawk is connected to ttyUSB0
stty < /dev/ttyUSB0
speed 921600 baud; line = 0;
min = 1; time = 0;
-brkint -icrnl -imaxbel
-opost
-isig -icanon -iexten -echoNo idea, but better keep ntpd.
ttyping quick and dirty, and require one instance on both ends (so it needs two UART's).
This tool mostly needed to check latency of external UART like USB ones. One instance should be started on responder (better to use internal SoC tty), other on target tty.
Result is total latency, which is slightly lower on SoC side.
Actually that "ping" may be done on MAVLink, but for that need to implement responder on fcu side. And anyway TIMESTAMP are working similar.
kevin-george
commented
6 years ago I think I will be keeping ntpd_driver running.
Right now there are 3 clock times :) - offboard computer, gps time & pixhawk system clock
With your ntpd_driver I can get the first 2 to sync, is there any way to get the 3rd synced to the first 2 as well? Maybe by changing time_ref_source in px4_config.yaml from fcu to mavros?
vooon
commented
6 years ago sys_time should sync OBC and FCU time. Actually it is not synchronise clocks, but adjust message timestamps on both sides.
time_ref_source only change TimeReference.source filed.
Issue details
When running the px4.launch with default parameters, we see repeated and frequent hard syncing of clocks(The logs below show this). Is this expected behavior and/or something we should be concerned about? Can we do something to decrease this hard-syncing?
MAVROS version and platform
Mavros: 0.21.0 ROS: Kinetic Ubuntu: 16.04
Autopilot type and version
[ ] ArduPilot [X] PX4
Version: 3.1
Node logs
Diagnostics
Check ID