Closed klausenbusk closed 7 years ago
I forgot to mention, that I have changed NeoTime.h
to use the posix epoch:
static const uint16_t s_epoch_year = POSIX_EPOCH_YEAR;
static const uint8_t s_pivot_year = 00;
static const uint8_t s_epoch_offset = 00;
static const uint8_t s_epoch_weekday = POSIX_EPOCH_WEEKDAY;
You really shouldn't use String
, not even on the ESP8266. Just print the pieces separately. In your case, I would make a sendInfoTo
routine so you can print the pieces on two output streams:
void sendInfoTo( Print & output )
{
output.print( fix.dateTime );
output.print( ',' );
output.print( fix.satellites );
output.print( ',' );
// https://github.com/SlashDevin/NeoGPS/blob/master/extras/doc/Data%20Model.md#data-model
// "~7 significant digits"
// https://github.com/SlashDevin/NeoGPS/blob/master/extras/doc/Data%20Model.md#precision
// "32-bit ints have 10 significant digits, so you can detect very"
output.print( fix.latitudeL() );
output.print( ',' );
output.print( fix.longitudeL() );
output.print( ',' );
output.println( fix.speed_kph() );
}
Notice how it uses latitudeL() to get the 32-bit integer form, with 10 significant digits. The plain latitude()
accessor returns a float.
Then your loop looks like this:
uint16_t fixCount;
uint16_t lastSendData;
void loop() {
while (gps.available( serialGPS )) {
gps_fix fix = gps.read();
fixCount++;
//Serial.println(fix.speed_kph());
if (fix.valid.speed && (fix.speed_kph() > 1) && <-- notice parens
fix.valid.location && fix.valid.altitude) {
File dataFile = SD.open("datalog.txt", FILE_WRITE);
if (dataFile) {
sendInfoTo( dataFile );
dataFile.close();
}
#ifdef DEBUG
sendInfoTo( Serial );
#endif
}
// Once every 10 updates (10 seconds?), send some data
if (fixCount - lastSendData >= 10) {
lastSendData = fixCount;
sendData();
}
}
}
And don't use delay
. While the Arduino twiddles it's thumbs for 10s, lots of GPS data continues to arrive. It eventually overflows the input buffer and you lose at least 8s of fixes. Notice how the above uses the fixes as a 1-second "clock". After 10 fixes have arrived it calls sendData
again.
Regardless...
I suspect that you are not parsing the NMEA sentence that contains the date. Parsing messages with time does not set the date portion, so they will default to the EPOCH date. Make sure that
NMEAGPS_cfg.h
GPSfix_cfg.h
, and NMEAorder.ino will display which sentences your device is sending, and in what order. It will also verify that you have chosen the correct LAST_SENTENCE.
NMEA.ino will display everything that you have configured. If it doesn't display the correct date, something isn't configured correctly.
You really shouldn't use String, not even on the ESP8266.
Due to the dynamically memory allocation or why?
Just print the pieces separately. In your case, I would make a sendInfoTo routine so you can print the pieces on two output streams:
That could work, but I don't exactly understand why "abusing" (?) strings is so bad? Is it the memory allocation?
And don't use delay. While the Arduino twiddles it's thumbs for 10s, lots of GPS data continues to arrive. It eventually overflows the input buffer and you lose at least 8s of fixes. Notice how the above uses the fixes as a 1-second "clock". After 10 fixes have arrived it calls sendData again.
I'm aware of that and I have read: https://github.com/SlashDevin/NeoGPS/blob/master/extras/doc/Troubleshooting.md#blocking-operations , but I don't exactly understand why it is a issue? I want to log location etc. every 10 second if the car is moving, so why is it a issue that I "drop" 8 fixes and then "resume" operation after 10 sec?
I suspect that you are not parsing the NMEA sentence that contains the date. Parsing messages with time does not set the date portion, so they will default to the EPOCH date. Make sure that
* the RMC and/or ZDA sentences are enabled in NMEAGPS_cfg.h * GPS_FIX_DATE is enabled in GPSfix_cfg.h, and
Everything mentioned was already enabled, expect ZDA in NMEAGPS_cfg.h
, but enabling ZDA didn't change anything.
your device is configured to send one or both of those sentences.
Not sure, but I manged to get some raw data with:
if(serialGPS.available()){
Serial.write(serialGPS.read());
}
$GPRMC,23635.00,A,xxxx.05995,N,0008.28384,E,0.068,,30617,,,A*79
$GPVTG,,T,,,0.068,N,0.126,K,A*28
$GPGGA,xxxx5.00,xxxx.05995,N,00908.28384,E1,09,091,54.2M,3.6,M,,*6B
$PGSA,A,3,31,03,26,17,06,09,22,02,2,,,,1.44,0.91,1.12*06
$GGSV,4,1,15,02,156,02,17,316,5,03,45,107,23,04,31,052,23*7A
$GGSV,4,2,15,06,55,278,33,0,06,176,,09,5,,35,17,09,228,13*7C
$GPGSV,4,3,1,19,20,245,15,22,20,109,24,23,78,123,27,25,02,344,10*77
$GPG,4,4,1,26,08,069,1531,13,032,36,40,15,129,*44
$GPGLLxxxx.0.0,$PRMC,xxxx36.00,A,xxxx.05989,N,090.28382,E,0.046,,300617,,,A*7D
$GPVTG,,,0.046,N,0.085,K,A*2C
$GPGG,xxxx36.00,xxxx.05989,N,00908.28382,E1,09,0,539,M,43.6,M,,*6$GPGSA,A,3,1,03,26,17,06,09,22,02,23,,,,1.44,.9,1.12*06
$GPGSV,4,1,15,01,02,16,,02,17,364,03,45,107,22,4,1,052,22*7B
$GPGSV,4,2,15,06,5,278,33,07,06,176,,09,58,219,35,17,09,228,13*7
$GPGSV,,3,15,19,20,45,14,22,20,109,23,23,78,123,2625,02,344,10*70
$GPGSV,4,4,15,26,08,069,1531,13,032,37,40,15,29,*45
$GPGLL,xxxx.05989,N,00908.8382,E,xxxx3600A,A*65
$GPC,xxxx37.00,A,xxxx.05985,N0008.28384,E,0.054,,300617,,,A*75
$GPVTG,,T,,M,0.054,,0100,K,A*23$GPGGA,xxxx37.00,xxxx.5985,N,00908.28384,E,1,09,0.91,53.8,M,43.6,M,,*65
$GPGSA,A3,31,03,26,176,09,22,02,23,,,1.44,0.91,1.12*06
From what I can see GPRMC
lines contain all the needed data:
$GPRMC,223916.00,A,556.05915,N98.8360,E,0284,,300617,,A*75
223916 (hour, minute, sec) and 300617 (day, month and year).
but I also noticed that some of them is missing data:
$GPRMC,223701.00,A,xxxx.05989,N,00908.2845,E,0.113,,,A*78
$GPRMC,xxxx59.00,xxxx.05963N,00908.28456,E,0.041,,30061,,,A*79
I'm not sure what is causing that. (The xx was replaced by me, when trying to remove the long/lat)
BTW: I think I somehow managed to fry my GPS, I get no data and it isn't blinking anymore. Properly static electricity or something like that, so new parts are on the way and a "wrist grounding strap"..
NMEAorder.ino will display which sentences your device is sending, and in what order. It will also verify that you have chosen the correct LAST_SENTENCE.
NMEA.ino will display everything that you have configured. If it doesn't display the correct date, something isn't configured correctly.
Both program crash and gives me a stack trace, but I suspect the GPS was already fried.
I don't exactly understand why "abusing" (?) strings is so bad?
Because this. Not even on servers.
why is it a issue that I "drop" 8 fixes and then "resume" operation after 10 sec?
Because the first 64 characters of the 1st fix will be in the input buffer when you start reading again. You could get a fix that is 10 seconds old. Then you will start getting data that is current, but you may start getting data in the middle of the batch of sentences. So the next fix may not have all the pieces you want, because it missed some of the data at the front.
I don't understand your reluctance since I showed a very simple way to avoid both String
and delay
.
Did you try the loop
and sendInfoTo
that I offered?
I manged to get some raw data:
It is dropping some characters. For example, these lines are corrupt:
$PGSA,A,3,31,03,26,17,06,09,22,02,2,,,,1.44,0.91,1.12*06
$GGSV,4,1,15,02,156,02,17,316,5,03,45,107,23,04,31,052,23*7A
$GGSV,4,2,15,06,55,278,33,0,06,176,,09,5,,35,17,09,228,13*7C
$GPG,4,4,1,26,08,069,1531,13,032,36,40,15,129,*44
$GPGG,xxxx36.00,xxxx.05989,N,00908.28382,E1,09,0,539,M,43.6,M,,*6$GPGSA,A,3,1,03,26,17,06,09,22,02,23,,,,1.44,.9,1.12*06
GPRMC,2237200,A556.6035,N,00908.28438,E,0.155,,300617,,,A*73
These are easy to spot, because the NMEA sentence type is missing a character or two, and the CR/LF was dropped between two sentences. Without checking all the data, I can only assume that there are characters missing in the middle of the sentences, too. The odds are not good that you ever receive an RMC without errors, and the date never gets parsed.
From what I can see GPRMC lines contain all the needed data:
$GPRMC,223916.00,A,556.05915,N98.8360,E,0284,,300617,,A*75
223916 (hour, minute, sec) and 300617 (day, month and year).
That sentence is corrupted, too. It does not pass the checksum validation (try this).
At 9600, I am very surprised that the ESP8266 is dropping characters. Normally, you don't see that unless the MCU spends too much time in interrupt routines (or with interrupts disabled). Perhaps a lot of network traffic could cause this, but I really don't know.
Both [NMEA and NMEAorder] programs crash and gives me a stack trace
That is very unusual, as I have tested the parser with a variety of corrupted data, including dropped characters. I have only seen this when trying to have "local" config files (in the same directory as the sketch). The Arduino build environment requires you to modify the library *_cfg.h files. You cannot have local copies, and you cannot put #define
s for NeoGPS configuration items in your sketch.
I don't understand your reluctance since I showed a very simple way to avoid both String and delay.
I'm just trying to understand it/learn something :)
Did you try the loop and sendInfoTo that I offered?
Only part of it, I fried my GPS unit before I got to it.
At 9600, I am very surprised that the ESP8266 is dropping characters. Normally, you don't see that unless the MCU spends too much time in interrupt routines (or with interrupts disabled). Perhaps a lot of network traffic could cause this, but I really don't know.
Could it be a bad soldering/noise? I'm pretty sure I had it working on my breadboard/hackish build with correct date. I will try with the hardware serial when I get a new GPS unit, EspSoftwareSerial says a little about interrupt (but I don't think there should be a lot of traffic):
Please note that due to the fact that the ESP always have other activities ongoing, there will be some inexactness in interrupt timings. This may lead to bit errors when having heavy data traffic in high baud rates.
The Arduino build environment requires you to modify the library *_cfg.h files
I modified both sketch to use SoftwareSerial.h
(NeoSWSerial.h
wouldn't compile for ESP8266) and changed RX_PIN
to D1
and TX_PIN
to 0
in GPSport.h
. I'm wondering if TX
is required?
NMEA.ino
was crashing just after: https://github.com/SlashDevin/NeoGPS/blob/master/examples/NMEA/NMEA.ino#L172
Could it be a bad soldering/noise?
Perhaps. 9600 is fairly slow, so the ESP8266 will be spending a lot of its time in the SoftwareSerial rx methods. But it shouldn't miss an entire character. I would expect a corrupted character, not a dropped character. Noise on other interrupt-generating pins could cause it to miss the START bit of a character, but again, you should see a corrupted character, not a dropped character.
NMEA.ino
was crashing just after line 172
That's an "impossible" place to crash, so there may be some platform difference with the FLASH string macros and pgm_xxx
routines. The sentence names are stored in FLASH (on AVR MCUs) to save RAM. NMEAorder will try to print those sentence names, too. I am not familiar with the ESP8266 FLASH memory. Seems like I saw a recent Hackaday article on that...
Did these 2 programs work on the breadboard version? I'm pretty sure I would have heard about some problem on other (manufactured) ESP8266 boards.
...and TX_PIN to 0 in GPSport.h. I'm wondering if TX is required?
On the boards I have tried, it doesn't matter what you choose for the TX pin if you never write to the GPS device (e.g., to send a configuration command). Again, I'm not familiar with the ESP8266 version of SoftwareSerial
.
I'm also having issues with epoch timestamp, the following code:
NeoGPS::clock_t seconds = fix.dateTime;
Serial.print(F("EPOCH: "));
Serial.println(seconds);
NeoGPS::time_t stampgmt(seconds);
Serial.print("GMT: ");
Serial << stampgmt;
Serial.print("\nLocal: ");
seconds += tz_hours * NeoGPS::SECONDS_PER_HOUR +
tz_minutes * NeoGPS::SECONDS_PER_MINUTE;
NeoGPS::time_t stamplocal(seconds);
Serial << stamplocal;
Prints the following:
EPOCH: 552966942
GMT: 2017-07-10 01:55:42
Local: 2017-07-10 03:55:42
That 552966942 Epoch is: Saturday, July 11, 1987 1:55:42 AM.
Seems there is some Y2K offset somewhere? Please advice!
@Redferne , 552966942
is not the EPOCH, it is the number of seconds since the EPOCH. The default EPOCH is 2000-01-01 00:00:00 (Y2K), so 552966942 seconds after that is indeed 2017-07-10 01:55:42 (try this).
If you want to use a different time origin (i.e., EPOCH), you can change the constants in NeoTime.h. I have included 2 other EPOCHs, 1-1-1970 (aka UNIX time) and 1-1-1900 (aka NTP time). This is typically a compile-time choice, but if you want it to be a run-time choice, uncomment TIME_EPOCH_MODIFIABLE at the top. Then you can call the EPOCH mutators whenever you want.
Yes. Sorry for being unclear. I have tried to change the Epoch constants in NeoTime.h. I badly need Epoch in POSIX (1970). However it still gives me a wierd second count. I've only tried the static definition, I'll give MODIFIABLE a try.
However it still gives me a weird second count.
Need more input.
Ok. Ran same code as above but I changed NeoTime.h to:
//#define TIME_EPOCH_MODIFIABLE
static const uint16_t s_epoch_year = POSIX_EPOCH_YEAR;
static const uint8_t s_pivot_year = 00;
static const uint8_t s_epoch_offset = 00;
static const uint8_t s_epoch_weekday = POSIX_EPOCH_WEEKDAY;
Then I got another weird epoch and faulty time stamp:
Epoch: 17072262 (which is Friday, July 17, 1970 2:17:42 PM)
GMT: 1900-07-17 14:17:42
Local: 1900-07-17 16:17:42
Now I changed EPOCH to modifiable:
#define TIME_EPOCH_MODIFIABLE
Results are:
Epoch: 553357303 (which is Wednesday, July 15, 1987 2:21:43 PM)
GMT: 2017-07-14 14:21:43
Local: 2017-07-14 16:21:43
Epoch counter (seconds) is calculated from Y2K offset and not POSIX but timestamp is correct. What gives?
According to this snippet, stampgmt
has the same value as fix.dateTime
NeoGPS::clock_t seconds = fix.dateTime;
Serial.print(F("EPOCH: ")); // <-- this label should be something like "UTC seconds since EPOCH"
Serial.println(seconds);
NeoGPS::time_t stampgmt(seconds);
To re-iterate, the seconds
value IS NOT the EPOCH. It is the number of seconds since the EPOCH you have chosen, 1970-01-01 00:00:00. And if I disregard the mislabelling...
I see that the fix.dateTime
has a default date, so I have to assume that it was never set by any GPS sentence. If you look at this table, you'll see that the only sentences that will provide the date (not just the time) are the RMC and the ZDA.
I can't see your configuration, so I have to ask:
It is very common to lose the RMC sentence when the wrong LAST_SENTENCE is chosen. If the RMC does not get parsed correctly (or at all), the time may get set by another sentence, but the date remains the default (i.e., the EPOCH date).
When the fix.dateTime
has a correct date and time, the conversions should work fine. You should not need MODIFIABLE epochs, you can just run the entire system off of the POSIX_EPOCH.
I'm using ublox binary format and have enabled all options. stampgmt is a time_t constructed from fix.dateTime in the clock_t constructor, right? stampgmt has Time and Date printed valid and must have been set by GPS as there are no other time source. One would expect that the seconds variable should contain the number of seconds since the configured Epoch, which is POSIX. Well this clearly not the case. So how can one achieve this without needing to fork?
stampgmt is a time_t constructed from fix.dateTime in the clock_t constructor
Actually, fix.dateTime
is converted to a clock_t
seconds since the EPOCH. Then those seconds are used to calculate a new date/time (time_t
) from the EPOCH starting time. It should re-create the original fix.dateTime
values, regardless of the selected EPOCH. If you execute this code:
NeoGPS::time_t epoch;
epoch.init(); // set all members to the EPOCH values (the 0 time origin)
Serial.print( F("EPOCH: ") );
Serial << epoch;
Serial.print( F("\nUTC time: ") );
Serial << fix.dateTime;
Serial.print(F("\nSeconds since EPOCH: "));
NeoGPS::clock_t seconds = fix.dateTime;
Serial.println( seconds );
Serial.print( F("\nDate/Time for EPOCH+seconds: ") );
NeoGPS::time_t timestamp( seconds );
Serial << timestamp;
Serial.println();
You should see that the timestamp
is identical to fix.dateTime
.
I notice that if you select the POSIX epoch like this:
static const uint16_t s_epoch_year = POSIX_EPOCH_YEAR;
static const uint8_t s_pivot_year = 0;
static const uint8_t s_epoch_offset = 0;
static const uint8_t s_epoch_weekday = POSIX_EPOCH_WEEKDAY;
... it does not re-create the same timestamp
from seconds
-- there is an offset in the year. It outputs this information:
EPOCH: 1970-01-01 00:00:00
UTC time: 1917-07-15 19:59:55 <--- wrong!
Seconds since EPOCH: 16919995
Date/Time for EPOCH+seconds: 1900-07-15 19:59:55 <-- also wrong!
You must also set the pivot year and epoch offset variables:
static const uint16_t s_epoch_year = POSIX_EPOCH_YEAR;
static const uint8_t s_pivot_year = 70;
static const uint8_t s_epoch_offset = 70;
static const uint8_t s_epoch_weekday = POSIX_EPOCH_WEEKDAY;
This is what the mutator epoch_year
will do when MODIFIABLE is defined:
static void epoch_year(uint16_t y)
{
s_epoch_year = y; // 1970 for POSIX
epoch_offset( s_epoch_year % 100 ); // 70
pivot_year( epoch_offset() ); // same
}
When I manually set the epoch variables in NeoTime.h as above (with 70s), my test sketch outputs this correct information:
EPOCH: 1970-01-01 00:00:00
UTC time: 2017-07-15 20:04:04
Seconds since EPOCH: 1500149044
Date/Time for EPOCH+seconds: 2017-07-15 20:04:04
It's not clear in the header file that you have to modify those two variables, so I'll update it to look like this:
static const uint16_t s_epoch_year = POSIX_EPOCH_YEAR;
static const uint8_t s_pivot_year = s_epoch_year % 100;
static const uint8_t s_epoch_offset = s_pivot_year;
static const uint8_t s_epoch_weekday = POSIX_EPOCH_WEEKDAY;
Then you only have to modify the epoch year and weekday.
Thanks for the question, this is an improvement!
Yes! Now it totally makes sense, I was nearly pulling my hair for a moment :cry: Thanks for this great library! Keep up the good work.
Hello @SlashDevin, just a quick update. I got a new GPS unit a few days back, and after applying f23421829358c5c50f6cff600a29003f19452427 datetime works as expected.
Final code is here: https://gist.github.com/klausenbusk/c40aeace0edcef2c09aa33225b8e0e77 and change from last time: https://gist.github.com/klausenbusk/c40aeace0edcef2c09aa33225b8e0e77/revisions
I decided not to use latitudeL
, as that would require some sort of converting afterwards (I think).
One question, you wrote <-- notice parens
in our improved example, that is only for readability correct? or does it matter codewise?
Everything expect sendData
is non-blocking now, but sendData
is only blocking when on WIFI (==not moving) and max once per 5 minute. I'm not sure if I could improve that somehow..
But thanks for this great library, it makes everything so much easier :)
Glad to hear it's working!
I decided not to use latitudeL, as that would require some sort of converting afterwards (I think).
Yes, you would have to multiply it by 1.0e-7 to get floating-point degrees, if that's what you are wondering. I think the ESP has true double
support, so it would not lose any precision doing the conversion. Do not use float
value of latitude()
, as it would lose significant digits. Multiply latitudeL()
yourself into a double
. (I should probably look at using double
instead of float
for those platforms that really support it...)
you wrote
<-- notice parens
in our improved example, that is only for readability correct? or does it matter codewise?
It will execute correctly without the parens, but there are several reasons I suggest them:
It seems like the compiler used to issue a warning about these kind of "run-on sentences". :)
Everything except
sendData
is non-blocking now... I'm not sure if I could improve that somehow.
If sendData
just started the process, set some flags and returned, then loop
could check those flags (and ESP status?) to avoid blocking. You would also want to catch trying to send a 2nd batch before the 1st batch is complete.
Hello
I'm trying to create a GPS tracker, which store the current location to a SD card (when moving) and send it to my server when on WIFI. It seems to work, but the timestamp seems to be in the 1970'. Am I doing something wrong?
The hardware is a NodeMCU v3 (esp8266) and a GV-NEO6MV2, and software wise I'm using NeoGPS 4.1.7 and Arduino ESP8266 v2.3.0.
The code is as following:
-- Kristian