Upload fails and queued data

[LoZio]

Hi, I cannot understand after some trials the logic for uploads. When you are logging to a file it is easy to accumulate fixes. But what happens when you have only uploads (i.e. to OpenGTS) and the network is not available? Does the fix is queued for next available upload slot or is discarded? In the latter case, it would be a serious handicap to require 100% network availability everywhere. Thanks

[mendhak]

For that you'd need something like the auto send. When the auto send cycle comes in, and if you're auto sending to OpenGTS, then it should parse the GPX file and resend each point to the OpenGTS server. You can also do this manually through the Upload menu.

Guaranteeing that the points always make it through would require tracking each point and a status and involves quite a bit of overhead. I also don't know what the impact of using sync adapters would be as ideally it should apply not only to OpenGTS but custom URL and all upload targets as well.

[LoZio]

Sorry, I don't understand the first part of your post. Are you saying that currently auto send parses gpx data, or that is should to behave like I said? There are other tools on the market (and in house ones, I wrote one for my company) that simply once a point is acquired it is put in a queue, then when connection is available the queue is spooled, similar to the GPX solution you wrote above. This ensures you to have a real "logger", and not a "maybe logger". I suppose the main "problem" is that your program is thought to be an "interactive" tool, so you can start logging to gpx for example and then upload once you open the tool again. There are lots of cases where it is not like this, think about unattended devices that should log (fleet tracking for example). Sync adapters are a way, but simple queue/dequeue routines activated by connection changes or timers do the work. My 2c

[ambrosa]

Dear "LoZio", many times I've tried to ask mendhak to add some buffering feature regarding real-time remote transmission (OpenGTS and CustomURL). As you say, many other tracking app have this feature and this feature is MANDATORY in my opinion.

I'm not totally conviced by mendhak's reasons. My idea is simple, asyncronous and without any overhead. Dirty code ? Maybe... Use a simple FIFO buffer (a simple dynamic memory array ? temporary file on device ?) managed by OpenGTS (and/or CustomURL) function (in the flow below, you can change OpenGTS with CustomURL: the same behaviours):

Now the app do this job:

• GPSLogger collect a GPS point
• if enabled, run OpenGTS routine
• OpenGTS routine: try to send it
• OpenGTS routine: if fail to send, simply return.

Add a FIFO buffer:

• GPSLogger collect a GPS point
• if enabled, run OpenGTS routine
• OpenGTS routine: put GPS point into the FIFO buffer
• OpenGTS routine: if FIFO has data, try to send it
• OpenGTS routine: if fail to send, simply return. If connection is ok, loop until FIFO is empty.

Instead to try to send ONE point, it try to send ONE OR MORE points...

The only problem is the size of the buffer. If I collect 1 point every 30 seconds, for 12 hour buffer there needs 1440 entries. 100KB memory usage ? I don't have any Android experience... The goal can be 24 hours buffer: the worst scenario is 1 point every 10 seconds => 8640 entry into FIFO buffer for 24 hours

And If user quit app ? Points into FIFO buffer should be 'persistent' so at next run (with OpenGTS enabled) the app can retry to send buffered data with the same schema: when a point is collected, call OpenGTS routine, try to send FIFO buffer content.... User can delete FIFO buffer with a simple Menù option.

[LoZio]

Ambrosa I'm with you with async. I'm not with you with the double memory/disk FIFO queue. I wrote a system with the very same purpose. The mobile app works using a sqlite db.

1. get a fix
2. queue to db
3. try to dequeue (send)
4. on success, remove the sent data
5. goto 1

Point 3 is a simple while loop that scans the database and looks for not sent records (select * from points where sent=false/update points set sent=true where id=xx), being them flagged as "ToSend" and not-deleted rows (select * from points/delete from points where id=xx) (it depends on your idea of db lifecycle and maintenance).

The only 2 things I worked out after some testing is:

• you need a callback routine linked to the network connectivity changed event. When fired && network is up, try to dequeue
• you need a timer for the same reason to clean up the queue
• you need to ponder how many sending threads to spawn when de-queuing. One is not efficient, too many will drain the battery and slow the system. I found my recipe but it depends on the protocol (I'm not using gprmc and opengts).
• would be nice to add SSL support fot HTTPS connections since it's your location data travelling on the net.

Having a memory structure AND a persistence on the disk is double effort for nothing and you can incur in Android killing your memory hungry app. The cost of I/O is really low for this kind of program and sqlite APIs are built-in into the system.

In any case I agree with you, this is mandatory if you want a logger and not a maybe-logger as I wrote before. Let's see how it evolves, I continue to use my app. This is a real pity because Gps Logger is a real cool program and is just two steps away from being great. Bye

[mendhak]

Definitely possible, but I think it'll have a big impact so it's all about having some spare time. You say you've got an app that does this already, is the code available to look at, or maybe you could have a look at GPSLogger's code and see what's involved, contribute a patch, etc?

[LoZio]

I wrote the app but it is a corporate project. I'm looking at your program since I cannot distribute my code. I'm more than happy to help, tomorrow having some time I'll put some "ideas" here.

[ambrosa]

@LoZio (preface: I have no experience in Android programming.).

Misunderstanding: not a "double memory/disk FIFO queue". I mean a simple fifo buffer. Memory OR disk. It was just an idea. I know that sqlite can be a perfect solution but I don't know its impact in Android world. For this reason I think a more simple memory buffer like an array OR a diskfile (or Sqlite ....). Memory buffer is not persistent. Diskfile or sqlite yes, they are. Your solution looks to be 'perfect' but a too complex. One problem is that we also need to manage response from remote server for setting the flag "Sent" for a point (i.e. CustomURL server should answer "ACK" or "NAK" or something else).

It's more simple if app check network every time it needs to send a point: no "callback routine". Exactly it does already: it's only needed to add the buffer.

1. get a fix
2. enter remote communication routine (OpenGTS or CustomURL)
3. OpenGTS (or Custom URL) queue to db
4. OpenGTS (or Custom URL) try to dequeue (send)
5. OpenGTS (or Custom URL) on success, remove the sent data. If queue empty -> exit , if queue not empty -> goto 4
6. OpenGTS (or Custom URL) on failure -> exit
7. OpenGTS (or Custom URL) goto 1

No timer, no complex code, no callback routine. HTTPS ? Not needed in my opinion. Ok, security is a big deal but for this task ... not.

Mandhak has right: GPSLogger is simple. Keep it simple :-)

PS: my little home project based with GPSLogger: http://motolive.zapto.org

[LoZio]

@ambrosa Your logic is basically ok, but when dealing with mobile device connectivity issues are the one that make the difference. You just cannot wait the next fix to send data if you have something to send, it makes the whole process really inefficient. Suppose you're driving through a canyon on on a road with tunnels. The time you have a fix you have no connectivity, and between tunnels you just sit waiting for a fix while you could send the data. I think that your point n.4 can be also triggered on a connectivity change. The good thing is that it is really simple, no complicate code needed. Just register the procedure that runs in (4) with the appropriate broadcast receiver. Take a look here http://developer.android.com/training/monitoring-device-state/connectivity-monitoring.html. This way you can always dequeue data when connection is available, spending ten lines of code and yo ucan skip the timer part. About https: it is terribly inefficient and may be not useful for everyone, but if i want to use it, I should be able to. In Android this leads to thinking about the way you start a connection, since you must choose the right options about treating the certificate given from the server. Again, not 2 lines of code but definitely worth implementing, surely at low priority.

Another point of attention could be how to determine if the data was sent ok or not, each server may behave differently. For example OpenGTS always responds with a 200 HTTP status code even for errors, it just gives you back "ERROR" in a "200 Ok" page.

If anyone is interested in testing, I can give accounts for testing on my OpenGTS server. Bye

[ambrosa]

I drive motorcycle since 30 years ago and I'm a GPS tracking fan since years. Using stand-alone devices or, in more recent time, with Android phone. For sports I use Endomondo and Runtastic. For driving I use GPSLogger with CustomURL coupled with my server MotoLive. Also I record the KML track so if I loose some point (transmissione failure), later I have the full track.

I know very well issues about 2G/3G no signal, drive in foreign countries without roaming (but only wifi @ hotel at the evening), tunnel and so on.

Usually a good tracking is made with 1 fix every 20 sec (or 30 sec). Trying to send data one time every 30 sec. in my opinion is enough. Without use callback routine where data are available.

But it's only my opinion... :-)

CustomURL server can answer exactly OpenGTS does. So if data is NOT processed correctly, OpenGTS answer with HTTP 200 and "ERROR". Only "ERROR" or there is also some errore code i.e. "ERROR 123" ? And if data is processed correctly, OpenGTS aswer with ? "OK" ? None ? Can you give me some details ? So I put in my MotoLive server and we have a test platform for CustomURL. Recording is granted to demo user (user "demo", pass "demo")

[LoZio]

Loosing the ability to push data for saving maybe 10 lines of code is not a great idea in my opinion. OpenGTS returns ERROR or OK and nothing else, both in a 200 page. Other (smarter) server reply with several 50x codes to allow the client to better handle the failure. In our case, a simple boolean ERROR/OK behaviour is sufficient. I do have a server ready to use if you want to try a real OpenGTS server (don't know what motolive is)

[ambrosa]

MotoLive.zapto.org is a my personal home project. For me, my friends and parents. It's like OpenGTS server but more focused to motorbike trip and real-time tracking. It's a small web server using PHP5 + SQlite3 + Lighttpd , it runs on a small ARM board (www.cubieboard.org). Mass storage: 16GB microSD card class 10.

Some time ago I asked to mendhak to add a new transmission protocol : CustomURL. OpenGTS protocol lacks many features and data. MotoLive can receive data with CustomURL and OpenGTS protocols.

Today I will modify my MotoLive "CustomURL" and "OpenGTS" code to add "ERROR" and "OK" answer. In my opinion it's ok. There is no need to have more detailed error code.

[LoZio]

[OT] I too have created some "compact" solution based on RaspberryPi, never tried Cubie but we ordered some Bananapi since Rapi performance on net and sd are too low [/OT] Back to topic: it's good to have some system that behaves "like" opengts, but for testing I suppose it's better to use a "real" system with all of its timings and bugs :) Anyway don't know your skill but adding custom protocols with all the data you need to OpenGTS is definitely possible, and you inherit for free all the fleet management capabilities. You can also write some lines of PHP and "integrate" your system with OpenGTS, a la Traccar for example.

[ambrosa]

"Anyway don't know your skill but adding custom protocols with all the data you need to OpenGTS is definitely possible" DONE: it's "CustomURL" protocol added by mendhak about 2 months ago :-) But I'm not interested into "OpenGTS server". It doesn't match my needs. For this reason I've created from scratch "MotoLive server".

Again about your idea to have a "callback routine" called when network is reachable. Starting from 3 years ago I've tested many Android tracking GPS apps. For sport (Runtastic, Endomondo...) or drive (GPSLogger, MyTracks ...). Because often I'm into mountains (trekking, hiking, driving with my motorbike..), 2G/3G usually is weak. Often while you drive/trek, network signal appear then disappear, is weak, is strong, no data connection (only pure GSM voice) etc. etc. A very UNSTABLE network signal. But it's a very common situation. My fear: if you use "callback routine", many time is called (because network signal appear and then disappear) without sending data => you drain battery. I've experience many times about this issue. And so I disable data connection to save battery. For this reason I think that a more simple approach (simple trying to send data every fix as I've explained before) can reduce this issue.

[OT] I've also a Raspberry but it's performance is poor. Cubieboard (and CubieBoard 2 dual core) run very fast ! I'm very happy with it. [/OT]

[LoZio]

I referred OpenGTS because it is one of the officially supported servers, without the need for custom urls. In any case the "upload" should be abstracted by enclosing in a class that has a different instance for different providers (custom url, osm, opengts, ...) because each of them gives you back different status code and data. About the callback: you maybe right, since you had those experiences and I had different ones (and finally implemented and using callback+timers since the times we used VHF radio for tracking). Of course the main problem to solve is to have async upload, the callback can be easily added later with a flag to use it or not depending on what you want each time. It should be a Sunday afternoon task...

[ambrosa]

[OT] VHF radio ? mmhh AX25 ? I'm a Radio Amateur and for year I was very active with UHF and VHF digital transmission (AX25) @9600 baud. But now ... Radio Amateur concept is dead. Except for some strange things like Meteor Scatter :-) [/OT]

The callback idea is correct in a "binary" fashion into area with good 2G/3D coverage: one time you have 2G/3G signal, then you have no signal, after 30 second signal is good again. Think a car into a city tunnel.

But driving into remote country (or mountains) with weak signal ... needs a completely different approach like I said before. It's better trying slowly the network connection (i.e. one time every fix) than callback an 'upload routine' immediately after Android say "Hello, network is up again" (and after 0.5 second network is down again....).

And I'm totally agree with you about "abstract" data. GPSLogger should:

• get a fix
• save data into sqlite buffer
• run 'upload provider' enabled by user: can be "SAVE FILE LOCALLY INTO A GPX FILE" "SAVE FILE LOCALLY INTO A KML FILE" "SEND REMOTE WITH CUSTOM URL" "SEND REMOTE WITH CUSTOMURL" ... and all provider take data from sqlite buffer.

But this require to rewrite all routine. I don't think that mandhak is agree....

[mendhak]

Oh I do agree! It's just a matter of time; I think doing this won't be a quick fix because there's a lot of 'stuff' going on in the code, and I need to find time to do this, that's why I was asking if anyone could look at the code and see how they would do it. GPSLogger was originally created just for GPX files as a one-way-write, since then has evolved a lot.

The structure will need thinking over, to handle not just log points but annotations, the user stopping logging while items are still in buffer, things like that.

Eventually it'll make the sequence much cleaner; when a point is received, writing just once and then outputting files later. You could even choose not to clear the database and then store a history of locations and regenerate files when wanted.

[LoZio]

@ambrosa [OT] It was actually something more than amateur radio :), we developed GPS avionics since the beginning, then tracking devices of any size and purpose. We used any mean to transfer data. Today is the turn of 3g networks :):) [/OT] @mendhak If the user stops logging while data is in the queue there is no problem. The data already captured will be sent when possible. A simple "Flush" button can delete all local data that is not sent, if the user wants to (maybe to avoid sending spurious data, tests, or just to clean up the things if they went south). My "idea" is to create a very simple class incapsulating the queue. Methods addNewFix(fix_data_structure) getNextToSend(): returns a data structure that includes the fix data above and an ID, maybe the unique ID of the row in the db clearSentFix(ID): the ID from above call. This deletes the row/marks it sent. This is what is really needed. Some sugar: clearAllQueue(): drop and recreeate the fix table (also useful to call sometimes to delete database journal files) getFixCount(): to know how many fixes are waiting to be uploaded, to notify the user there is local data waiting.

After obtaining a fix call addNewFix. Then (if we go synchronous) a loop like while network is "online" && haveMoreData ( fix=getNextToSend() ret=send_data(fix) if ret==Ok clearSentFix(fix.ID) haveMoreData=getFixCount() ) To make it simple you should have only one "remote" service configured. This is my idea of the basic workings. Creating a callback enabled program will simply require to register the broadcast handler and inside this call the loop above (in that case a global locking flag must be used).

Some assembly required. Results may vary. Bye

[ambrosa]

If you want to see a GPS tracking app with a very good "queue" management, see www.btraced.com (available for iPhone and Android). It has also a very nice layout :-)

But the app itself is not very good: it drain battery very very fast, Distance Filter and Time Filter use an "OR" logic (a point is recorded whichever filter come first), no "annotation" (waypoint) and use a proprietary protocol (XML based) to trasmit data to remote server (but it's very well documented: I've implemented it in minutes into my server).

But queue works very fine :-)

[LoZio]

This is from the guys at OpenGTS developement and of course it sports queue management. https://play.google.com/store/apps/details?id=org.opengts.client.android.cgtsfre Just released to the public (and no, I'm not associated with them)

[ambrosa]

Oh, thanks. I'll try it in these days.

Yesterday I made a 400 Km trip into mountains with my motorbike. Tracked with GPSLogger (50%) and Btraced (50%). GPSLogger was a disaster. Without caching/queue data I loss transmission about 30% fix (but I've also recorder the GPX file so I have all data). Btraced worked fine.

And I'had an issue with my phone regarding 3G connection. Till now I had 3 Android device (Samsung Ace, Samsung galaxy W, Sony Xperia T). All of them had a problem (I suppose is Android bug): with 3G enabled, if you go in and out many times into zone where phone need to switch from 3G to 2G and viceversa, sometimes 3G module crash and you need to restart the phone to have network connectivity again. No other solution works. Queue data is mandatory....

[lcgerke]

I've been using btsync (aka sync on google play store) to synchronize the data folder from gpslogger to my other machines with very good success.

seems like it sort of makes all the other auto-send stuff moot. highly recommended.