How to see if there are unsent informations and if yes force to report them ?

[Valeryan24]

Hello,

MozStumbler is a fantastic idea to collect and provide a collaborative and open-source map of GSM cells, WiFi access points correlated to GPS locations for Firefox OS.

I have a question regarding the reports done :

I noticed that sometimes, there can be several minutes and lots of scanned positions (with a GPS fix) between 2 reports, if we stop the detection (because going into a building for example, or low battery), even with 3G data or wifi available, sometimes MozStumbler does not send the last reports immediately, we have to wait the next session.

Or if I wait many seconds / minutes after stopping detection, I see the number of reports sent increase 2 or 3 times more if many have to be uploaded, and normally number of reports approximately matches number of detections.

But if I quit MozStumbler before all the reports are done, and then restarts it, or if I put the phone on plane mode, or restart manually the phone, or if I have the problem described in issue 249 (crash), often when I restart MZ, all counts (locations, WiFi, reports) are reset back to 0 !

I know that all sent reports are well saved by Mozilla (I can see the Leaderboard points and map coverage increasing), but what happens to the reports not sent at the last close of MozStumbler ?

Is it sure that there are always sent on the next session ? Is there a way (with logs, exploring cache datas of MZ) to verify if there are not, on the several days of scanning, sometimes some locations and datas (GPS, wifi, cells...) that have been well scanned, written on the phone but never sent to Mozilla ?

And if yes is there a way to manually upload them, now ? This is not a matter of points (I don't care), but it can be frustrating when I spend lots of time, especially when walking on the town to catch all the small areas to get locations and wifi points on the most complete way, so having less locations than if I drive by car, so less reports sent per minute, and as I can not manually force all reports before stopping MZ, to think that sometimes all or part of what I've done is useless because never sent..

Thanks in advance.

[cpeterso]

hi @Valeryan24: to reduce network traffic, MozStumbler saves measurements locally and uploads a "report" when it has at least 20 measurements. If you exit the app (or it crashes), the on-screen counter is reset to 0, but unsent reports should still be saved, ready for upload. Our UI's terminology is admittedly a little confusing because we show counts for "satellites", "locations", "access points", and "reports" and then "points" on the leaderboard which are all different things! <:\

The "Stop Scanning" button will try to force an immediate report upload, if the network is available.

[Valeryan24]

Hello,

It would be great to be able to read in the logs (directly from the app or through a text editor) easily which reports / positions / WiFi and cell points have been well sent and registered by Mozilla or not, if possible. Or to have his own Map available with all the locations scanned :)

[cpeterso]

A simple solution might be to change the UI so it displayed "Reports: X of Y", where X is the number successfully uploaded and Y is the number of unsent reports + X. I know I would find the information helpful. :)

Implement this feature would not be very difficult. The UI would just need a way to query Reporter.java's mReports.length(). This data even would be restored after the stumbler restarts because Reporter.java reads back the unsent reports from the saved preferences when starting:

https://github.com/mozilla/MozStumbler/blob/b44bffed991f410968107941c73558d561b2ff11/src/org/mozilla/mozstumbler/Reporter.java#L68

[Valeryan24]

Hello,

Any planification for commits implementing more details for sent reports as suggested above (xxx of yyy) ?

As you can also see in issue #276 I still get situations where I can not know if missing reports after a session are finally well sent or never.

Ex : see http://i.imgur.com/8J4T99G.jpg

There are the scannings of 3 days.

• Picture 1 : Only 2 reports for 152 registered positions sent. I left the MozStumbler application on the top of screen (without clicking on home button) during 1 h after stopping analysis, but no more report sent. On the night, I put the phone on plane mode, next morning when I restarted MozStumbler, no information that the 150 missing reports were sent, and I have no way to verify it as all numbers were back to zero.
• 2 and 3 : Scanning of sunday, 3 different drive sessions, without closing MozStumbler between each one. At the end of the day, 10723 reports for a total of 10977 positions. So 250 unsent.
• 4 and 5 : On the next morning, after plane mode on the night, same thing as before : all MozStumbler counts are reset to 0, and no information that the 250 missing reports have (or never) well been sent to Mozilla.
• 6, 7 and 8 : The positions scanned during the morning are almost all sent this time, but no more.
• 9 : Upgrade to MozStumbler 0.8.2, all counts reset to 0 again, one drive session, 1147 reports for 1158 locations.

So it's frustrating because almost at each analysis, I wonder if some reports are missing or not, having the real info would be very useful !

[Valeryan24]

Or as suggested, a "Force report" button to send all the last missing locations if we have WiFi or 3G connection after a session would be great. Thanks !

[cpeterso]

I investigated this problem some more. Pressing the "Stop Scanning" button tries to force any unsent reports to be uploaded, unless there is no network connectivity. So we might have a bug when trying forcing the reports (but I don't really see any obvious problems when I reviewed the code and adb logcat logs), so the problem might just be that no network connectivity is detected when "Stop Scanning" is pressed. (There very well could be a bug in our NetworkUtils.isNetworkAvailable() function.)

[Valeryan24]

This is not so simple : sometimes, when I stop the analysis, it sends some reports, so I have well network connectivity (at least 3G, often WiFi, too), but not all of them.

Ex on picture 3 : when I stopped, I had a total of 10977 locations scanned, the reports were approx. 10600, it sends many of them to reach 10723, but still 254 are missing. On the previous screenshot (N° 2), 7388 of 7552 only were sent, it seems when I stopped the analysis the last measured reports were sent, but "somewhere" 254 on this day were not.

That was my first question : is there a way, on the cache or logs of MozStumbler, to verify which locations have been or not reported ?

If I well understand, there should be 1 report for each location recorded. So if some are never sent and if this is not a network bug, could it be that some locations are incorrect, incomplete (GPS fix was missing...) and that would mean the count of locations (or reports) is false ?

[cpeterso]

Yes, there should be 1 report sent for each location scanned. We should make the UI description clearer. I had to double-check the code to make my understanding of a "report" was correct.. <:)

I was able to reproduce a version of this problem with debug logging. The force report failed (leaving the screen with 3 locations scanned, but only 1 reported) due to a server disconnect. I resumed scanning and stopped again; this time the force report succeeded (leaving the screen with 3 locations scanned and 3 reported).

Maybe intermittent server problems are causing problems? But that seems unlikely to be the cause of this bug report because I don't think there is any reason the last report would be more likely to cause a server or network problem.

@hannosch: Do you know why the server might hang up on the client? The client threw an exception when reading the SSL socket, so I assume the client was trying to read the server's response.

javax.net.ssl.SSLException: Read error: ssl=0x73b6a758: I/O error during system call, Connection reset by peer
 at org.apache.harmony.xnet.provider.jsse.NativeCrypto.SSL_read(Native Method)
 at org.apache.harmony.xnet.provider.jsse.OpenSSLSocketImpl$SSLInputStream.read(OpenSSLSocketImpl.java:673)
 at libcore.io.Streams.readSingleByte(Streams.java:41)
 at org.apache.harmony.xnet.provider.jsse.OpenSSLSocketImpl$SSLInputStream.read(OpenSSLSocketImpl.java:657)
 at libcore.io.Streams.readAsciiLine(Streams.java:201)
 at libcore.net.http.HttpEngine.readResponseHeaders(HttpEngine.java:579)
 at libcore.net.http.HttpEngine.readResponse(HttpEngine.java:827)
 at libcore.net.http.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:283)
 at libcore.net.http.HttpURLConnectionImpl.getResponseCode(HttpURLConnectionImpl.java:497)
 at libcore.net.http.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:134)
 at org.mozilla.mozstumbler.Reporter$1.run(Reporter.java:222)
 at java.lang.Thread.run(Thread.java:841)

[hannosch]

@cpeterso I thought that after Doug's patch landed, we catch all Exception classes and if any kind of error happened we just try to sent the data again. So if there's a connection error, timeout, SSL error it should just re-try.

I do see a number of dropped connections on the server, but those are expected when dealing with cellular networks. A lot of them are client side connection timeouts (the client goes away, maybe moved to an area with no or flaky data connections). Another source is server side timeouts, for example if the client takes too long to upload the data. I think the current limit is 2 minutes for any one request, which is probably not enough if you try to upload a large batch over a non-3G connection.

Without any more details, I don't know what to look for. But in general I expect to see lots of connection related problems and we just have to re-try or maybe add some more checks and not upload data over really slow connections.

[Valeryan24]

Last example : at 8:15 on the morning, 239 locations registered, 238 reports sent. At 13:03, I made a second ride, 506 locations in total, but only 309 reports.

When I arrived and just before stopping analysis, there were still 238 reports sent, those from the first drive session, I stop scanning and then 309 - 238 = 71 reports are sent.

So (you can verify it on the screenshot) I had full 3G and WiFi network working. But 11 minutes after stopping, no more report sent and almost 200 missing !

When I restart scanning at 14:03, this time immediately reports are sent up to 505, fine.

So my question : if I had closed MozStumbler, launched it again with all counts reset to zero, would also have the missing reports nevertheless been sent ?

If no, there is a bug. If yes as you tell me, perfect, but in this case I would have no way to know it, as the reports number would not have increased...

Perhaps, if it is possible, it could be great to add the information on past sessions and reports - not on the main screen because there are already many informations, but just in the top right menu, in addition to "Preferences / parameters" and "About MozStumbler", why not a row with "Activity log", which would open a screen listing, per day or per session, the date / hour / number of locations scanned / nb of reports sent / nb of WiFi access detected, with the most recent on the top ?

It would allow to have its own history and be sure our reports are useful.

[cpeterso]

@hannosch: the stumbler will retry failed uploads, but not immediately. The problem some people (including me) see surprisingly often is that last upload fails. Pressing the "Stop Scanning" button will force an upload, but if that fails, we never get around to retrying (because we've stopped scanning, so no new data is coming in). The client could retry more aggressively, but in my testing, I see socket exceptions on the forced upload almost 50% of the time. Maybe we are seeing the Android socket bug that is worked around in @illarionov's PR #282.

[hannosch]

Sounds like the Android socket bug is the culprit. Neither the network nor the server are so bad it would fail 50% of the time.

[Valeryan24]

There are 2 situations where reports are missing :

• Fail to upload last reports : I made a very long trip by car on Saturday, scanned more than 11.700 different locations, at the arrival 10.600 were sent, last report 3 minutes ago. I stopped analysis and waited, but no more report sent, I knew there were some datas still collected and I had 3G network connection. One hour later, I, just restarted and immediately stopped analysis, 89 new reports were sent. If I had killed MozStumbler, not sure there would have been on the next start...
• On a total of more than 11.700 locations scanned and registered, "only" 10.689 were sent, it means that 10% (!) of them were never reported. That's a lot ! Why ? Server fail, 3G network connection not available on the moment to send the 100 last locations ? But normally MZ should retry after to send all datas. If I look at the map, now we have the exact details of all the locations, I can see at several places on my trip that, times to times, a part of the road is missing (without reports), and I'm sure locations were collected because I checked regularly that MozStumbler was active, scanning and detecting both locations / GPS positions / WiFi access. Sorry I haven't the screenshot, it failed to register, but again it seems there are some "ghost" locations that MZ says it scanned but that are never reported... And I can see on the map where exactly.

[Djfe]

implemented with v0.13.0 -> can be closed

[cpeterso]

Yep! Thanks, @Djfe.