Export aborts with error

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Log data.
2. Choose to keep uploaded files and export GPX.
3. Upload.

What is the expected output? What do you see instead?
Expected: a clean upload. Instead: Cell & wifi data is exported, then uploaded, 
finally GPX is exported and eventually I get a message that the upload failed. 
On one occasion during pload I noticed that the number of files went down one 
by one, then jumped up by ~8–10 files, then slowly went down again, then 
jumped up again. The whole cycle repeated many times until the number of files 
was at zero. On other occasions the number of files went down steadily.

What version of the product are you using? On what operating system?
0.7.9 (the Dec 27 version from Google Drive.) OS is Cyanogenmod 10.2 on Nexus S.

Please provide any additional information below.

Original issue reported on code.google.com by mich...@vonglasow.com on 23 Jan 2014 at 7:35

[GoogleCodeExporter]

Build is e569140.

Original comment by mich...@vonglasow.com on 23 Jan 2014 at 7:42

[GoogleCodeExporter]

Hey Michael,

this thing is a bit complicated: as many uploads cause troubles on the server 
side, in 0.7.9 the number of wifis and cells per file was increased (meaning 
less files, but larger file size). When doing that I was unaware, that the 
server denies to large file. I haven't yet found the exact reason for that. The 
solution in the 'old' client (<0.4.99) was to count bytes on each file. Rather 
than implementing that solution, I prefer to fix things on the server side.

I already have a new prototype for this (also including your excellent mapnik 
visualisation on the front-end :-) I'll try to push things forward as fast as 
possible, but am  quite busy right now..

The good thing on this issue is, it's annoying, but no data is lost.. Once we 
get things fixed on the server side, uploads will run smoothly..

Cheers
Toby

Original comment by wish7code on 23 Jan 2014 at 9:11

• Changed state: Started

[GoogleCodeExporter]

Good to know. However, that means I need to keep the data that failed to upload 
and retry the upload when the server side is fixed, correct?

Original comment by mich...@vonglasow.com on 26 Jan 2014 at 12:12

[GoogleCodeExporter]

Actually... yes :-(

Original comment by wish7code on 26 Jan 2014 at 11:06

[GoogleCodeExporter]

Just had another aborted upload for a really short log (a few seconds long, 
with 14 WiFis and 1 cell). The error message I got was the same, but I doubt 
file size was an issue here. I suspect there might be something else, 
considering also that I haven't been able to upload a single file with the 
version in question.

Original comment by mich...@vonglasow.com on 8 Mar 2014 at 11:44

[GoogleCodeExporter]

Any news on this one yet?

I just retried the upload with 14 WiFis and 1 cell. The two XML log files are 
posted to the server and accepted with HTTP 200 OK. However, the subsequent 
HEAD operation for the two files returns HTTP 400 Not Found. For WiFi, the path 
that the app tries is:
http://www.openbmap.org/upload_wifi/maps/stanton_V1_log20140306155751-wifi.xml

I tried downloading one of my earlier uploads (from August 2013) and got the 
same error.

Can we rule out an incorrect path as the culprit here? File size is really 
unlikely, especally the cell log could hardly be any smaller...

Original comment by mich...@vonglasow.com on 7 Apr 2014 at 10:22

[GoogleCodeExporter]

Hey Michael,

sorry, guess I've missed your last comment from March 8th. 

> Can we rule out an incorrect path as the culprit here?
Guess so, there's really no magic at this point.

Would you mind sending me the two above mentioned files for further examination?

The good news: new server stack is steadily progressing. Basically the new 
stack is already processing uploads, displays maps and supports Firefox 
compatible geo-location, but needs some more design 'polishing'. In the next 
couple of days I've got my holydays so there might be the chance to bring 
things forward then. Thus we could use the new stack to address the issue 
server-side. Yet no promise :-)

Cheers
Toby

Original comment by wish7code on 8 Apr 2014 at 8:46

[GoogleCodeExporter]

Sorry, forgot about the file altogether. Here it is.

Any news about the new server stack? Or anything else about the issue?

If this is going to take longer, could you push out an interim release of 
Radiobeacon which will restore the old upload behavior for the moment? I'm 
sitting on a database of 100 or so logs, which I would love to upload, but 
can't... and I'd like to free up some space in my DB again.

Original comment by mich...@vonglasow.com on 20 May 2014 at 9:34

Attachments:

• V1_log20140306155751-wifi.xml

[GoogleCodeExporter]

Hey Michael,

thanks for the file. I spent some time debugging the upload and surprisingly
using Google Chromes Advanced Rest Client the file uploads fine, see attached 
screenshot.

So my first suspicion, that the upload was denied due to invalid wifi ssids 
(one ssid contains a dot) was wrong. At the moment, I really don't have a clue, 
what's going on server-side. Unfortunately I don't have the old servers php api 
available, so everything is trial-and-error.

I'll contact Mike to check whether we could integrate the new server stack 
somehow (basically running under a sub-domain at openbmap.org). Right now the 
new server stack covers all basic features, so we might give it a try..

As a quick-fix, I could also offer to upload the exported wifi files via ftp. 
That way, they'll get uploaded for sure.

Original comment by wish7code on 25 May 2014 at 12:02

Attachments:

• screenshot.png

[GoogleCodeExporter]

Thanks for the offer, I'll get back to that.

Looking at
https://code.google.com/p/openbmap/source/diff?spec=svnb0f7bc9f0b0aab2bcb812ece3
62f5000dd070ce3&r=b0f7bc9f0b0aab2bcb812ece362f5000dd070ce3&format=side&path=/and
roid/src/org/openbmap/soapclient/FileUploader.java
I see that one of the changes introduced in 0.7.9 was to check for the presence 
of the uploaded file, as the server will send HTTP 200 even in the case of 
failed uploads.

So the failed uploads may have been an issue much earlier, just that nobody 
noticed with the earlier Radiobeacon versions. Right now I'm wondering how many 
of my logs got uploaded at all...

I'll compile a list of all my logs (luckily I've kept all I've ever uploaded) 
and send it to you, so you can check on the server side what got uploaded and 
what did not.

On a side note, it would be nice if I could log in to openbmap.org and see a 
list of the logs I submitted, maybe even with import status. Right now there's 
no reliable way of telling how much of my data made it to the server.

Original comment by mich...@vonglasow.com on 28 May 2014 at 4:03

[GoogleCodeExporter]

Attached are lists of all my (attempted) uploads. I started getting error 
messages around January 1, thus files with a 2014 timestamp will likely not be 
in the database.

Original comment by mich...@vonglasow.com on 2 Jun 2014 at 7:36

Attachments:

• openbmap-cellular.txt
• openbmap-wifi-main.txt
• openbmap-wifi-processed.txt
• openbmap-wifi-krank.txt

[GoogleCodeExporter]

I see that 0.8.3 switched to the new server. I just updated to 0.8.4 (from 
F-Droid, reports itself as "Unknown build") and tried to upload a short log (14 
WiFis, 2 cells). However, after uploading I still get the error message that 
the upload failed.

I can't tell at this point whether this was really in issue with the upload or 
whether it's just the verification that fails (I have a suspicion that the data 
might have made it to the server but verification belives it didn't).

Original comment by mich...@vonglasow.com on 18 Dec 2014 at 11:56

[GoogleCodeExporter]

PS: I just tried to upload three logs of areas that are blank on the map so 
far. If they show up on the map, I will recognize them. If that is the case, we 
will know if it is the upload or the verification that's faulty.

How long does it take for an upload to appear on the map?

Original comment by mich...@vonglasow.com on 18 Dec 2014 at 12:22

[GoogleCodeExporter]

Hey Michael!

Thanks for reporting. I just browsed the code again. 0.8.4 didn't come with 
great changes, especially the upload part remained unchanged compared to 0.8.3

On the server-side I checked the upload folder: there're no files from your 
user account, although we've got a couple of other people using 0.8.4. So for 
some strange reason the files didn't find their way to the server. Could you 
provide a catlog the next time this happens?

> How long does it take for an upload to appear on the map?
Basically there 4 steps following a upload: 
(1) "file import" loads the raw data into the database,
(2) "processing" aggregates the data and calculates a estimated position based 
on raw data
(3) "statistics" calculates user, cell provider and country stats
(4) "maps generation" updates the map tiles

The bottlenecks are steps (1) and (2), as they are quite ressource intensive. 
At the moment I'm experimenting with different import schedules. On approach is 
to run steps (1) and (2) as often as possible, meaning several times an hour, 
to keep the number of files with each import small, the other approach is to 
import only on a daily to save server ressources for other tasks. I still 
haven't found the optimal solution for that.

Original comment by wish7code on 18 Dec 2014 at 9:43

[GoogleCodeExporter]

I just attempted another upload with a packet capture running, and I found that 
my uploads were rejected with "401 Unauthorized".

Are the old openbmap accounts valid on radiocells.org? I've tried to recreate 
an account with the same name, but it told me my user name and e-mail address 
were already in the database. However, I can't log in on the web site with my 
openbmap credentials. I've requested a password reset (and my account name a 
few minutes later) but didn't get an e-mail yet.

Original comment by mich...@vonglasow.com on 22 Dec 2014 at 3:11

[GoogleCodeExporter]

PS: still no mail from radiocells.org. Can you check my account data? My user 
name is stanton, you have the e-mail address.

Original comment by mich...@vonglasow.com on 22 Dec 2014 at 4:57

[GoogleCodeExporter]

Email address for user stanton is fine, I checked.
I'll have a look at the password reset function right now and come back to 
you...

Original comment by wish7code on 22 Dec 2014 at 5:04

[GoogleCodeExporter]

Send you a PM just a couple of seconds ago..

Original comment by wish7code on 22 Dec 2014 at 7:32

[GoogleCodeExporter]

Just a short update: 37 files have arrived (as of yesterday evening)..
See https://radiocells.org/stats/scoreboard

For the records: password reset link also has been fixed. Now works again 
automatically..

Original comment by wish7code on 23 Dec 2014 at 7:37

[GoogleCodeExporter]

I am seeing cells (though not at all zoom levels), which are definitely from 
those uploaded files, but no wifis. Is that a tile rendering/refresh issue? How 
long does it take for updates to be reflected in the tiles (when the tile has 
been rendered before the update)?

Original comment by mich...@vonglasow.com on 25 Dec 2014 at 7:30

[GoogleCodeExporter]

Update: I am seeing wifis in some (though not all) areas. Looks like we're 
getting there...

Original comment by mich...@vonglasow.com on 27 Dec 2014 at 9:47

[GoogleCodeExporter]

I'll rebuild the map and check the 'krank' wifi files also to see wether this 
helps..

Original comment by wish7code on 28 Dec 2014 at 7:07

[GoogleCodeExporter]

There are still some areas in which I don't see any wifis on the map. One is 
around Colmar, France; I collected data there in June 2014. I am seeing cells 
but no wifis. Cells disappear at lower zoom levels, which indicates a tile 
refresh issue, though this is not necessarily the sole cause.

Original comment by mich...@vonglasow.com on 1 Jan 2015 at 10:33

[GoogleCodeExporter]

I see a lot of contributions (wifis created at May 24) in the Colmar region.
The main database is currently being rebuilt from scratch, so stay tuned till 
tomorrow. Then we can analyse in more detail..

Please note: due to the database rebuilt, tiles won't be refreshed today..
Cheers
Toby

Original comment by wish7code on 2 Jan 2015 at 8:11

Attachments:

• Colmar.png

[GoogleCodeExporter]

Yes, these are mine. I'm seeing this happen in a few areas: cells but no wifis. 
I'll wait and see...

Original comment by mich...@vonglasow.com on 2 Jan 2015 at 11:13

[GoogleCodeExporter]

I am seeing wifis in areas where they were previously missing – this looks 
like all my uploads up to a few days ago are now being rendered.

However, this only works at zoom level 8 – other zoom levels have cells but 
no wifis. Looks like there's still a tile refresh issue. You might want to look 
at the OSM rendering stack and see if you could reuse some of that logic in 
order to mark tiles as "dirty" when new data is added in an area.

Original comment by mich...@vonglasow.com on 3 Jan 2015 at 2:41

[GoogleCodeExporter]

The server makes use of Hiroshis excellent tileman stack 
(https://github.com/osmfj/tileman), to my knowledge also used for hosting the 
Japanese OSM website. Thus in theory it supports on-the-fly rendering of 
missing tiles. The problem here is, that our server isn't powerful enough to 
render the wifi layer on-the-fly, so it runs into timeouts.

I will modify the rendering stack so it'll (pre-)render a country if there've 
been recent uploads. Let's see wether this helps..

Original comment by wish7code on 3 Jan 2015 at 7:27

[GoogleCodeExporter]

That probably makes sense for lower zoom levels. Putting a cap on it (i.e. no 
more than one complete re-render per 24 hours) probably makes sense.

If you can tweak the rendering stack in more detail, for the higher zoom levels 
it might make sense to pre-render only those tiles that have been requested 
earlier (and more frequently/recently requested ones first).

Original comment by mich...@vonglasow.com on 3 Jan 2015 at 7:32