Export aborts with error

GoogleCodeExporter commented 9 years ago

What steps will reproduce the problem?
1. Log data.
2. Choose to keep uploaded files and export GPX.
3. Upload.

What is the expected output? What do you see instead?
Expected: a clean upload. Instead: Cell & wifi data is exported, then uploaded, 
finally GPX is exported and eventually I get a message that the upload failed. 
On one occasion during pload I noticed that the number of files went down one 
by one, then jumped up by ~8–10 files, then slowly went down again, then 
jumped up again. The whole cycle repeated many times until the number of files 
was at zero. On other occasions the number of files went down steadily.

What version of the product are you using? On what operating system?
0.7.9 (the Dec 27 version from Google Drive.) OS is Cyanogenmod 10.2 on Nexus S.

Please provide any additional information below.

Original issue reported on code.google.com by mich...@vonglasow.com on 23 Jan 2014 at 7:35

GoogleCodeExporter commented 9 years ago

Build is e569140.

Original comment by mich...@vonglasow.com on 23 Jan 2014 at 7:42

GoogleCodeExporter commented 9 years ago

Hey Michael,

this thing is a bit complicated: as many uploads cause troubles on the server 
side, in 0.7.9 the number of wifis and cells per file was increased (meaning 
less files, but larger file size). When doing that I was unaware, that the 
server denies to large file. I haven't yet found the exact reason for that. The 
solution in the 'old' client (<0.4.99) was to count bytes on each file. Rather 
than implementing that solution, I prefer to fix things on the server side.

I already have a new prototype for this (also including your excellent mapnik 
visualisation on the front-end :-) I'll try to push things forward as fast as 
possible, but am  quite busy right now..

The good thing on this issue is, it's annoying, but no data is lost.. Once we 
get things fixed on the server side, uploads will run smoothly..

Cheers
Toby

Original comment by wish7code on 23 Jan 2014 at 9:11

Changed state: Started

GoogleCodeExporter commented 9 years ago

Good to know. However, that means I need to keep the data that failed to upload 
and retry the upload when the server side is fixed, correct?

Original comment by mich...@vonglasow.com on 26 Jan 2014 at 12:12

GoogleCodeExporter commented 9 years ago

Actually... yes :-(

Original comment by wish7code on 26 Jan 2014 at 11:06

GoogleCodeExporter commented 9 years ago

Just had another aborted upload for a really short log (a few seconds long, 
with 14 WiFis and 1 cell). The error message I got was the same, but I doubt 
file size was an issue here. I suspect there might be something else, 
considering also that I haven't been able to upload a single file with the 
version in question.

Original comment by mich...@vonglasow.com on 8 Mar 2014 at 11:44

GoogleCodeExporter commented 9 years ago

Any news on this one yet?

I just retried the upload with 14 WiFis and 1 cell. The two XML log files are 
posted to the server and accepted with HTTP 200 OK. However, the subsequent 
HEAD operation for the two files returns HTTP 400 Not Found. For WiFi, the path 
that the app tries is:
http://www.openbmap.org/upload_wifi/maps/stanton_V1_log20140306155751-wifi.xml

I tried downloading one of my earlier uploads (from August 2013) and got the 
same error.

Can we rule out an incorrect path as the culprit here? File size is really 
unlikely, especally the cell log could hardly be any smaller...

Original comment by mich...@vonglasow.com on 7 Apr 2014 at 10:22

GoogleCodeExporter commented 9 years ago

Hey Michael,

sorry, guess I've missed your last comment from March 8th. 

> Can we rule out an incorrect path as the culprit here?
Guess so, there's really no magic at this point.

Would you mind sending me the two above mentioned files for further examination?

The good news: new server stack is steadily progressing. Basically the new 
stack is already processing uploads, displays maps and supports Firefox 
compatible geo-location, but needs some more design 'polishing'. In the next 
couple of days I've got my holydays so there might be the chance to bring 
things forward then. Thus we could use the new stack to address the issue 
server-side. Yet no promise :-)

Cheers
Toby

Original comment by wish7code on 8 Apr 2014 at 8:46

GoogleCodeExporter commented 9 years ago

Sorry, forgot about the file altogether. Here it is.

Any news about the new server stack? Or anything else about the issue?

If this is going to take longer, could you push out an interim release of 
Radiobeacon which will restore the old upload behavior for the moment? I'm 
sitting on a database of 100 or so logs, which I would love to upload, but 
can't... and I'd like to free up some space in my DB again.

Original comment by mich...@vonglasow.com on 20 May 2014 at 9:34

Attachments:

V1_log20140306155751-wifi.xml

GoogleCodeExporter commented 9 years ago

Hey Michael,

thanks for the file. I spent some time debugging the upload and surprisingly
using Google Chromes Advanced Rest Client the file uploads fine, see attached 
screenshot.

So my first suspicion, that the upload was denied due to invalid wifi ssids 
(one ssid contains a dot) was wrong. At the moment, I really don't have a clue, 
what's going on server-side. Unfortunately I don't have the old servers php api 
available, so everything is trial-and-error.

I'll contact Mike to check whether we could integrate the new server stack 
somehow (basically running under a sub-domain at openbmap.org). Right now the 
new server stack covers all basic features, so we might give it a try..

As a quick-fix, I could also offer to upload the exported wifi files via ftp. 
That way, they'll get uploaded for sure.

Original comment by wish7code on 25 May 2014 at 12:02

Attachments:

screenshot.png

GoogleCodeExporter commented 9 years ago

Thanks for the offer, I'll get back to that.

Looking at
https://code.google.com/p/openbmap/source/diff?spec=svnb0f7bc9f0b0aab2bcb812ece3
62f5000dd070ce3&r=b0f7bc9f0b0aab2bcb812ece362f5000dd070ce3&format=side&path=/and
roid/src/org/openbmap/soapclient/FileUploader.java
I see that one of the changes introduced in 0.7.9 was to check for the presence 
of the uploaded file, as the server will send HTTP 200 even in the case of 
failed uploads.

So the failed uploads may have been an issue much earlier, just that nobody 
noticed with the earlier Radiobeacon versions. Right now I'm wondering how many 
of my logs got uploaded at all...

I'll compile a list of all my logs (luckily I've kept all I've ever uploaded) 
and send it to you, so you can check on the server side what got uploaded and 
what did not.

On a side note, it would be nice if I could log in to openbmap.org and see a 
list of the logs I submitted, maybe even with import status. Right now there's 
no reliable way of telling how much of my data made it to the server.

Original comment by mich...@vonglasow.com on 28 May 2014 at 4:03

GoogleCodeExporter commented 9 years ago

Attached are lists of all my (attempted) uploads. I started getting error 
messages around January 1, thus files with a 2014 timestamp will likely not be 
in the database.

Original comment by mich...@vonglasow.com on 2 Jun 2014 at 7:36

Attachments:

GoogleCodeExporter commented 9 years ago

I see that 0.8.3 switched to the new server. I just updated to 0.8.4 (from 
F-Droid, reports itself as "Unknown build") and tried to upload a short log (14 
WiFis, 2 cells). However, after uploading I still get the error message that 
the upload failed.

I can't tell at this point whether this was really in issue with the upload or 
whether it's just the verification that fails (I have a suspicion that the data 
might have made it to the server but verification belives it didn't).

Original comment by mich...@vonglasow.com on 18 Dec 2014 at 11:56

GoogleCodeExporter commented 9 years ago

PS: I just tried to upload three logs of areas that are blank on the map so 
far. If they show up on the map, I will recognize them. If that is the case, we 
will know if it is the upload or the verification that's faulty.

How long does it take for an upload to appear on the map?

Original comment by mich...@vonglasow.com on 18 Dec 2014 at 12:22

GoogleCodeExporter commented 9 years ago

Hey Michael!

Thanks for reporting. I just browsed the code again. 0.8.4 didn't come with 
great changes, especially the upload part remained unchanged compared to 0.8.3

On the server-side I checked the upload folder: there're no files from your 
user account, although we've got a couple of other people using 0.8.4. So for 
some strange reason the files didn't find their way to the server. Could you 
provide a catlog the next time this happens?

> How long does it take for an upload to appear on the map?
Basically there 4 steps following a upload: 
(1) "file import" loads the raw data into the database,
(2) "processing" aggregates the data and calculates a estimated position based 
on raw data
(3) "statistics" calculates user, cell provider and country stats
(4) "maps generation" updates the map tiles

The bottlenecks are steps (1) and (2), as they are quite ressource intensive. 
At the moment I'm experimenting with different import schedules. On approach is 
to run steps (1) and (2) as often as possible, meaning several times an hour, 
to keep the number of files with each import small, the other approach is to 
import only on a daily to save server ressources for other tasks. I still 
haven't found the optimal solution for that.

Original comment by wish7code on 18 Dec 2014 at 9:43

GoogleCodeExporter commented 9 years ago

I just attempted another upload with a packet capture running, and I found that 
my uploads were rejected with "401 Unauthorized".

Are the old openbmap accounts valid on radiocells.org? I've tried to recreate 
an account with the same name, but it told me my user name and e-mail address 
were already in the database. However, I can't log in on the web site with my 
openbmap credentials. I've requested a password reset (and my account name a 
few minutes later) but didn't get an e-mail yet.

Original comment by mich...@vonglasow.com on 22 Dec 2014 at 3:11

GoogleCodeExporter commented 9 years ago

PS: still no mail from radiocells.org. Can you check my account data? My user 
name is stanton, you have the e-mail address.

Original comment by mich...@vonglasow.com on 22 Dec 2014 at 4:57

GoogleCodeExporter commented 9 years ago

Email address for user stanton is fine, I checked.
I'll have a look at the password reset function right now and come back to 
you...

Original comment by wish7code on 22 Dec 2014 at 5:04

GoogleCodeExporter commented 9 years ago

Send you a PM just a couple of seconds ago..

Original comment by wish7code on 22 Dec 2014 at 7:32

GoogleCodeExporter commented 9 years ago

Just a short update: 37 files have arrived (as of yesterday evening)..
See https://radiocells.org/stats/scoreboard

For the records: password reset link also has been fixed. Now works again 
automatically..

Original comment by wish7code on 23 Dec 2014 at 7:37

GoogleCodeExporter commented 9 years ago

I am seeing cells (though not at all zoom levels), which are definitely from 
those uploaded files, but no wifis. Is that a tile rendering/refresh issue? How 
long does it take for updates to be reflected in the tiles (when the tile has 
been rendered before the update)?

Original comment by mich...@vonglasow.com on 25 Dec 2014 at 7:30

GoogleCodeExporter commented 9 years ago

Update: I am seeing wifis in some (though not all) areas. Looks like we're 
getting there...

Original comment by mich...@vonglasow.com on 27 Dec 2014 at 9:47

GoogleCodeExporter commented 9 years ago

I'll rebuild the map and check the 'krank' wifi files also to see wether this 
helps..

Original comment by wish7code on 28 Dec 2014 at 7:07

GoogleCodeExporter commented 9 years ago

There are still some areas in which I don't see any wifis on the map. One is 
around Colmar, France; I collected data there in June 2014. I am seeing cells 
but no wifis. Cells disappear at lower zoom levels, which indicates a tile 
refresh issue, though this is not necessarily the sole cause.

Original comment by mich...@vonglasow.com on 1 Jan 2015 at 10:33

GoogleCodeExporter commented 9 years ago

I see a lot of contributions (wifis created at May 24) in the Colmar region.
The main database is currently being rebuilt from scratch, so stay tuned till 
tomorrow. Then we can analyse in more detail..

Please note: due to the database rebuilt, tiles won't be refreshed today..
Cheers
Toby

Original comment by wish7code on 2 Jan 2015 at 8:11

Attachments:

Colmar.png

GoogleCodeExporter commented 9 years ago

Yes, these are mine. I'm seeing this happen in a few areas: cells but no wifis. 
I'll wait and see...

Original comment by mich...@vonglasow.com on 2 Jan 2015 at 11:13

GoogleCodeExporter commented 9 years ago

I am seeing wifis in areas where they were previously missing – this looks 
like all my uploads up to a few days ago are now being rendered.

However, this only works at zoom level 8 – other zoom levels have cells but 
no wifis. Looks like there's still a tile refresh issue. You might want to look 
at the OSM rendering stack and see if you could reuse some of that logic in 
order to mark tiles as "dirty" when new data is added in an area.

Original comment by mich...@vonglasow.com on 3 Jan 2015 at 2:41

GoogleCodeExporter commented 9 years ago

The server makes use of Hiroshis excellent tileman stack 
(https://github.com/osmfj/tileman), to my knowledge also used for hosting the 
Japanese OSM website. Thus in theory it supports on-the-fly rendering of 
missing tiles. The problem here is, that our server isn't powerful enough to 
render the wifi layer on-the-fly, so it runs into timeouts.

I will modify the rendering stack so it'll (pre-)render a country if there've 
been recent uploads. Let's see wether this helps..

Original comment by wish7code on 3 Jan 2015 at 7:27

GoogleCodeExporter commented 9 years ago

That probably makes sense for lower zoom levels. Putting a cap on it (i.e. no 
more than one complete re-render per 24 hours) probably makes sense.

If you can tweak the rendering stack in more detail, for the higher zoom levels 
it might make sense to pre-render only those tiles that have been requested 
earlier (and more frequently/recently requested ones first).

Original comment by mich...@vonglasow.com on 3 Jan 2015 at 7:32

ghost commented 9 years ago

I too am having issues uploading. I began getting upload errors beginning Aug 3 of this year, and ever since then I have been unable to upload my collected data. I'm using radio beacon version 0.8.8 from F-Droid. Running CyanogenMod 12.1 Android 5.1.1 on Nexus 6.

wish7code commented 9 years ago

Thanks for reporting! Technically there've been no breaking changes server side since Aug 3. Nevertheless we've had some high server loads in the last couple of days. So side-effects on the upload function may have happened, although I wouldn't expect such effects at first glance.

May I ask about your upload method? Did you use 2G, 3G or wifi networks? Which speed? Cheers Toby

TimSC commented 9 years ago

I'm having upload problems today. I'm using wifi.

Interesting project by the way!

wish7code commented 9 years ago

OK, I double checked the uploads, unfortunately the files didn't make it to the server. On the other hand, I just tried to upload from my phone and it seems to work.

Could you share a logcat if the problems persists?

Thanks & cheers Toby

TimSC commented 9 years ago

@wish7code I spotted something in logcat: Error while uploading/storage/sdcard0/Android/data/org.openbmap/filesV1_log20150824103650-wifi.xml: Server reply 401

I'll try update my account info. It might be helpful to have a friendly error message!

Update: upload success!

btw, it was not very easy finding the upload button.

wish7code commented 9 years ago

Glad to hear! Took me a while until I realise, this was actually a 401 password issue:-) Recently I already pushed a change to the repos which display a "Bad user name or password" instead of the generic error message.

Still fixing some other issues and then publish everything on F-Droid...

btw, it was not very easy finding the upload button.

Good point, following your suggestion I opened feature request #90. First idea, add a button in the top menu bar...

Cheers Toby

TimSC commented 9 years ago

Thanks, I look forward to contributing data.

b166er commented 8 years ago

Hello, i can't upload few files for a long time. after i click to upload, i got the message: "Export error Try again later. If problem persists, please file a bug at http://code.google.com/p/openbmap/issues/list"

i tried to upload also with current version 0.8.11 (from f-droid), but nothing helps. what is wrong with this files? and where i can upload this files or can i send them to someone? thanks. screenshot_2015-12-15-12-29-39

wish7code commented 8 years ago

Strange enough: at least some of your uploads have made it to the server (see http://www.radiocells.org/stats/scoreboard)

Could you try the following: go to Settings --> Advanced settings and activate 'Skip uploap' and 'Keep uploaded files'. Then export a session as normal.

Following that you'll find the xml files in the openbmap folder (usually sdcard\Android\data\org.openbmap\files).

Zip them and send them to team@radioc......org and we'll take care of the upload manually.

To debug, could you provide a logcat too?

Thanks & Cheers Toby

b166er commented 8 years ago

for the record: email sent with files and log. Problem solved by dev-team, thank you for the email-reply.

openbmap / radiocells-scanner-android

Export aborts with error #58