search

openbmap / radiocells-scanner-android

WLAN and cell tower scanner for Radiocells.org
https://www.radiocells.org
Other
55 stars 26 forks source link

No Cells in table cell_zone in cz.sqlite #201

Open thuryn opened 7 years ago

thuryn commented 7 years ago

I can see Czech republic has 14024 of Cells. When I click on download, the table cell_zone is void. wifi_zone is ok. On android, there is no records in cell_zone table too and therefore location is not recognized without wifis.

wish7code commented 7 years ago

Good catch, thanks for reporting! This was related to our recent problems with our backup server and should be fixed by now!

Cheers Toby

thuryn commented 7 years ago

Well, thank you for response, but cz.sqlite (build from yesterday) still does not contain any row in cell_zone table.

wish7code commented 7 years ago

That's odd. Have you downloaded https://cdn.radiocells.org/cz.sqlite ? I just double-checked that the file: it's 3.5mb in size with 14024 cells.

thuryn commented 7 years ago

OK, my mistake - sql browser does not show me the whole data set. Now I can see all data ... but :-) The table wifi_zone is void. Next - the table cell_zone contain MNC as string value and contains value of "01". But radiocells-nlp-provider has MNC variable as integer. So, phone gets MNC as number (1) and there is not success to select the cell, because 1 != "01". Moreover, I cannot see the cellId I'm on - it's not in database, but I've uploaded my scan a few days ago.

Could I ask you to solve the problems one by one? First, the data type of the MNC column should be solved. Thank you for care :-)

thuryn commented 7 years ago

I've soled the problem with data in db temporarily - I've downloaded the whole DB and remove everything except Czech rep. There are lot of LTE cell station I want to add, I need to add them by scanner. So it would be nice to have country export working and possibility to add scans into DB fast.

wish7code commented 7 years ago

The table wifi_zone is void.

Fixed by now..

I'll take care of the remaining issues in the next couple of days...

thuryn commented 7 years ago

Thanks a lot. I can see data in both tables (y). May I ask one more question - when I upload new data from scanner how long does it take to the database? Is this manual process or automated?

thuryn commented 7 years ago

Just for your information - I've found on my phone (Honor 8, Android 7.1) that location resolving does not work when phone sleeps - OS stops scanning of WiFis. But it is possible to find location by cell id in this case. I've made some changes in radiocells nlp provider to be able find the location even when wifi scans stops. I'll create pull request when it is ready. I'm close to final solution, but still testing because lot os cells are not in db. That's why I'have ask you about database updates from scans.

wish7code commented 7 years ago

Hey @thuryn!

Sorry for my late reply:

Is this manual process or automated? When I upload new data from scanner how long does it take to the database?

I've found on my phone (Honor 8, Android 7.1) that location resolving does not work when phone sleeps - OS stops scanning of WiFis.

I know that 7.1 is more aggressive on power-saving, but haven't had the chance to test on a real device. Did you enable background scanning in the Android settings?

We might experiment with different wifi scanning modes (see https://github.com/openbmap/radiocells-scanner-android/blob/master/android/app/src/main/java/org/openbmap/services/wireless/WirelessLoggerService.java#L432 and https://developer.android.com/reference/android/net/wifi/WifiManager.html#WIFI_MODE_FULL_HIGH_PERF for example). Nevertheless this could lead to significant battery drain..

thuryn commented 7 years ago

Thank you for reply. Regarding Wifi scanning - I'm quite satisfied with powersaving. I want to save battery as much as possible - so I've implemeted logic - start scanning, when scan from wifi is not available in 20s (phone is sleeping), try to find location by cells only - I get it. I havn't played with scanning modes. This could be some option in settings - so you could choose strategy how to scan when phone sleeps - Cells only/More aggressive wifi scanning (which is more precise, but needs more battery).

thuryn commented 7 years ago

I've just downloaded the new cz.sqlite database, but table wifi_zone is void.

wish7code commented 7 years ago

Issue confirmed, I'll have to investigate:

Somehow the correct database (~25 mb) get overwritten with a cell only database (3,5 mb).

thuryn commented 7 years ago

I've just downloaded the new DB and it seems be ok, but ... :-) In the DB there is cell id ID: 33564477 - this ID leads us to location 50.082976029, 14.5199754307. It was strange to me so I've found in some web app, that this cell ID has different location - see: http://www.gsmweb.cz (sorry, it is in czech), but when you put cell id in the form, you can see different location of this cell. Can you imagine, what's wrong?

thuryn commented 7 years ago

I've found what causes the problem described in my previous comment. The problem in TelephonyManager. I've found that methods getNeighboringCellInfo() and getAllCellInfo() returns void lists. I've found that there are some other vendors (Samsung) who does not implement these methods. I've just fired bug on Huawei (my phone is Honor) to implement one of these methods. Event though, I'm looking for other solution how to get cell information. I'm going to try listener to get all cell infos - this could be solution for all phones as a fallback in the case there is no info about cells from getAllCellInfo() or getNeighboringCellInfo() methods. I'll notice you about result.

thuryn commented 7 years ago

The result makes me really sad. I've try to listen on events from TelephonyManager and still no success. It seems, there is no solution for that. So we have to live with this. Maybe Huavei fix this. But what about Samsung.

Anyway, could you, please, briefly describe, how the imported data is processed into database? I would like to check the algorithm, because I'm persuaded that cell_zone contains suspicious data. So I would like to watch history and modification for one specific cell tower with known position. Is it possible to do that?

wish7code commented 7 years ago

[EDIT: IMAGE REMOVED]

Hey @thuryn,

found a interesting pattern here, see the measurement data for cell 33564477 mentioned above. It looks like this cell has measurements from all across the city. It still haven't looked into details, but this might be measurements from a stalled cell phone (reporting outdated cell information).

thuryn commented 7 years ago

That's really interesting, this is my scan from my phone. So I probably infect the whole DB with this faulty scans :-( Do you have the log file sent by my phone? What's inside? If you don't have some file, I'm going to look into a log file on my phone a debug the whole problem. Maybe the problem is related to getAllCellInfo problem. Anyway, could I ask you to remove the map from the previous post? It was great to see the problem, but it would be good to remove it for security reason now.

wish7code commented 7 years ago

Do you have the log file sent by my phone? What's inside?

Have you assigned a separate SWID for your test client at https://github.com/openbmap/radiocells-scanner-android/blob/91ccdff05628840d57b0186dbe7e09d045cb99c8/android/app/src/main/java/org/openbmap/RadioBeacon.java#L159 (set to anything other than RadioBeacon)?

Having a separate SWID eases filtering and removing bad data a lot.

thuryn commented 7 years ago

Well, I've just changed the SWID to "Radiobeacon-thuryn" temporarily.

I've found, I havn't deleted the old scan files and I found that 33564477 is in one file only - the snippet looks as:

file V2_230_log20170728181527-cellular.xml

So it seams, it's correct. Location is correct also.

thuryn commented 7 years ago

I've found one more problem with getCellLocation method today. It's related to nlp part, but it can be the same problem as in scanner. When I look to my phone at work I can see the location of my home. I looked into log (the custom one I'd created) I can see that getCellLocation always returns the same location during my way to work. It seems that when phone runs on battery, the getCellLocation returns the same value :-( :-( I've fired to Huawei the bug with getAllCellInfo and I'm eager to see the fix. I'm going to find some workaround anyway.

thuryn commented 7 years ago

So, I can confirm the root of the last issue (with sucked cellId). Phone always return the same cellId when it sleeps. You can be 30 km away from the original location and telephony manager returns always the same value. I've found some solution (with wakening of the phone) - I'll implement this improvement and when it works fine, I'll implement it in scanner too. It's great that OS saves the battery, but when is switch all thinks off, it's not so clever solution :-(

thuryn commented 6 years ago

I can confirm that the problem of "stucked" cellId is here and is known bug of many androids. I've a some solution with setting display on - it's not so nice, but it works. The questions are:

wish7code commented 6 years ago

I can confirm that the problem of "stucked" cellId is here and is known bug of many androids

If you've saved some links by instance, I'd be highly interested (e.g. which devices, modems, which Android versions, ...)

how to detect this situation automatically in scanner to avoid "bad" data. how to heal the current data sent by scanner till now

We've got some server side filtering in place, i.e. to 'large' cells are discarded. Obviously in your example our filtering criteria was too conservative as the filter wasn't triggered in your case

On the device filtering could rely on https://developer.android.com/reference/android/telephony/CellInfo.html#getTimeStamp() (approximate time of this cell information in nanos since boot)

I would assume that the staled cell still has the initial timestamp, so we could discard if System.currentTimeMillis() - cellInfo.getTimeStamp() > THRESHOLD_VALUE

Other ideas welcome of course..

wish7code commented 6 years ago

Since it's nanos since boot, it's something like

    float age = (float) ((SystemClock.elapsedRealtimeNanos() - cell.getTimeStamp()) / 1000000000);
    if (age > (float) MAX_AGE_SECONDS) {
        Log.w(TAG, String.format("Cell measurement to old: %d s, skipping cell", age);
        return null;
    }
thuryn commented 6 years ago

Unfortunatelly, I have just CellLocation at hand. No CellInfo is available (this is definitely strange on new phone with Android 7, but I've fired 2 bugs to Huawei regarding TelephonyManager). But I have to CellLocation only). I can see a problem defining what "big cell" has a meaning - there could be one cell in 20 km on country side. And when I have no neighbors info it's not easy to check the data.

thuryn commented 6 years ago

The last build of cz.sqlite database has zero rows in cell_info table again. The data in database is crucial - the regular user cannot notice this change immediately, but thing stops to work. Can I help you to improve the build process to: