search

sobrus / FastLacellsGenerator

Fast lacells.db database generator script for LocalGSMBackend by n76
GNU General Public License v3.0
44 stars 10 forks source link

OCID doesn't work anymore #5

Closed takiainen closed 7 years ago

takiainen commented 7 years ago

OpencellID has a new owner now (since the beginning of april), and it seems FastLacellsGenerator cannot generate database from ocid data anymore. I get the "gzip: stdin: not in gzip format" error. And ocid.csv file in /tmp remains 0 byte, while mozilla.csv file size increases (when the script is running).

After a while the process finishes with "Download error" and you don't get any database.

sobrus commented 7 years ago

Hi!, Yes, it seems that there are big changes at OpenCellID. I've contacted them to provide new download link.

edit: They have replied:

" Hi Sebastian, Thanks for writing in. We recently took over maintainership of OpenCellID and we're currently migrating databases. We've paused certain features until Tuesday, you'll have it back up then. :)

Regards, Sagar Shewarmani

Partnerships & Sales at Unwired Labs ----- Cell + Wifi geolocation | Addresses | Maps | TimeZone API w:www.unwiredlabs.com e: sagar@unwiredlabs.com "

So it seems like we have to wait :)

takiainen commented 7 years ago

Thanks! Meanwhile I commented out all the ocid related lines in the script and managed to output a database file from Mozilla's data.

IzzySoft commented 7 years ago

They didn't say which Tuesday, did they? I just tried, it's still not working. All I've got is a quite basic HTML page telling me I'd have to "log in" first. Maybe they've run into unexpected trouble with their migration. Would have hoped they'd write something on that page we're redirected to, to keep us updated. Would save them from answering some emails…

Unfortunately, I didn't keep the .csv from the last successful run, otherwise I could have at least updated the Mozilla source without losing existing OCID data…

sobrus commented 7 years ago

Yes, it seems like it is still down, and I also didn't make copy of latest data :(

Fortunately, I've managed to add radiocells.org data source, and it seems (at least for Poland area) that their data are almost just as good. For example, area (lac) 11035: MLS - 530 entries MLS + radiocells.org - 919 entries MLS + opencellid - 917 entries (but database generated on 10.04.2017). Unfortunately, they offer only uncompressed combined cell+wifi data files in sqlite format, so they cannot be filtered on-the-fly and are quite slow to download.

Script was largely rewritten to keep it plain and simple while including new features (including Izzy's backup idea, which turned out to be more useful that I thought).

As for opencellid.org I did create an account there, but still no option to download any data.

So, for now, you can try to use radiocells.org instead. Feel free to share some thoughts or fix some bugs ;)

It should be also possible to add option to download several country files from radiocells.org (for example D-A-CH ) and combine them into one CSV in a loop. I just don't know how useful it would be compared to downloading whole world database, so I haven't implemented it yet to reduce script complexity.

IzzySoft commented 7 years ago

I just received a mail from Unwired explaining where I can find the download after having logged in. Checking that place luckily turned up the URL which we can use in the script. Patched it on my end, and successfully created 1) an lacells.db including data from MOZ and OCI plus 2) a PR with the fix :smile_cat:

sobrus commented 7 years ago

Great :) Now we have 3 working sources :) 👍 I've updated script and config sample to reflect changes from old api keys to new tokens