Open mar-v-in opened 7 months ago
We had also AppleWifiNlpBackend, could you please merge it in microG?
Interesting... is there a natural successor to it?
Apple does not allow us to access their location service. Also, in contrast to most Google APIs we're using, usage is not strictly required for interoperability, so we don't have a legal basis for using their API in most jurisdictions.
Unfortunately there is no natural successor I'm aware of. However there are also other open source projects (e.g. geoclue, the location service of Linux desktop) affected, so I'm hoping for some coordinated effort to bring a similar open source service to life.
Can we just continue to run MLS in some jurisdiction that isn't concerned about patents?
This would sort of kill microG for me, can we just use Apple's NLP? It was always more accurate anyway?
As I mentioned in the /e/ OS Community I hope something better than MLS is used because ever since microG went to 0.3.0 every app but maps (Magic Earth) is not accurate at all and is off by 50-100 miles. I agree with using Apple's NLP. Unless a good accurate open source NLP is found.
Even if we have to put Apple's NLP behind a radio button with a big disclaimer "Apple doesn't want you to do this, you're on your own"
-------- Original Message -------- On 14 Mar 2024, 12:53 pm, ToughDBlue wrote:
As I mentioned in the /e/ OS Community I hope something better than MLS is used because ever since microG went to 0.3.0 every app but maps (Magic Earth) is not accurate at all and is off by 50-100 miles. I agree with using Apple's NLP. Unless a good accurate open source NLP is found.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>
mentioned in the /e/ OS Community I hope something better than MLS is used because ever since microG went to 0.3.0 every app but maps (Magic Earth) is not accurate at all and is off by 50-100 miles. I agree with using Apple's NLP. Unless a good accurate open source NLP is found.
It is all about the data. The more users contribute data to the service, the more precise the results become.
mentioned in the /e/ OS Community I hope something better than MLS is used because ever since microG went to 0.3.0 every app but maps (Magic Earth) is not accurate at all and is off by 50-100 miles. I agree with using Apple's NLP. Unless a good accurate open source NLP is found.
It is all about the data. The more users contribute data to the service, the more precise the results become.
I understand but when I had microG 0.2.27 it all worked great. How about going back to that NLP method?
As I wrote above:
Apple does not allow us to access their location service. Also, in contrast to most Google APIs we're using, usage is not strictly required for interoperability, so we don't have a legal basis for using their API in most jurisdictions.
As a last resort option we could of course have that included and put a big warning that using it is a breach in ToS and Apple can sue your for financial compensation if you used that feature when trying to enable it, but if we can figure out somehting else I would definitely prefer that.
If you have problems with location in current versions of microG I suggest to enable local location learning in settings and then use an app that requests both GPS and network based location. As soon as you get a GPS fix, it would then remember the correct location of nearby wifi networks and use it instead of MLS, so MLS' data quality doesn't matter anymore.
As I wrote above:
Apple does not allow us to access their location service. Also, in contrast to most Google APIs we're using, usage is not strictly required for interoperability, so we don't have a legal basis for using their API in most jurisdictions.
As a last resort option we could of course have that included and put a big warning that using it is a breach in ToS and Apple can sue your for financial compensation if you used that feature when trying to enable it, but if we can figure out somehting else I would definitely prefer that.
If you have problems with location in current versions of microG I suggest to enable local location learning in settings and then use an app that requests both GPS and network based location. As soon as you get a GPS fix, it would then remember the correct location of nearby wifi networks and use it instead of MLS, so MLS' data quality doesn't matter anymore.
Understandable, don't want to breach any TOS. Didn't know that. I'd rather something else be figured out as well. I don't see where to enable local location learning in the microG settings. Can you recommend apps that request both GPS and network based location?
SatStat would do the trick. Also apps that use Google Maps API of microG often do request both locations through microG's API.
SatStat would do the trick. Also apps that use Google Maps API of microG often do request both locations through microG's API.
That works! Although my weather app says "Current Location" and not the actual city and state. The map in the weather is correct but again just says "Current Location" and not city and state. I have Nominatim enabled. Is there a way to reset Nominatim?
I feel that the loss of UnifiedNLP together with this makes for an unacceptable step back. @mar-v-in I know there were technical reasons behind dropping UnifiedNLP but consider that it was working fine or well enough not to notice the issues for many, including on newer versions of Android, and that there was a small but relatively healthy "ecosystem" of backends and not just Apple, for instance I was an avid user of offline backends, and the GSM Location backend wasn't limited to "learning" but could download cell databases for local use. That was amazing and given I use mostly open source applications, I do lose a lot of the usefulness of microG when network location goes away.
but could download cell databases for local use.
This was intended to be a feature in the new implementation as well (+ importing wifi and cell databases from file). With the learning code, there already is the logic to handle location data once it's in the database, we "just" lack the downloading/importing.
working fine or well enough not to notice the issues for many
I think there might be a bias here. As you said, you've been mostly working with offline backends, which are faster in processing and less likely to run into timeouts or similar issues than online backends.
I am convinced that the single process approach and centralized code will in the long run improve the results, but I do recognize that short term there are downsides as long as we lack some important features (with download/import of data being one of the major lacking features).
The way that I see it, one of the primary purposes for which people use their smartphones is navigation. Personally, I'd say navigating is a more common use case for a smartphone than voice calls.
If they don't have a fast, accurate location provider (ie if all they have is GPS), they can't navigate with their phone. The lock-on time is too high and it's not usable effectively indoors or in dense cities.
If people can't navigate with their phone on MicroG, they won't use MicroG on their phone. Losing the ability to provide navigation - via whatever means - isn't just a feature regression, it's an existential threat to MicroG usage.
It's important to do things the right way, but it's also important to make sure the project has a healthy enough base of users that it survives.
Does anyone know how accurate OpenCellID and other downloadable databases actually are? Maybe if their TOS allows we could add a menu to download their data (maybe on a country by country basis).
we "just" lack the downloading/importing.
@mar-v-in
Downloading can be worked around for some (I e.g. do that on my Linux machine, using FastLacellsGenerator, which includes MLS and OpenCellID, and then push the database to all my devices – also saves me from downloading and generating it multiple times). Easiest way would probably be supporting that SQLite database directly. Not sure if the corresponding Backend could be used stand-alone to download and generate the database on-device for those who need it – but having it supported would be a good first step IMHO.
Btw: I saw that TowerCollector now supports custom MLS URLs for upload, so there might be some "aggregators" for that, too. Not sure if they could be accessed the way microG currently accesses MLS, though. But an offline database is what some of us would definitely like to have anyway :stuck_out_tongue_winking_eye:
If people can't navigate with their phone on MicroG, they won't use MicroG on their phone.
Yeah I've been using microG now for a while, however having to mess about whenever attempting to order an Uber is becoming too much.
Sandboxed GMS is looking a lot better in terms of quality of life.
I really don't want to have to deal with downloading local towers etc, I just want a decent enough NLP provider (Apple or Google)
I vote for return UnifiedNLP.
Or someone self-hosted database server.
I vote for return UnifiedNLP.
This doesn't really solve the privacy issues; this would push people to Google and Apple for Location Services, weakening the privacy/FOSS projects.
I think there are some values in trying to create new tech for Location Services; whether it's open or closed.
One thing I suggested is hashing access point details to use as IDs instead of raw data. (for example, hash(mac_address, SSID, ...)
)
That way you can create a public database but have it be relatively private; someone revealing their SSID won't have the same effect on their location privacy as it usually has; of course there's other public databases like Wigle that can make this an issue, but they seem very bad and outdated that accuracy is... eh. As you would only really know the mac and SSID if you know where the network is anyway; of course you wouldn't hash individual properties as that doesn't really give you many benefits I think Signal's [username implementation]() would be a good read for anyone implementing something like it. This would automatically deal with some issues like when someone changes SSIDs and location but not the router, etc.
The issue with such an approach rather than just publishing raw data, is that protocol ossification can be an issue. One solution to this is allow users to backup their contribution data and never delete it; so in the case of an upgrade to the network, one can re-upload new data.
I think something like this can also have value in a closed ecosystem, which would have better privacy, as in a map it is still very possible to do some analysis of contributed points and dates and things; I personally think that a closed system that uses a private protocol is better than a completely open, or than closed but not private to the operator; with a closed system with a private protocol, we could potentially move the data in a private way to other projects; and data analysis is harder.
TowerCollector is an interesting project, the maintainer is busy ATM and it's not his focus, so a fork to include WiFi, Bluetooth data, etc and be compatible with such a protocol would be needed.
This would be a lot of work and research; so it's very likely something like this isn't actually developed; it's still interesting to debate this, especially what improvements can be made to location services overall :P
Or someone self-hosted database server.
/e/foundation seems to be offering to do that (thankfully!)
This doesn't really solve the privacy issues; this would push people to Google and Apple for Location Services, weakening the privacy/FOSS projects.
It would push me to using the local databases I was using before UnifiedNLP was retired. Can't speak for others of course.
I also wonder: how many independent developers are probably going to want to be contributing to this repository, with PRs that must rightly not break things and be approved, as opposed to tinkering by writing their own module in their own repository, maybe using a simple existing module as a template, and publishing it there? I think that was a powerful aspect of the approach.
FYI, my plan for the next version is to add:
Import and Export of the local database
Will that include the cell database as used by LocalGSM – and maybe auto-detect if a fresh copy has been placed in a defined location (as LocalGSM does)? Idea is to easily distribute an existing database to multiple devices for offline use (in addition to or instead of online checks like Ichnaea).
Or someone self-hosted database server.
/e/foundation seems to be offering to do that (thankfully!)
And @rtsisyk offered the same on Mozilla's thread on this.
There doesn't seem to be any shortage of people/organizations willing to do the hosting.
FYI, all of us and a bunch of others are in communication with each other. We're aiming to ensure there is a single successor of MLS endorsed and supported by all the libre software projects involved, including microG
Relatedly, one issue I've had myself and seen many reports of with the internal UNLP implementation in recent GmsCore versions beyond the Mozilla db staleness:
Just enabling any of the Mozilla UNLP sources at all seems to "pollute" GPS results to the point that even with a good GPS lock geolocation is flaky or completely anomalous.
You can see this with SatStat on the map page where the blue UNLP "circle" and/or pin is constantly turning grey ("stale data") and/or flickering on/off constantly even when the device is stationary. Seems to me that a good GPS lock should override UNLP inputs especially if the UNLP inputs are extremely unstable.
I can open a separate issue on this if necessary. Just seems that this should be fixed along with enabling new MLS hosts, or else I think it will undermine the utility of those as well.
The network location provider is (and should be) independent of GPS and having a GPS fix and the results from GPS are not considered in it. The blue circle in SatStat displays the network location provider, the red circle shows the GPS location.
There is also the fused location, merged from both. This should use GPS once GPS is available and network location if not.
The flickering of the blue location is probably caused by the update interval. MicroG only provides an update if updated data exists, which due to rate limiting might be less often then SatStat's requested interval.
Perhaps my analysis of the problem was not correct.
But the observation that any enabling of the Mozilla UNLP sources often results in completely anomalous geolocation results is consistently demonstrable. I have talked to various people from all around the world that were getting results like placing them in the middle of the Atlantic ocean or something, until they disabled the Mozilla UNLP data sources in the microG utility.
After disabling the Mozilla switches, they got good geolocation results.
I actually just saw the Fused location provider in SatStat for the first time after acquiring a Pixel7Pro I installed CalyxOS on.
Prior to that, I had never seen that data source displayed in SatStat on my older devices at all. (Some of which are running older versions of LOS, some CarbonROM. All using Qualcomm SoCs.)
The fused provider is not available on all devices, that's why there is also a fused location API in Play Services that is also provided by microG; that's why some apps need microG to display locations, they use the Play Services fused location provider.
If erroneous network location data claims to be more accurate then GPS, it might be preferred over GPS data. I've never seen this as a relevant issue in practice, but it could explain the behavior you described. Indeed this could be improved in microG's fused provider by making low accuracy GPS still override high accuracy network location if the network location is off by too far from the GPS.
If it's helpful I could try to go back to the people who observed that behaviour and ask them to re-enable the Mozilla switches and collect logs or something.
I will try it on my devices again and check.
(FWIW: I disabled/uninstalled the Mozilla UNLP backend ~2 years ago due to the data staleness in a city that might be the most extensively geomapped in the world: San Francisco, California)
After disabling the Mozilla switches, they got good geolocation results.
I just tested this and can confirm it doesn't place me way out of where I actually am anymore.
Can someone ELI5 what the "Remember GPS Location" option does?
Can someone ELI5 what the "Remember GPS Location" option does?
When an app requests both GPS and network location (like e.g. SatStat and apps with a map using Google Fused location API) and this option is enabled, microG will store a record of the GPS location and the information used for network location at that point in time in a local database. If later, for the same network information, a location is requested, the local database is used instead of requesting from Mozilla.
There are a bunch of restrictions, like GPS locations being ignored if their accuracy is too low and the local database entries first having to accumulate some certainty (by having multiple occurrences of the same network), so its effect might not be visible immediately.
The feature is disabled by default as the contents of the database can be used to recover a user's partial location history (if they use GPS).
After disabling the Mozilla switches, they got good geolocation results.
I just tested this and can confirm it doesn't place me way out of where I actually am anymore.
Can someone ELI5 what the "Remember GPS Location" option does?
In order to get accurate geolocation results I've had to disable Mozilla services since microG went to versions higher than 0.2.27. Even enabling "Remember GPS location" puts me far away from where I'm really at.
microG gained the ability to customize MLS URL recently commit, but this nightly build isn't updated til now.
microG gained the ability to customize MLS URL recently commit, but this nightly build isn't updated til now.
Now you just need another MLS clone service. Afaik that doesn't exist yet.
I know @mar-v-in is involved in that planning by the FOSS community as stated above, as a stopgap before building something better from the ground up. (The last significant public discussion of that I know of went down in flames, 'nuff said about that for now.)
But even if/when such a clone MLS service goes online, and even if it bootstraps its database by importing the existing MLS database, that data is so stale as to be (IMHO) pretty close to useless in most regions of the world. The microG users I support are starting to lose patience after standalone UNLP backends were deprecated in May 2023.
Coarse location (eg for country-level S/W license checks etc) can usually be done trivially via GeoIP. IMHO what we need a UNLP service for is for more precise location when GPS is not available or feasible, and to "jumpstart" a relatively high-accuracy location while waiting for a GPS fix. Cell towers are typically too far apart to be useful for that, the MLS WiFi/BT data was never public and the whole field is polluted by a gaggle of patent trolls dying to sue anyone that tries to field any sort of RF-emitter-based geolocation service. (Skyhook [now part of Qualcomm], Geoscope, Enovsys, etc)
Building a new online db of RF emitters from scratch will take years to get decent global coverage. There needs to be a robust new set of incoming data inputs to make it feasible, and there's no way that the small microG userbase can make a useful dent in that, it will have to come from a LOT more endpoints.
microG gained the ability to customize MLS URL recently commit, but this nightly build isn't updated til now.
Now you just need another MLS clone service. Afaik that doesn't exist yet.
I know @mar-v-in is involved in that planning by the FOSS community as stated above, as a stopgap before building something better from the ground up. (The last significant public discussion of that I know of went down in flames, 'nuff said about that for now.)
But even if/when such a clone MLS service goes online, and even if it bootstraps its database by importing the existing MLS database, that data is so stale as to be (IMHO) pretty close to useless in most regions of the world. The microG users I support are starting to lose patience after standalone UNLP backends were deprecated in May 2023.
Coarse location (eg for country-level S/W license checks etc) can usually be done trivially via GeoIP. IMHO what we need a UNLP service for is for more precise location when GPS is not available or feasible, and to "jumpstart" a relatively high-accuracy location while waiting for a GPS fix. Cell towers are typically too far apart to be useful for that, the MLS WiFi/BT data was never public and the whole field is polluted by a gaggle of patent trolls dying to sue anyone that tries to field any sort of RF-emitter-based geolocation service. (Skyhook [now part of Qualcomm], Geoscope, Enovsys, etc)
Building a new online db of RF emitters from scratch will take years to get decent global coverage. There needs to be a robust new set of incoming data inputs to make it feasible, and there's no way that the small microG userbase can make a useful dent in that, it will have to come from a LOT more endpoints.
I only noticed everything in passing and would like to add something to your statement. In addition to a fresh database, you also need users who fill the database with content. In my opinion, WLAN APs should definitely be included alongside mobile radio masts. But I don't think it's that easy to have a large user base so that the database is filled with up-to-date content. The application would also have to be able to run on many devices in order to collect the data.
I hope the FOSS community is already working on it :)
Unfortunately I am too stupid to program, but I have a lot of funny ideas in stock.
@Sapiosenses , may be use downloadable OpenCellID database for self-hosted backend. For now I use proprietary (push server proprietary anyway) Google endpoint https://www.googleapis.com/geolocation
as custom service URL and all work fine.
Respectfully if I may ask, if there is a longer term plan to directly support importing of cellular / wifi information created offline, can I suggest using this as a starting point ? : https://gitlab.com/deveee/Local-GSM-Backend
What makes this microG location provider a bit different is that rather than injesting a CSV it supports importing a sqlite database that can be created offline (via the lacells script) or directly on device.
I recall using the 'import CSV' functionality microG had circa Android 4.4 and importing large cell tower lists took a very long time.
@mar-v-in can OpenCelliD be used?
curl --request POST --url https://unwiredlabs.com/v2/process --data '{"token": "<Your_API_Access_Token>","radio": "LTE","mcc": <MCC>,"mnc": <MNC>,"cells": [{"cid": <CID>}],"address": 0}'
"balance":100 request in 24 hours?
Any update on what the plan is for this?
@Sapiosenses , may be use downloadable OpenCellID database for self-hosted backend. For now I use proprietary (push server proprietary anyway) Google endpoint
https://www.googleapis.com/geolocation
as custom service URL and all work fine.
I'm not sure what the relationship of that Google URL is to the OpenCellID database.
Certainly most microG users would not be thrilled about the idea of querying Google geolocation databases, giving them yet another way to track us.
Re: OpenCellID, they have their own database validity problems. Here's an example posted on their community forum in late March: https://community.opencellid.org/t/database-cleanup-germany/1262
In fact, someone else brought up this issue with them back in August, and apparently nothing has been done about it yet: https://community.opencellid.org/t/usefulness-of-opencellid-3g-data-in-germany/1147
I think the big picture here is that this is a major undertaking, which not only requires a large amount of continuous data input and revalidation to stay useful (and it's not clear where this will come from), but there are also various lawsuit-happy corporations that are trying to sue anyone that compiles such databases and makes them available to the public. (Which was probably the biggest reason why the Mozilla MLS was ultimately shutdown: Skyhook Corp legally threatened them over it.)
Unfortunately the original Mozilla thread on the matter where some details were being discussed was locked when someone started trolling and namecalling. I get the impression that discussion is happening in a more private venue now.
Adding the (optional) functionality of the previous GSM Location module and of the Tower Collector app to microG would help to store only the needed OpenCellID information on the phone and to supply new information to the database.
Adding the (optional) functionality of the previous GSM Location module and of the Tower Collector app to microG would help to store only the needed OpenCellID information on the phone and to supply new information to the database.
That wouldn't mean adding the module's functionality, but restoring it. It was removed a few microg releases back (0.2.28). I.e. there's no point in suggest it as something new here.
Adding the (optional) functionality of the previous GSM Location module and of the Tower Collector app to microG would help to store only the needed OpenCellID information on the phone and to supply new information to the database.
That wouldn't mean adding the module's functionality, but restoring it. It was removed a few microg releases back (0.2.28). I.e. there's no point in suggest it as something new here.
Your tone sounds a bit strange to me. Are you a moderator of this thread?
My comment is a reaction to the comment above mine. Support of modules has been removed from microG. So both GSM Location and Dejavu modules cannot be used any more. But Dejavu functionality has been added to microG since. I hope that using an imported database can be added to microG also. Tower Collector supports feeding the OpenCellID database. AFAIK this was not possible with microG in the past. With adding this functionality to microG it could contribute to update that database.
Tower Collector supports feeding the OpenCellID database. AFAIK this was not possible with microG in the past. With adding this functionality to microG it could contribute to update that database.
It would greatly benefit the database if local emitters could be automatically uploaded without requiring an external app yeah! Although unfortunately @Sapiosenses has a point (even if it's not easy to quantify):
There needs to be a robust new set of incoming data inputs to make it feasible, and there's no way that the small microG userbase can make a useful dent in that, it will have to come from a LOT more endpoints.
Seeing the state of Mozilla database, I hope mar-v-in & friends have some special plans for its successor 🤞
There needs to be a robust new set of incoming data inputs to make it feasible, and there's no way that the small microG userbase can make a useful dent in that, it will have to come from a LOT more endpoints.
Looking at the Stats of OpneCellID, there's quite a huge dataset available there. Contributors have multiple ways to add to it, and it has been around for quite a long time (I'm using it for years with microG, and actively contribute to it for years with TowerCollector). I could not find numbers on the contributors/contributions on their site¹, but I'd say it'd be a solid base until replacements for MLS are firmly established.
The database can be generated off-device (FastLacellsGenerator was already mentioned) or on-device with a helper like the former LocalGSM module (which also was already mentioned here) for those who'd prefer that. It will take quite a while until a MLS replacement with comparable coverage is established, so it would be good to have this (with the added benefit that it does not require a permanent network connection).
¹ Wikipedia states 49k registered contributors (anonymous uploaders have to be added to that) shipping more than 1 million measurements every day on average. And with microG chiming in, that might not make a big "dent" but would certainly further improve those numbers.
Adding the (optional) functionality of the previous GSM Location module and of the Tower Collector app to microG would help to store only the needed OpenCellID information on the phone and to supply new information to the database.
Those things are in the works as we speak.
A way to submit RF emitters from microG, and a way to import static tower databases.
It's hard to say when those features will appear however. Right now there is a lot of work going on to try to build a replacement for the Mozilla Location Service, which is going down for good next month.
Adding the (optional) functionality of the previous GSM Location module and of the Tower Collector app to microG would help to store only the needed OpenCellID information on the phone and to supply new information to the database.
That wouldn't mean adding the module's functionality, but restoring it. It was removed a few microg releases back (0.2.28). I.e. there's no point in suggest it as something new here.
it never existed in GmsCore, it was using external UNLP backends to provide the data to a UNLP API in the GmsCore module.
When microG changed the architecture to address issues with aggressive power-management killing the standalone backend processes and interrupting their function, they didn't import all of that stuff into the core module, just a little bit.
Adding some of that functionality is currently being worked on. (In fact it's the reason why GmsCore releases >0.2.27 are flagged as "preview" releases: because of the regression in the UNLP functionality. They will revert to "stable" when the developer feels that the UNLP functionality is working adequately again)
There needs to be a robust new set of incoming data inputs to make it feasible, and there's no way that the small microG userbase can make a useful dent in that, it will have to come from a LOT more endpoints.
Looking at the Stats of OpneCellID, there's quite a huge dataset available there. Contributors have multiple ways to add to it, and it has been around for quite a long time (I'm using it for years with microG, and actively contribute to it for years with TowerCollector). I could not find numbers on the contributors/contributions on their site¹, but I'd say it'd be a solid base until replacements for MLS are firmly established.
When I took a quick look at the OpenCellID stats and their support forum about a week ago, seems like their db is not exactly free of the kinds of problems that MLS had the last few years either.
For example someone in Germany was complaining back in November or December that since 3G was shut down entirely in the country recently, there are now tens of thousands of tower records for decommissioned towers in Germany still in their db.
And then another person in Germany just last month complained on their fora about exactly the same thing.
So they clearly have a data maintenance issues there as well.
In a nutshell, I don't think it's an easy matter to replicate MLS. But I'm sure Marvin is working with some competent people who will come up with a solution. I just don't know how useful it's going to be in the short term. (Especially if they are just cell towers. Those are not very accurate geolocation sources. Mozilla when it was in its prime was quite good, largely because of the WiFi/BT data which is far more precise since the signals are much shorter range. But no one will be getting that data as it was never made public and it's all out of date now anyway.)
As was announced at https://github.com/mozilla/ichnaea/issues/2065, Mozilla will retire MLS soon. The final deadline for third parties seems to be set to June 12.