Closed BillyCroan closed 8 months ago
BillyCroan
commented
9 months ago I see the same 500 int server error no matter which 'Data Service' I try to use. Interesting. Maybe its related to location services if that fires before the weather service?
BillyCroan
commented
9 months ago ah HA! Check this out: https://get.geojs.io/ So geojs.io gave up the ghost. Why not use gelclue2? /usr/libexec/geoclue-2.0/demos/where-am-i It uses nearby wifi networks to nail down position. Better than geoip because if you're on a vpn geoip will get it wrong. Alternately the user should be able to specify the coordinates somehow to bypass geojs.io or perhaps there could be multiple location providers in a dropdown or preference order.
BillyCroan
commented
9 months ago also, i noticed in melange when I switched on 'manual location': nfo t=2024-01-06T05:54:24Z [weather@mockturtl#15]: Error calling URL: Not Found,
Using the URL /search/ and /reverse/ (with slashes) is no longer supported. Please use URLs as given in the documentation.
Examples how to change the URL:
You use: https://nominatim.openstreetmap.org/search/?q=Berlin
Change to: https://nominatim.openstreetmap.org/search?q=Berlin
You use: https://nominatim.openstreetmap.org/search/US/Texas/Huston
Change to: https://nominatim.openstreetmap.org/search?q=Huston, Texas, US
See github issue #3134 for more details.
error t=2024-01-06T05:54:24Z [weather@mockturtl#15]: Retrying in the next 225 seconds...
This sounds like evnen the nominatim api at OSM has changed.
I have had some success with "https://ipinfo.io/$public_ip/json" before.
But I also would like to suggest something I've pondered about. Do we really need to look up location more than one time ever for a given network? i.e. once we know the gps coordinate of my house and my gateway is 192.168.0.1 with mac address aa.bb.dd.ee.dd.ff, we probably don't ever need to look that up again.
That gets trickier with the use of mobile hotspots I suppose, but most networks aren't on roaming hotspots, so it seems silly to query every time just in case my house moved. There is also a standard way to detect that a network is a mobile hotspot. I've seen Android 'know' when the wifi its using is another phone's hotspot vs a stationary wifi network.
To lessen the load on geoip services, I would suggest we identify the discrete network we're on, and count the number of times a geoop service has given back the same answer for that network, and after 5 or 6 we save the 'persistant coordintes' of that network, and don't query for it again.
We could probably also identify the ASN we're on, and classify that ASN as a mobile ASN or a stationary ASN.
Another hiccup i've seen... My IPv6 service comes from Hurriane Electric's TunnelBroker service in my city's downtown. It should be local enough for the same weather. but it ip geolocates 250 miles away! I'm not trying to vpn to another place. but IPv6 tunnelbrokers 'emerge' to the public web in a different location sometimes. So geoclue might be a smarter way to determine location. And consider the possibility that a client's v4 and v6 addresses could geolocate to different locations.
Gr3q
commented
9 months ago Hi,
You should update the applet first because it's out of date.
I'll look into geoclue, but I don't know why https://get.geojs.io/ doesn't work for you
BillyCroan
commented
9 months ago I saw the problem on 3.3.1 as well, but i think its mostly on geojs.io's shoulders, other than it would be nice to have a little explanation of how "saved locations" works. i.e. if there is any saved location, does the applet just use that and no longer tries to geolocate? If so I think the Manual Location slider should cause the saved locations list to grey out when Manual location is turned off.
The way I got it working was:
1) I added a location in 'saved locations' with my lat/lon from google maps, and city, country, timezone filled out.
2) I didn't know what to put for 'Search Entry'. That field could use a tool-tip or note to explain it. So I just put the word "Home"
3) I switched Manual Location on
4) In the 'Location' box, I typed "Home". which is odd, because its tool tip implies its supposed to be a globally unique city name or address, and says nothing about checking against the saved locations list below (Another opportunity to update a tooltip)
re: geojs.io. it consistently 500's from my laptop at home over ipv6, and succeeds over ipv4. my v6 works everywhere else, its tunnelbroker from hurricane electric. From my phone on carrier data, geojs.io succeeds on ipv6. I'll track down the v6 error with geojs.io. Maybe they blocked Hurricane Electric. others are reporting similar problems: https://app.gitter.im/#/room/#jloh_geojs:gitter.im
GeoJS.io no longer maintained: The contact info on geojs.io is bouncing. The response from the remote server was: 550 5.1.1 [contact@geojs.io](mailto:contact@geojs.io): Recipient address rejected: User unknown in virtual mailbox table Also, their own status page is broken.... https://status.geojs.io/ So they might not be around much longer in case you want to start looking for an alternative.
re: GeoClue. Another benefit is that it multiple available methods of geolocation, not just ip geoloation. It will use gps receivers if available, and it will use cell tower data via 3/4g and gprs if available. network gps servers (NMEA from trucking and maritime environments). And if no direct devices provide gps coordinates, it can use the names and mac addresses of nearby wifi access points to figure out the location.
GeoClue is maintained by the gnome folks though it works fine in kde, cinnamon, whatever. And GeoClue has pre-existin tooling to let the user turn on/off 'location services' all at once, or per app which is nice from a privacy respecting perspective.
Gr3q
commented
8 months ago the location entry box is for either searching for a location, or the contents match the "Search Entry" field on any of the Saved location it's going to use that.
When the applet is not breaking you can use the arrows to switch between them (or to your saved location), maybe I need to make sure they are accessible even when the applet is in an error state.
I looked into geoclue (meaning I implemented it). It only provides lat/long - which is enough - but the online solutions give me the timezone, city and country as well. So I will decide on which order I'm going to try it (or geoclue and online at the same time and only fail if I don't get anything)
momodesuka
commented
8 months ago Applet version/Build date Version 3.3.1（2023-10-31 02:58:49） Cinnamon version 6.0.0
I have encountered the same issue. got Service Error when I use VPN. if turn off the VPN, it works well!
BillyCroan
commented
8 months ago the location entry box is for either searching for a location, or the contents match the "Search Entry" field on any of the Saved location it's going to use that.
@Gr3q Can we get text stating that on the config window. even a tool-tip note would help. I don't think I'm an idiot but it was not obvious to me that the "Location" and "Search entry" were connected to each-other.
It might also help if the whole "Saved Locations" section was hidden unless you turn on 'Manual Location'
@endlessbeloved I suppose it comes down to a question of intent. Does the user want to see the weather of where their computer physically is at the moment, or where "their IP is".
I know for me, "where the computer physically is" is my intent. GeoGlue2 accomplishes that. Though it doesn't provide timezone and city. I'm sure there's other resources to connect a GPS coordinate to timezone and city though. Probably even ones that don't require network traffic.
Wait a sec. computers know their own timezone typically. cat /etc/timezone in Linux.
City is trickier and I'm not sure why the City name would be needed. But in many countries, there is a lot of areas that are not IN or part of a city. I live in city X, and a metropolitan region that goes by the name of its largest city, city Y. I tell locals I live in X, and anyone else I live in Y.
I have friends who live outside any city, 50-100 miles away from the nearest metro area, far enough to have different weather for sure. They don't have a city. "Nearest City" would be a better way to solicit city, but know it could be far enough to have different weather.
I would imagine that fetching the forecast is not based on a city name, but on GPS coordinates right? i.e. what dot is closest to my dot, read that forecast. That avoids the messiness of multiple city names Istanbul/Constantinople and dialect/accent/locale differences in how a city's name might be represented in text..
Gr3q
commented
8 months ago @Gr3q Can we get text stating that on the config window. even a tool-tip note would help. I don't think I'm an idiot but it was not obvious to me that the "Location" and "Search entry" were connected to each-other.
I should definitely do that, it is indeed unclear. The locations section is also a stock component so I did not have many options explaining but a text description should help.
It might also help if the whole "Saved Locations" section was hidden unless you turn on 'Manual Location'
You can technically switch to your saved locations by using the arrows on the applet, this is why it's not dynamically hidden. But I guess I think about it because those controls are not in the config.
@endlessbeloved I suppose it comes down to a question of intent. Does the user want to see the weather of where their computer physically is at the moment, or where "their IP is".
On automatic location the user's real coordinates are always preferred, just like when you use your phone. I simply didn't check if GJS has the capability to expose that before. When I have time to figure out to use it reliable and get all the error messages from I will make a release that uses it then falls back to geoIP services.
Wait a sec. computers know their own timezone typically. cat /etc/timezone in Linux.
I know your set timezone. It's not that same as knowing a set of coordinates' timezone, especially because some users can use more than 1 instance of the applet (using the applet not for their own location, but for some other location)
City is trickier and I'm not sure why the City name would be needed. But in many countries, there is a lot of areas that are not IN or part of a city. I live in city X, and a metropolitan region that goes by the name of its largest city, city Y. I tell locals I live in X, and anyone else I live in Y.
I have friends who live outside any city, 50-100 miles away from the nearest metro area, far enough to have different weather for sure. They don't have a city. "Nearest City" would be a better way to solicit city, but know it could be far enough to have different weather.
I'm not debating the usefulness/accuracy of the city/country info at all. Simply it was in the applet from the start so I have to keep it supported. There is a fallback where it displays the lat/long instead of that info so it will not break, just saying.
I would imagine that fetching the forecast is not based on a city name, but on GPS coordinates right? i.e. what dot is closest to my dot, read that forecast. That avoids the messiness of multiple city names Istanbul/Constantinople and dialect/accent/locale differences in how a city's name might be represented in text..
You are right the applet requests weather info purely on GPS coordinates. On auto, I get it from geoip services (currently), on manual, addresses get converted to coordinates via nominatim
BillyCroan
commented
8 months ago I thought worth sharing some gnome code I just came across. There's GeoClue and GeoCode apparently.
GeoClue is what I was blabbing about before and it can get your location by gps coordinate and an accuracy in meters.
GeoCode looks like it can lookup (given a GPS coordinate) what the closest city is to that gps coordinate without calling out to a web service.
https://discourse.gnome.org/t/commandline-tools-for-geocode/15051/2?u=jrj
I guess I am assuming that when possible and reliable, you'd like to reduce dependencies on web services. I know I would. Now if only gnome provided a libweather that predicted the weather without using any web services, that would be a trick!
Gr3q
commented
8 months ago I'm using geoclue as the primary source in the next version, but if you can recommend a better geoip service I can implement that as well.
These are already implemented:
My problem is that I have no way to test (or I just don't know how) which one has the best coverage/reliability.
Technically I could call all of them at the same time or let the user choose between them...but out of the 2 I would prefer the former.
BillyCroan
commented
8 months ago maxmind comes to mind.
But all of these webservices are middlemen, right? When I want to know where an ip is, i run a whois lookup.whois 1.2.3.4 % [whois.apnic.net] % Whois data copyright terms http://www.apnic.net/db/dbcopyright.html
% Information related to '1.2.3.0 - 1.2.3.255'
% Abuse contact for '1.2.3.0 - 1.2.3.255' is 'helpdesk@apnic.net'
inetnum: 1.2.3.0 - 1.2.3.255 netname: Debogon-prefix descr: APNIC Debogon Project descr: APNIC Pty Ltd country: AU org: ORG-RQA1-AP admin-c: AR302-AP tech-c: AR302-AP abuse-c: AA1412-AP status: ASSIGNED PORTABLE mnt-by: APNIC-HM mnt-routes: MAINT-AU-APNIC-GM85-AP mnt-irt: IRT-APNICRANDNET-AU last-modified: 2020-11-25T06:34:44Z source: APNIC
irt: IRT-APNICRANDNET-AU address: PO Box 3646 address: South Brisbane, QLD 4101 address: Australia e-mail: helpdesk@apnic.net abuse-mailbox: helpdesk@apnic.net admin-c: AR302-AP tech-c: AR302-AP auth: # Filtered remarks: helpdesk@apnic.net was validated on 2021-02-09 mnt-by: MAINT-AU-APNIC-GM85-AP last-modified: 2021-03-09T01:10:21Z source: APNIC
organisation: ORG-RQA1-AP org-name: Resource Quality Assurance org-type: LIR country: AU address: 6 Cordelia Street, South Brisbane e-mail: research@apnic.net mnt-ref: APNIC-HM mnt-by: APNIC-HM last-modified: 2023-09-05T02:15:46Z source: APNIC
I'm not saying whois is perfect. it doesn't give gps coords and that's a big problem I think. But when provider A is unreliable, and B isn't great either, looking for C or D might not be as good as circumventing them all.
How do they get their database?
BillyCroan
commented
8 months ago Here's one that I think geoclue itself uses as a fallback. I found it in geoclue's sourcecode: https://geoip.fedoraproject.org/city
As for which one has the best reliability...... I suggest keeping counters in the applet. Also store the date/time of the first attempt to use each service. If geoclue isn't working or available, then every time your applet uses a geolocation service, the applet should increment a counter of how many times it thinks it succeeded, and how many times it failed. And if it failed, it should try another service until it has tried them all, then wait X seconds where X is an exponential backoff.
Give the user a button to send their counters and that datetime to you if they're having trouble. I'd be reluctant to send more data than that for privacy reasons. Perhaps the button launches a mailto: link. and you could include in the message body, those counters, and a prompt to add any other information or their location or isp if they feel relevant. Or the button could just display a text box and say "copy this into an email to me@mydomain.com" or link to a github ticket and ask people to paste their textbox contents there.
I know that won't help with assessing the accuracy of the coordinates, but most of the errors I have experienced were http errors resulting in no coords given.
Ultimately, I'd like for you to not have to worry about which is most reliable. I want to get geoclue/geocode to be responsible and reliable, and weather@mockturtl (as well as the dozens of other location-related apps in linux) can all then lean on one project (geoclue) that reliably and quickly gets the location of the computer.
But in the short term (year or two) I'd say, Any time the applet is showing 'Unavailable' or 'Error', and someone clicks on it, it should go straight to the settings tab where that "report problems" button is waiting for them. They should see the nature of the problem (can't get ip, ca't get latlon, or can't get weather from OWM). That can help you identify what services work better than others. And if you share that information with the geoclue folks, then cay improve that part.
Gr3q
commented
8 months ago I implemented both GeoClue and GeoCode.
maxmind comes to mind.
Maxmind is good, but downloading their full IP table is not feasible, (storing on disk, checking for updates, re-downloading just for this applet is overkill). Also, you need an account which I try to avoid for auto location.
Here's one that I think geoclue itself uses as a fallback. I found it in geoclue's sourcecode: https://geoip.fedoraproject.org/city
That looks good as well. Honestly, I would prefer this over all other geoip services and yeet all others.
If the geoip service that Fedora runs doesn't work for someone then I don't think I could do anything more. I don't think they block IPs, countries or they filter by user-agents either.
Applet version/Build date
Version 3.2.13 (2023-04-28 02:24:56)
Cinnamon version
5.6.8
Distribution
Debian 12
Graphics hardware and driver used
Intel Corporation Alder Lake-UP3 GT2 [Iris Xe Graphics]
Applet name and maintainer
weather@mockturtl @Gr3q
What happened?
Panel shows "Service Error" instead of weather. It wasintermittant at for about a day now completely constant.
I thought maybe my API key was saturated, but my usage should be around 10% of whats allowed, and I logged into OWM and looked around and there was no indicaton that my key was at fault. Same error occurs if I reset settings to defaults (don't use my key).
In melange, isee a error 500 when calling URL. I added the DEBUG file to try to get more out of it. I'd sure like to know the full URL it's calling, so that I could verify the 500 in curl and maybe report it to OWM not here if I knew it wasn't an applet problem
Other information
error t=2024-01-06T05:33:46Z [weather@mockturtl#15]: Retrying in the next 105 seconds... info t=2024-01-06T05:33:48Z [weather@mockturtl#15]: Refreshing in progress, refresh skipped. info t=2024-01-06T05:33:49Z [weather@mockturtl#15]: Error calling URL: Internal Server Error,
500 Internal Server Error
error t=2024-01-06T05:33:49Z [weather@mockturtl#15]: Retrying in the next 105 seconds... info t=2024-01-06T05:33:52Z [weather@mockturtl#15]: Error calling URL: Internal Server Error,
500 Internal Server Error
error t=2024-01-06T05:33:53Z [weather@mockturtl#15]: Retrying in the next 120 seconds... info t=2024-01-06T05:33:53Z [weather@mockturtl#15]: App is currently refreshing, refresh skipped in main loop info t=2024-01-06T05:35:53Z [weather@mockturtl#15]: Error calling URL: Internal Server Error,
500 Internal Server Error
error t=2024-01-06T05:35:53Z [weather@mockturtl#15]: Retrying in the next 135 seconds... info t=2024-01-06T05:35:53Z [weather@mockturtl#15]: App is currently refreshing, refresh skipped in main loop