OSM data not loading in Rapid

eneerhut commented 7 months ago

Description

I have had an issue on both macOS and Windows where OSM data does not load in Rapid. I tried the same area in iD Editor and the data loads straight away.

This is the area I am trying to map, but the same issue happened to me yesterday in New York City, and other occasions in different places. If I'm reading the console log correctly, it looks like Rapid is struggling to get the data from OSM's API.

Screenshots

Version

2.2.4

What browser are you seeing the problem on? What version are you running?

Chrome v120.0

The OS you're using

win

Steps to reproduce

Visit rapideditor.org.
Select 'Get started'
Zoom into an area and attempt to map.

The browser URL at the time you encountered the bug

https://rapideditor.org/edit#background=Bing&datasets=fbRoads,msBuildings&disable_features=boundaries&map=20.00/-38.14955/144.35848

The auto-detected useragent string for your browser (leave blank if you're manually filling this form out)

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36

tsmock commented 7 months ago

That URL is currently working for me. There are a couple of possibilities here:

The OSM API was having issues (this has happened in the past) -- this is probably what was happening, since ERR_CONNECTION_CLOSED is typically done server side.
Your internet service is/was having difficulties routing to the OSM API
A browser addon was causing issues

The first two problems should be transient; the last one would not be.

For the potential ISP issues, if you open the browser console/network activity windows, can you get the API data directly? Have you tried iD? Have you tried JOSM?

For a potential browser addon problem, what browser addons do you have installed? Specifically, do you have anything like the Firefox Facebook Container addon?

Bonkles commented 7 months ago

Agree with Taylor on these points- though i can say I was able to access the OSM API contemporaneously with Ed when he was having difficulties, and it loaded just fine for me. i think that rules out any sort of OSM-wide issue. No other conference goers were experiencing this either.

eneerhut commented 7 months ago

Thanks for the responses. It happened a over a few days so it seems too regular to be an OSM API issue. I thought I remember it happening over both office and home Wifi so that could rule out my ISP, but I will test in the office tomorrow.

It could be extension related, so I will try a few different things including incognito and report back.

jjiglesiasg commented 7 months ago

OSM data is not loading in Rapid as an APP. And also any work done is NOT saved and the App stays in a loop "uploading" but such uploading Never happens and all the work is trashed. ANNOYING!!!

jjiglesiasg commented 7 months ago

No other portion of the road, no rivers, just that uploaded

jjiglesiasg commented 7 months ago

ID is not doing any better

eneerhut commented 7 months ago

New day, new network, no issues. I'll report back if I see any issues with OSM data loading.

bhousel commented 7 months ago

Thanks, it looks like have been some actual outages on the OSM service over the past few days.

I think Rapid should do a better job handling these issues.

From @eneerhut's screenshot, he's clearly not getting OSM data, but also not seeing any of the error messages. We do have some code to handle when the API is unavailable, but it doesn't cover this situation with the connection closed. We should detect it and let the user know.

I think for all these kinds of failures, maybe the right course of action is to direct our users towards reporting the issue on openstreetmap/operations, since it's more for them to deal with.

jjiglesiasg commented 7 months ago

Keep not loading anything in Rapid, and just portions in ID after awaiting a lot..

tsmock commented 7 months ago

@jjiglesiasg : There was a persistent API scraper that was taking the OSM API offline for everyone else.

The server admins did create a fix for it, such that ignoring 408 request timeout errors would be a really bad idea. Like "blocked for 12 hours" bad idea. Right now, it looks like it is restricted to one API endpoint (which I don't think iD or Rapid use), but if it is a malicious actor trying to bring OSM down, I would not be surprised if those rules are extended to the rest of the OSM API.

In the future, before blaming Rapid, iD, or JOSM, please check to see if is something that is happening for everyone else (you can check openstreetmap/operations and the prometheus dashboard, ask on #osm IRC, OSM World Discord, etc.).

In this case, the OSM API was down between 03:40 and 07:00 UTC.

Should we handle issues when uploading better? Yes. We should never lose data. But we need to know what is happening. What were the network responses? Did the OSM server respond with a "good" response and the data never actually made it to the DB? Did the OSM server respond with an undocumented response? We can guess, but that means that any fix will be a shot in the dark, and could be fixing the wrong problem.

jjiglesiasg commented 7 months ago

Thanks for the info, but I do not remember place the blame in ID, RAPID or JOSM for the situation. I just pointed that was happening in ID and RAPID who are the apps that I use. And yes, was annoying and I lose data because if you closed any of the apps without saving, the data is/was scrapped, not keep for future upload.

eneerhut commented 7 months ago

Noticed this one again. This time using MapRoulette to cycle through San Francisco crosswalks. I am able to zoom into the same area in a new Rapid Editor/iD window and see all the data.

Could it be possible that cycling through such crosswalk challenges in MapRoulette causes OSM rate limiting?

bhousel commented 6 months ago

Quick update on this, I've done a bunch of work over the past week on improving how we detect issues with the OSM API and surface them to users. It wasn't just one thing, but a bunch of weaknesses in the code that had accumulated over time.

Now we're properly detecting when the user is under a rate limit, and we show users how much time is remaining with a count down.

Here's a summary I sent our team earlier:

osm.loadFromAPI was detecting the 509 rate limit error, but only doing anything about it if the user was logged out: it would emit an event so that status.js should show the red bar and tell the user to log in
if the user was logged in already, it did nothing
if there was a rate limit, we weren’t keeping track of it, and just firing off more API calls anyway
there were situations where these calls would just stay in inflight forever, so once the rate limit lifted, we woudn’t retry the fetch, leaving just blank spots on the map
status.js was listening to the apichange event wrong - so it never showed the red bar (#1304)
status.js was showing the red bar when backups couldn’t be saved (local storage error), but not clearing on reset (#1120)

facebook / Rapid