Fetch data using HOTOSM project list

ramyaragupathy commented 6 years ago

Current flow

Choose a time frame (start time, end time)
Pan and zoom into an area using the map on the right
Hit Fetch Data
Overpass query is constructed to fetch all edits within the map view within a specific time frame

Integrating HOT OSM tasks

[x] Fetch project list through an API request
[x] Load the list in the web page
[x] Choose a project id
[x] Choose a time frame
[x] Hit fetch data
[x] Send a HOTOSM API request to fetch the complete project information: tags, boundary
- [x] Construct an overpass query with the project's polygon
- [x] Send an overpass request

ramyaragupathy commented 6 years ago

Current design with entire active project list populated in the web page:

[x] Fetch project list through an API request

[x] Load the list in the web page

ramyaragupathy commented 6 years ago

[X] Send a HOTOSM API request to fetch the complete project information: tags, boundary 🎉

ramyaragupathy commented 6 years ago

[x] Construct an overpass query with the project's polygon

[x] Send an overpass request

Above two tasks are done, but the overpass request fails. Approach has been to extract smaller polygons of the bigger multipolygon and run a overpass query against each of the polygon. But since polygon url is big, requests are failing.

No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:8000' is therefore not allowed access. The response had HTTP status code 414.

Response:

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>414 Request-URI Too Long</title>
</head><body>
<h1>Request-URI Too Long</h1>
<p>The requested URL's length exceeds the capacity
limit for this server.<br />
</p>
<hr>
<address>Apache/2.4.18 (Ubuntu) Server at lz4.overpass-api.de Port 443</address>
</body></html>

ramyaragupathy commented 6 years ago

Other option was to get the bbox bounds of each polygon and then run overpass query against the rectangular bounds of a polygon instead of the polygon itself. This time there's no 414 error but 429: too many requests failing the query

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
  <meta http-equiv="content-type" content="text/html; charset=utf-8" lang="en"/>
  <title>OSM3S Response</title>
</head>
<body>

<p>The data included in this document is from www.openstreetmap.org.
 The data is made available under ODbL.</p>
<p><strong style="color:#FF0000">Error</strong>: runtime error: open64: 0 
Success /osm3s_v0.7.54_osm_base Dispatcher_Client::request_read_and_idx::rate_limited. 
Please check /api/status for the quota of your IP address. </p>

</body>
</html>

Note: Polygon bounds was obtained using terraformer library


 let polygonTerraform = new terraformer.Primitive(polygon)
 let polygonBbox = polygonTerraform.bbox() //returns [west, south, east, north]

mmd-osm commented 6 years ago

<h1>Request-URI Too Long</h1>
<p>The requested URL's length exceeds the capacity

basically means that you're trying to send a request via HTTP GET which is too long. Switch to HTTP POST instead.

runtime error: open64: 0 
Success /osm3s_v0.7.54_osm_base Dispatcher_Client::request_read_and_idx::rate_limited.

means that you're trying to send too many and / or too expensive requests. Allow for some more wait time between individual requests. Check the status on /api/status on the wait time before trying the next query.

ramyaragupathy / hotosm-viz

Fetch data using HOTOSM project list #7