search

eamena-project / eamena-arches-dev

development repository of the EAMENA Arches-powered database platform
https://eamena.org/home
GNU Affero General Public License v3.0
5 stars 3 forks source link

requests.get() on GeoJSON URL not completed #40

Open zoometh opened 10 months ago

zoometh commented 10 months ago

Running request.get() on few GeoJSON urls taks a long time, even do not complet, for example:

GEOJSON_URL = "https://database.eamena.org/api/search/export_results?paging-filter=1&tiles=true&format=geojson&reportlink=false&precision=6&total=90&advanced-search=%5B%7B%22op%22%3A%22and%22%2C%2234cfea78-c2c0-11ea-9026-02e7594ce0a0%22%3A%7B%22op%22%3A%22~%22%2C%22lang%22%3A%22en%22%2C%22val%22%3A%22Sistan%22%7D%2C%2234cfea87-c2c0-11ea-9026-02e7594ce0a0%22%3A%7B%22op%22%3A%22%22%2C%22val%22%3A%22e6e6abc5-3470-45c0-880e-8b29959672d2%22%7D%7D%2C%7B%22op%22%3A%22and%22%2C%2234cfea81-c2c0-11ea-9026-02e7594ce0a0%22%3A%7B%22op%22%3A%22lt%22%2C%22val%22%3A%222021-07-01%22%7D%2C%2234cfea4d-c2c0-11ea-9026-02e7594ce0a0%22%3A%7B%22op%22%3A%22%22%2C%22val%22%3A%22%22%7D%2C%22d2e1ab96-cc05-11ea-a292-02e7594ce0a0%22%3A%7B%22op%22%3A%22%22%2C%22val%22%3A%22%22%7D%2C%2234cfea8a-c2c0-11ea-9026-02e7594ce0a0%22%3A%7B%22op%22%3A%22%22%2C%22val%22%3A%22%22%7D%7D%5D&resource-type-filter=%5B%7B%22graphid%22%3A%2234cfe98e-c2c0-11ea-9026-02e7594ce0a0%22%2C%22name%22%3A%22Heritage%20Place%22%2C%22inverted%22%3Afalse%7D%5D"
resp = requests.get(GEOJSON_URL)
data = resp.json()

It is not linked to the size/number of HP to retrieve (here 90 ..total=90..)

zoometh commented 7 months ago

This issue is related with the Arches/EAMENA API and/or Python requests.get().

  1. There are 89 HP on the API (which is OK),

image

  1. but the GET collect much more (> 20,000)

image

Obviously, a copy/paste of 1 (the API) works

url = "https://raw.githubusercontent.com/eamena-project/eamena-arches-dev/main/projects/sistan/examples/sample_2.json"
resp = requests.get(url)
data = resp.json()
len(data['features'])

image