Error when downloading OSM map tile info

[RUiN-jiarun]

Hi! Thanks a lot for your excellent work. I am trying to run demo.ipynb in the repository, but the following error occurred.

`[2023-06-26 09:56:04 maploc INFO] Getting https://api.openstreetmap.org/api/0.6/map.json...

JSONDecodeError Traceback (most recent call last) Cell In[2], line 28 26 # Query OpenStreetMap for this area 27 from maploc.osm.tiling import TileManager ---> 28 tiler = TileManager.from_bbox(proj, bbox + 10, demo.config.data.pixel_per_meter) 29 canvas = tiler.query(bbox) 31 # Show the inputs to the model: image and raster map

File /data/OrienterNet/maploc/osm/tiling.py:102, in TileManager.from_bbox(cls, projection, bbox, ppm, path, tile_size) 100 assert osm.box.contains(bbox_osm) 101 else: --> 102 osm = OSMData.from_dict(get_osm(bbox_osm, path)) 104 osm.add_xy_to_nodes(projection) 105 map_data = MapData.from_osm(osm)

File /data/OrienterNet/maploc/osm/download.py:35, in get_osm(boundary_box, cache_path, overwrite) 33 with cache_path.open("bw+") as fp: 34 fp.write(content) ---> 35 return json.loads(content_str)

File ~/anaconda3/envs/orienternet/lib/python3.8/json/init.py:357, in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw) 352 del kw['encoding'] 354 if (cls is None and object_hook is None and 355 parse_int is None and parse_float is None and 356 parse_constant is None and object_pairs_hook is None and not kw): --> 357 return _default_decoder.decode(s) 358 if cls is None: 359 cls = JSONDecoder

File ~/anaconda3/envs/orienternet/lib/python3.8/json/decoder.py:337, in JSONDecoder.decode(self, s, _w) 332 def decode(self, s, _w=WHITESPACE.match): 333 """Return the Python representation of s (a str instance 334 containing a JSON document). 335 336 """ --> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end()) 338 end = _w(s, end).end() 339 if end != len(s):

File ~/anaconda3/envs/orienternet/lib/python3.8/json/decoder.py:355, in JSONDecoder.raw_decode(self, s, idx) 353 obj, end = self.scan_once(s, idx) 354 except StopIteration as err: --> 355 raise JSONDecodeError("Expecting value", s, err.value) from None 356 return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)`

This seems to occur when downloading the OSM map's JSON file. I think it was caused by not correctly passing bbox information as a parameter into the query connection.
I checked get_web_data() function in maploc/osm/download.py, the link https://api.openstreetmap.org/api/0.6/map.json shows a message "The parameter bbox is required, and must be of the form min_lon,min_lat,max_lon,max_lat."

I don't know if this is due to my network issue, the parameter issue with urllib's request, or is it a potential bug?

[sarlinpe]

Thank you for reporting this. We should indeed improve the error handling, sorry about that. Does this occur with one of the examples queries or with your own? Can you please print the inputs to from_bbox?

[RUiN-jiarun]

I didn't change any code in demo.ipynb yet, this happened when running with the given example ("assets/query_zurich_1.JPG", "ETH CAB Zurich").

[RUiN-jiarun]

However this works fine on google collab. The error happened when I running the notebook locally. More information:
bbox param given to from_bbox(): -127.99999999988893,-128.0000000000764,128.00000000011107,127.9999999999236 (bottom, left, top, right) in get_osm(): 47.37714313498023 8.54732595638627 47.37962563561885 8.550980743858938

[RUiN-jiarun]

I supposed that this may not pass the bbox to request correctly in some cases (maybe due to urllib3 version or OSM API...)
So I modified this part of the code and it runs well on my machine:

(bottom, left), (top, right) = boundary_box.min_, boundary_box.max_
    # (RUiN) fix retrieving link
    bbox_str = f"{left},{bottom},{right},{top}"
    retrieving_link = "https://api.openstreetmap.org/api/0.6/map.json?bbox=" + bbox_str
    content : bytes = get_web_data(retrieving_link, {'accept': 'application/json'})
    # content: bytes = get_web_data(
    #     "https://api.openstreetmap.org/api/0.6/map.json",
    #     {"bbox": f"{left},{bottom},{right},{top}"},
    # )

[sarlinpe]

Nice catch! Can you instead try the following?

- result = http.request("GET", address, parameters, timeout=10)
+ result = http.request("GET", address, body=parameters, headers={"Content-Type": "application/json"}, timeout=10)

[RUiN-jiarun]

Thanks for your reply!
Unfortunately, this change does not work. It seems that passing a Dict directly through the body param is insufficient to meet the requirement. I decoded the information obtained in this way, which was displayed as "The parameter bbox is required, and must be of the form min_lon,min_lat,max_lon,max_lat.", indicating that this method cannot pass in bbox. Anyway this is a very subtle BUG. Perhaps you could make some minor modifications to the released code when you have suffcient time.
But otherwise, the entire work is fantastic. :)

[sarlinpe]

I will try to fix this once I’m back from vacation :) To make the debugging easier, could you please share your versions of Python and urllib3? Thank you!

[molonepa]

I'm also running into this issue when running demo.ipynb unmodified.

Using Python==3.10.6 and urllib3==2.0.3.

[RUiN-jiarun]

I'm also running into this issue when running demo.ipynb unmodified.

Using Python==3.10.6 and urllib3==2.0.3.

@molonepa You can try this method HERE instead. I was using urllib3==2.0.3 too and this modification works fine for me.

[molonepa]

@RUiN-jiarun I reinstalled the package after making those changes and the same error is occurring:

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
Cell In[2], line 28
     26 # Query OpenStreetMap for this area
     27 from maploc.osm.tiling import TileManager
---> 28 tiler = TileManager.from_bbox(proj, bbox + 10, demo.config.data.pixel_per_meter)
     29 canvas = tiler.query(bbox)
     31 # Show the inputs to the model: image and raster map

File ~/OrienterNet/maploc/osm/tiling.py:102 in TileManager.from_bbox(cls, projection, bbox, ppm, path, tile_size)
    100         assert osm.box.contains(bbox_osm)
    101 else:
--> 102     osm = OSMData.from_dict(get_osm(bbox_osm, path))
    104 osm.add_xy_to_nodes(projection)
    105 map_data = MapData.from_osm(osm)

File ~/OrienterNet/maploc/osm/download.py:40 in get_osm(boundary_box, cache_path, overwrite)
     38     with cache_path.open("bw+") as fp:
     39         fp.write(content)
---> 40 return json.loads(content_str)

File ~/.virtualenvs/maploc/lib/python3.10/json/__init__.py:346 in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    341     s = s.decode(detect_encoding(s), 'surrogatepass')
    343 if (cls is None and object_hook is None and
    344         parse_int is None and parse_float is None and
    345         parse_constant is None and object_pairs_hook is None and not kw):
--> 346     return _default_decoder.decode(s)
    347 if cls is None:
    348     cls = JSONDecoder

File ~/.virtualenvs/maploc/lib/python3.10/json/decoder.py:337 in JSONDecoder.decode(self, s, _w)
    332 def decode(self, s, _w=WHITESPACE.match):
    333     """Return the Python representation of ``s`` (a ``str`` instance
    334     containing a JSON document).
    335 
    336     """
--> 337     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338     end = _w(s, end).end()
    339     if end != len(s):

File ~/.virtualenvs/maploc/lib/python3.10/json/decoder.py:355 in JSONDecoder.raw_decode(self, s, idx)
    353     obj, end = self.scan_once(s, idx)
    354 except StopIteration as err:
--> 355     raise JSONDecodeError("Expecting value", s, err.value) from None
    356 return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[RUiN-jiarun]

@RUiN-jiarun I reinstalled the package after making those changes and the same error is occurring:

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
Cell In[2], line 28
     26 # Query OpenStreetMap for this area
     27 from maploc.osm.tiling import TileManager
---> 28 tiler = TileManager.from_bbox(proj, bbox + 10, demo.config.data.pixel_per_meter)
     29 canvas = tiler.query(bbox)
     31 # Show the inputs to the model: image and raster map

File ~/OrienterNet/maploc/osm/tiling.py:102 in TileManager.from_bbox(cls, projection, bbox, ppm, path, tile_size)
    100         assert osm.box.contains(bbox_osm)
    101 else:
--> 102     osm = OSMData.from_dict(get_osm(bbox_osm, path))
    104 osm.add_xy_to_nodes(projection)
    105 map_data = MapData.from_osm(osm)

File ~/OrienterNet/maploc/osm/download.py:40 in get_osm(boundary_box, cache_path, overwrite)
     38     with cache_path.open("bw+") as fp:
     39         fp.write(content)
---> 40 return json.loads(content_str)

File ~/.virtualenvs/maploc/lib/python3.10/json/__init__.py:346 in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    341     s = s.decode(detect_encoding(s), 'surrogatepass')
    343 if (cls is None and object_hook is None and
    344         parse_int is None and parse_float is None and
    345         parse_constant is None and object_pairs_hook is None and not kw):
--> 346     return _default_decoder.decode(s)
    347 if cls is None:
    348     cls = JSONDecoder

File ~/.virtualenvs/maploc/lib/python3.10/json/decoder.py:337 in JSONDecoder.decode(self, s, _w)
    332 def decode(self, s, _w=WHITESPACE.match):
    333     """Return the Python representation of ``s`` (a ``str`` instance
    334     containing a JSON document).
    335 
    336     """
--> 337     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338     end = _w(s, end).end()
    339     if end != len(s):

File ~/.virtualenvs/maploc/lib/python3.10/json/decoder.py:355 in JSONDecoder.raw_decode(self, s, idx)
    353     obj, end = self.scan_once(s, idx)
    354 except StopIteration as err:
--> 355     raise JSONDecodeError("Expecting value", s, err.value) from None
    356 return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

@molonepa Maybe you could check the content of the requested results or just type the request url into search bar of your web browser to see the result

[molonepa]

Seems like it's failing to retrieve the content (URL is https://api.openstreetmap.org/api/0.6/map.json?bbox=8.54732595638627,47.37714313498023,8.550980743858938,47.37962563561885). Printing content_str gives "The parameter bbox is required, and must be of the form min_lon,min_lat,max_lon,max_lat".

Opening the URL in the browser manually downloads a file map.osm which at a glance seems to contain the right json data.

[RUiN-jiarun]

Seems like it's failing to retrieve the content (URL is https://api.openstreetmap.org/api/0.6/map.json?bbox=8.54732595638627,47.37714313498023,8.550980743858938,47.37962563561885). Printing content_str gives "The parameter bbox is required, and must be of the form min_lon,min_lat,max_lon,max_lat".

Opening the URL in the browser manually downloads a file map.osm which at a glance seems to contain the right json data.

@molonepa OH, have you restarted your notebook after changing the code? If so, you could check the get_web_data() independently:

import json
from maploc.osm.download import get_web_data

content = get_web_data(address = "https://api.openstreetmap.org/api/0.6/map.json?bbox=8.54732595638627,47.37714313498023,8.550980743858938,47.37962563561885",  parameters = {"Accept": "application/json"})

content_str = content.decode("utf-8")
json.loads(content_str)

[molonepa]

@RUiN-jiarun I deleted and re-downloaded the repository and it seems to work now with your original modification to get_osm(). Not sure what was causing my issue but it's resolved.

[yicocc]

I supposed that this may not pass the bbox to request correctly in some cases (maybe due to urllib3 version or OSM API...) So I modified this part of the code and it runs well on my machine:

(bottom, left), (top, right) = boundary_box.min_, boundary_box.max_
    # (RUiN) fix retrieving link
    bbox_str = f"{left},{bottom},{right},{top}"
    retrieving_link = "https://api.openstreetmap.org/api/0.6/map.json?bbox=" + bbox_str
    content : bytes = get_web_data(retrieving_link, {'accept': 'application/json'})
    # content: bytes = get_web_data(
    #     "https://api.openstreetmap.org/api/0.6/map.json",
    #     {"bbox": f"{left},{bottom},{right},{top}"},
    # )

hello ,could I ask you some questions?

[sarlinpe]

https://github.com/facebookresearch/OrienterNet/commit/e3bacb5239c3b093c6a0581a639567e0f69a9383 fixes the issue and improves the reporting of errors. Please let me know if you face any other issue. Thank you.