Request Size Limit on local vector upload

[svHatch]

Environment Information

• geemap version: 0.13.1
• Python version: 3.9.12 (main, Apr 4 2022, 05:22:27) [MSC v.1916 64 bit (AMD64)]
• Operating System: Windows 10 Enterprise

Description

Trying to convert a local shapefile (5 Mb) into a GEE Geometry.

I got an error: "EEException: Request payload size exceeds the limit: 10485760 bytes." Which equates to 9.5367 Mb.

What I Did

ceez_shp = './local_data_path/marine_regions_ceez_simplified.shp'
ceez = geemap.shp_to_ee(ceez_shp)
Map.addLayer(ceez, {}, 'CEEZ')

---------------------------------------------------------------------------
HttpError                                 Traceback (most recent call last)
File ~\Anaconda3\envs\gee\lib\site-packages\ee\data.py:334, in _execute_cloud_call(call, num_retries)
    333 try:
--> 334   return call.execute(num_retries=num_retries)
    335 except googleapiclient.errors.HttpError as e:

File ~\Anaconda3\envs\gee\lib\site-packages\googleapiclient\_helpers.py:134, in positional.<locals>.positional_decorator.<locals>.positional_wrapper(*args, **kwargs)
    133         logger.warning(message)
--> 134 return wrapped(*args, **kwargs)

File ~\Anaconda3\envs\gee\lib\site-packages\googleapiclient\http.py:915, in HttpRequest.execute(self, http, num_retries)
    914 if resp.status >= 300:
--> 915     raise HttpError(resp, content, uri=self.uri)
    916 return self.postproc(resp, content)

HttpError: <HttpError 400 when requesting https://earthengine.googleapis.com/v1alpha/projects/earthengine-legacy/maps?fields=name&alt=json returned "Request payload size exceeds the limit: 10485760 bytes.". Details: "Request payload size exceeds the limit: 10485760 bytes.">

During handling of the above exception, another exception occurred:

EEException                               Traceback (most recent call last)
Input In [10], in <cell line: 3>()
      1 ceez_shp = '../../Documents/projects/oamapper/data/raw/geo/marine_regions_ceez_simplified.shp'
      2 ceez = geemap.shp_to_ee(ceez_shp)
----> 3 Map.addLayer(ceez, {}, 'CEEZ')

File ~\Anaconda3\envs\gee\lib\site-packages\geemap\geemap.py:1409, in Map.add_ee_layer(self, ee_object, vis_params, name, shown, opacity)
   1406         print("The provided palette is invalid.")
   1407         raise Exception(e)
-> 1409 map_id_dict = ee.Image(image).getMapId(vis_params)
   1410 tile_layer = ipyleaflet.TileLayer(
   1411     url=map_id_dict["tile_fetcher"].url_format,
   1412     attribution="Google Earth Engine",
   (...)
   1416     max_zoom=24,
   1417 )
   1419 layer = self.find_layer(name=name)

File ~\Anaconda3\envs\gee\lib\site-packages\ee\image.py:131, in Image.getMapId(self, vis_params)
    129 vis_image, request = self._apply_visualization(vis_params)
    130 request['image'] = vis_image
--> 131 response = data.getMapId(request)
    132 response['image'] = self
    133 return response

File ~\Anaconda3\envs\gee\lib\site-packages\ee\data.py:575, in getMapId(params)
    572   request['visualizationOptions'] = visualizationOptions
    573 # Make it return only the name field, as otherwise it echoes the entire
    574 # request, which might be large.
--> 575 result = _execute_cloud_call(
    576     _get_cloud_api_resource().projects().maps().create(
    577         parent=_get_projects_path(), fields='name', body=request))
    578 map_name = result['name']
    579 url_format = '%s/%s/%s/tiles/{z}/{x}/{y}' % (
    580     _tile_base_url, _cloud_api_utils.VERSION, map_name)

File ~\Anaconda3\envs\gee\lib\site-packages\ee\data.py:336, in _execute_cloud_call(call, num_retries)
    334   return call.execute(num_retries=num_retries)
    335 except googleapiclient.errors.HttpError as e:
--> 336   raise _translate_cloud_exception(e)

EEException: Request payload size exceeds the limit: 10485760 bytes.

[giswqs]

This is a limit enforced by Earth Engine. There is nothing geemap can do about it. You can either simplify the shapes, reducing the number of vertices to make the file smaller, or you need to upload the file using the GEE Code Editor.

[svHatch]

@giswqs - understood. However, the error message is misleading. My shapefile is 5 Mb, and the listed payload size limit is 9 Mb.

What are the extra bytes that are part of the request that are making it unable to upload?

geemap could get total size info after converting to geoJSON, then assess whether it would trigger a size limit, and then provide a more helpful error message: "Earth Engine limits API requests to {this size}. Your local file is {this size}. In order to use this local file you must upload it through {this site} and then reference it from geemap."

Something along those lines.

[giswqs]

The error message is from the earth engine API, not from geemap. From my experience, any file larger than 4.5 MB will likely fail to upload. Keep in mind that under the hood, the shapefile is being converted to GeoJSON, then to ee.FeatureCollection. The GeoJSON can be much larger than the original shapefile. A GeoJSON smaller than 10485760 bytes does not guarantee that it will be uploaded successfully. There could be other issues causing the upload to fail, e.g., encoding error.

If you have a better solution for handling the error, please feel free to submit a pull request.

[giswqs]

Here is the source code. https://github.com/giswqs/geemap/blob/79ed0b7cc317dac0de70f39af76f0b49dd0131d4/geemap/common.py#L1152