SpatialDays / catapult-workfinder

Tooling to find work to do for common sensing pipelines.
Other
0 stars 0 forks source link

limit order size to 5000 max #3

Closed ivica3730k closed 2 years ago

ivica3730k commented 2 years ago

USGS api only accepts a maximum of 5000 orders at the time

(env) ivica@icepad-2:~/Coding/new-ard/catapult-workfinder$ python3 -m workfinder.workfinder LANDSAT8 
2022-11-09 14:06:50,103 DEBUG shapely.geos geos.py:73 -- Found GEOS DLL: <CDLL '/home/ivica/Coding/new-ard/env/lib/python3.8/site-packages/shapely/.libs/libgeos_c-a68605fd.so.1.13.1', handle 206d780 at 0x7f5aae92b9a0>, using it.
2022-11-09 14:06:50,111 DEBUG shapely.geos geos.py:38 -- Trying `CDLL(libc.so.6)`
2022-11-09 14:06:50,112 DEBUG shapely.geos geos.py:55 -- Library path: 'libc.so.6'
2022-11-09 14:06:50,112 DEBUG shapely.geos geos.py:56 -- DLL: <CDLL 'libc.so.6', handle 7f5af83e2000 at 0x7f5aae92b9d0>
Limit is:  -1
2022-11-09 14:06:50,151 INFO root workfinder.py:64 -- starting work search for LANDSAT8
2022-11-09 14:06:50,198 INFO root __init__.py:70 -- TM_WORLD_BORDERS.geojson already available
/home/ivica/Coding/new-ard/env/lib/python3.8/site-packages/pyproj/crs/crs.py:141: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6
  in_crs_string = _prepare_from_proj_string(in_crs_string)
/home/ivica/Coding/new-ard/env/lib/python3.8/site-packages/pyproj/crs/crs.py:141: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6
  in_crs_string = _prepare_from_proj_string(in_crs_string)
2022-11-09 14:06:51,516 INFO root __init__.py:70 -- WRS2_descending.geojson already available

2022-11-09 13:54:04,266 DEBUG root espa_api.py:21 -- {
    "errors": [
        {
            "2 validation errors": [
                "Single sensor input count exceeds maximum of 5000 in olitirs8_collection_2_l1.inputs",
                "Order exceeds maximum allowable inputs of 5000"
            ]
        }
    ]
}
2022-11-09 13:54:04,266 WARNING root espa_api.py:25 -- 400 Client Error: BAD REQUEST for url: https://espa.cr.usgs.gov/api/v1/order
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/ivica/Coding/new-ard/catapult-workfinder/workfinder/workfinder.py", line 77, in <module>
    main()
  File "/home/ivica/Coding/new-ard/env/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/ivica/Coding/new-ard/env/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/ivica/Coding/new-ard/env/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ivica/Coding/new-ard/env/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/ivica/Coding/new-ard/catapult-workfinder/workfinder/workfinder.py", line 69, in main
    processors[param].submit_tasks(work)
  File "/home/ivica/Coding/new-ard/catapult-workfinder/workfinder/search/Landsat.py", line 44, in submit_tasks
    order_id = self._order_products(to_do_list)
  File "/home/ivica/Coding/new-ard/catapult-workfinder/workfinder/search/Landsat.py", line 100, in _order_products
    resp = self._espa.call('order', verb='post', body=order)
  File "/home/ivica/Coding/new-ard/catapult-workfinder/workfinder/api/espa_api.py", line 23, in call
    response.raise_for_status()
  File "/home/ivica/Coding/new-ard/env/lib/python3.8/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: BAD REQUEST for url: https://espa.cr.usgs.gov/api/v1/order
ivica3730k commented 2 years ago

See https://github.com/SpatialDays/catapult-workfinder/issues/2