API Issues

[frankwang95]

It appears the the scene database API is still behaving somewhat erratically. When requests are made, return strings look something like this:

{u'meta': {u'website': u'https://api.developmentseed.org/satellites/', u'name': u'sat-api', u'license': u'CC0-1.0', u'limit': 1, u'found': 903065, u'page': 1}, u'results': [{u'BPF_NAME_OLI': u'LO8BPF20170118013816_20170118022031.01', u'NADIR_OFFNADIR': u'NADIR', u'data_geometry': {u'crs': {u'type': u'name', u'properties': {u'name': u'urn:ogc:def:crs:EPSG:8.9:4326'}}, u'type': u'Polygon', u'coordinates': [[[81.37965, -78.07199], [75.26086, -76.90308], [69.35418, -78.09054], [75.52485, -79.38313], [81.37965, -78.07199]]]}, u'sunAzimuth': 77.06545633, u'cloudCover': 1, u'sceneCenterLatitude': -78.14117, u'cartURL': u'http://earthexplorer.usgs.gov/order/process?dataset_name=LANDSAT_8&ordered=LC81121162017018LGN00', u'GROUND_CONTROL_POINTS_VERSION': 4, u'cloud_coverage': 12.67, u'scene_id': u'LC81121162017018LGN00', u'GROUND_CONTROL_POINTS_MODEL': -1, u'row': 116, u'imageQuality1': 9, u'cloudCoverFull': 12.67, u'browseURL': u'http://earthexplorer.usgs.gov/browse/landsat_8/2017/112/116/LC81121162017018LGN00.jpg', u'browseAvailable': u'Y', u'BPF_NAME_TIRS': u'LT8BPF20170113003504_20170113025501.01', u'dayOrNight': u'DAY', u'TIRS_SSM_MODEL': u'PRELIMINARY', u'CPF_NAME': u'L8CPF20170101_20170331.02', u'FULL_PARTIAL_SCENE': u'FULL', u'DATA_TYPE_L1': u'L1GT', u'lowerLeftCornerLatitude': -78.09054, u'sceneStartTime': u'2017:018:02:18:55.6907450', u'dateUpdated': u'2017-01-17', u'sensor': u'OLI_TIRS', u'thumbnail': u'https://ad-thumbnails.s3.amazonaws.com/LC81121162017018LGN00.jpg', u'sunElevation': 23.65621635, u'satellite_name': u'landsat-8', u'acquisitionDate': u'2017-01-18', u'PROCESSING_SOFTWARE_VERSION': u'LPGS_2.6.2', u'lowerRightCornerLongitude': 75.52485, u'lowerRightCornerLatitude': -79.38313, u'sceneCenterLongitude': 75.38062, u'upperLeftCornerLongitude': 75.26086, u'date': u'2017-01-18', u'path': 112, u'lowerLeftCornerLongitude': 69.35418, u'GEOMETRIC_RMSE_MODEL_X': 0, u'GEOMETRIC_RMSE_MODEL_Y': 0, u'sceneStopTime': u'2017:018:02:19:27.4607420', u'upperLeftCornerLatitude': -76.90308, u'upperRightCornerLongitude': 81.37965, u'DATE_L1_GENERATED': u'2017-01-17 22:14:55', u'upperRightCornerLatitude': -78.07199, u'sceneID': u'LC81121162017018LGN00', u'RLUT_FILE_NAME': u'L8RLUT20150303_20431231v11.h5', u'ROLL_ANGLE': 0, u'receivingStation': u'LGN'}]}

This looks like valid json but the api does not respond to the limit argument (every query only returns one result) and moreover, the search source code in landsat-utils encounters an exception when trying to parse it:

  File "/usr/local/bin/landsat", line 8, in <module>
    landsat.landsat.__main__()
  File "/Library/Python/2.7/site-packages/landsat/landsat.py", line 488, in __main__
    print(main(args))
  File "/Library/Python/2.7/site-packages/landsat/landsat.py", line 357, in main
    geojson=args.geojson)
  File "/Library/Python/2.7/site-packages/landsat/search.py", line 141, in search
    result['total'] = r_dict['meta']['results']['total']
KeyError: 'results```

[drewbo]

@frankwang95 can you try updating to v0.13.1? The exception should be fixed in that version

[frankwang95]

Hi @drewbo, sorry I see that the parsing issue is fixed. Is there a fix to the limit issue as well? I've been making requests to your api directly so these changes are a bit slow to come to me - has the query conventions changed?

[drewbo]

The endpoint should support both older (lucene based) and newer syntax but I'm a bit behind on docs and testing to ensure consistency on that. If you append &limit=n to the end of your direct API request it should work; let me know if that isn't the case

[LuisSevillano]

I had the same issue and fixed updating to v0.13.1. Thanks @drewbo

[frankwang95]

Hi Drewbo, sorry for giving you so much trouble, but the limit argument to the search string is still giving me issues. Running the following string:

https://api.developmentseed.org/landsat?search=&limit=10

Returns only one result:

{"meta":{"found":910768,"name":"sat-api","license":"CC0-1.0","website":"https://api.developmentseed.org/satellites/","page":1,"limit":1},"results":[{"scene_id":"LC81000202017030LGN00","satellite_name":"landsat-8","cloud_coverage":19.53,"date":"2017-01-30","thumbnail":"https://ad-thumbnails.s3.amazonaws.com/LC81000202017030LGN00.jpg","data_geometry":{"crs":{"type":"name","properties":{"name":"urn:ogc:def:crs:EPSG:8.9:4326"}},"type":"Polygon","coordinates":[[[161.38522,57.878],[158.27701,58.39804],[157.37548,56.70892],[160.35236,56.20273],[161.38522,57.878]]]},"browseAvailable":"Y","browseURL":"http://earthexplorer.usgs.gov/browse/landsat_8/2017/100/020/LC81000202017030LGN00.jpg","sceneID":"LC81000202017030LGN00","sensor":"OLI_TIRS","acquisitionDate":"2017-01-30","dateUpdated":"2017-01-29","path":100,"row":20,"upperLeftCornerLatitude":58.39804,"upperLeftCornerLongitude":158.27701,"upperRightCornerLatitude":57.878,"upperRightCornerLongitude":161.38522,"lowerLeftCornerLatitude":56.70892,"lowerLeftCornerLongitude":157.37548,"lowerRightCornerLatitude":56.20273,"lowerRightCornerLongitude":160.35236,"sceneCenterLatitude":57.30952,"sceneCenterLongitude":159.3503,"cloudCover":1,"cloudCoverFull":19.53,"dayOrNight":"DAY","sunElevation":13.6553746,"sunAzimuth":163.05483232,"receivingStation":"LGN","sceneStartTime":"2017:030:00:26:23.6556890","sceneStopTime":"2017:030:00:26:55.4256860","imageQuality1":9,"DATA_TYPE_L1":"L1T","cartURL":"http://earthexplorer.usgs.gov/order/process?dataset_name=LANDSAT_8&ordered=LC81000202017030LGN00","ROLL_ANGLE":-0.001,"GEOMETRIC_RMSE_MODEL_X":10.185,"GEOMETRIC_RMSE_MODEL_Y":11.822,"FULL_PARTIAL_SCENE":"FULL","NADIR_OFFNADIR":"NADIR","PROCESSING_SOFTWARE_VERSION":"LPGS_2.6.2","CPF_NAME":"L8CPF20170101_20170331.02","RLUT_FILE_NAME":"L8RLUT20150303_20431231v11.h5","BPF_NAME_OLI":"LO8BPF20170130002250_20170130010416.01","BPF_NAME_TIRS":"LT8BPF20170126095930_20170126100746.01","GROUND_CONTROL_POINTS_MODEL":178,"GROUND_CONTROL_POINTS_VERSION":4,"DATE_L1_GENERATED":"2017-01-29 22:17:21","TIRS_SSM_MODEL":"PRELIMINARY"}]}

In fact, I could be wrong, but it seems that any query I construct returns this same result. The start_date and end_date arguments don't make any impact either on the returned result.

[drewbo]

No problem @frankwang95, I'd actually hit https://api.developmentseed.org/satellites/landsat and see if that improves things; the other endpoint is forwarding to this one and I think the query parameters may get dropped? cc: @scisco

We've been migrating more of our work to the Sat-Utils repos and actually just uploaded some documentation at http://docs.sat-utils.org/

[frankwang95]

You've been a huge help! Everything is working now and those docs will be handy in the future.