Searching returns no result

[matiaslee]

Hi people!

landsat is not returning any results. For example:

$ landsat search -p 228,81 --latest 5
{
    "limit": 25, 
    "results": [], 
    "status": "SUCCESS", 
    "total": 0, 
    "total_returned": 0
}
['Search completed!']

$ landsat search -p 23,35 --latest 5
{
    "limit": 25, 
    "results": [], 
    "status": "SUCCESS", 
    "total": 0, 
    "total_returned": 0
}
['Search completed!']

$ landsat search -p 226,84 --latest 5
{
    "limit": 25, 
    "results": [], 
    "status": "SUCCESS", 
    "total": 0, 
    "total_returned": 0
}

Any suggestion/pointer to solve this problem? Thanks in advance!

Matias

[matthewhanson]

Hi @matiaslee , I think this was due to a change on the server side of the API. We recently switched over to a new deployment. I've updated it and I think this should be resolved now.

Can you check and post back here? Thanks for posting the issue!

[matiaslee]

Hi @matthewhanson !

thanks for the quick answer. Still not working (I've installed the dev version of landsat-util):

$ landsat search -p 226,84 --latest 5
{
    "limit": 25, 
    "results": [], 
    "status": "SUCCESS", 
    "total": 0, 
    "total_returned": 0
}
['Search completed!']

$ landsat --version
landsat version 0.14.dev0

[matthewhanson]

@matiaslee thanks for checking, I'll dig deeper into this.

[lucasvr]

Hey guys! I'm also stuck with the same problem. @matthewhanson is there anything I can help you with to debug this issue?

[matthewhanson]

@lucasvr @matiaslee I haven't had a chance to look into this yet, but should have some time tomorrow.

The API endpoint landsat-util uses was recently updated that fixed some other issues. There should have been no functional changes, although some unused fields were dropped. It appears as though the sat-api endpoint is working fine ( https://api.developmentseed.org/satellites/ ), but it could be that landsat-util is using one of the few fields that were dropped (I added back in cloudCoverFull based on the original issue here, which is just a duplicate of coud_cover). So this is the first think I would check.

If you find any additional info please post here.

[lucasvr]

Thanks for the hint! I have found the problem after a quick investigation. The search string that indicates the desired cloud cover should be "cloudCover:[%s-TO-%s]" as opposed to "cloudCoverFull:[%s-TO-%s]". I will send you a PR with the patch in a few.

[matthewhanson]

@lucasvr Hold off on the PR, it's better if I patch it on the API side that way landsat-util users won't need to update their versions.

Thanks a lot for finding the problem !

[matthewhanson]

@lucasvr it looks like cloudCover is already provided in the metadata: https://api.developmentseed.org/satellites/landsat

Although it's an integer, and I'm not sure what it means, it doesn't seem to be consistent with the documentation: https://landsat.usgs.gov/what-difference-between-scene-cloud-cover-and-land-cloud-cover-scores

However, it looks like the master branch already has this change: https://github.com/developmentseed/landsat-util/blob/develop/landsat/search.py#L291

Are you using the current version?

[lucasvr]

Hi @matthewhanson. The metadata structure looks like this:

{
    "results": [{
        "cloud_coverage": float,
        "cloudCover": int,
        "cloudCoverFull": float,
    }]
}

When we use "cloudCoverFull" in the search string we get an empty response from the server - both when formatting the min/max values with %s as well as when using %f and casting the values to float.

On the other hand, both "cloud_coverage" and "cloudCover" are treated as valid filters by the server. I can successfully tweak the results when using them. The server handles arguments formatted as %s and as %f just fine.

Could you double-check that the server is accepting cloudCover as a valid filter? Perhaps you have a whitelist and that string is not included in it?

I'm using the most recent git snapshot (development branch).

[matthewhanson]

@lucasvr I see what has happened here, I added back in cloudCoverFull, but only on return of the records, so it can't actually search against that in the elasticsearch instance.

The API is set up to accept "cloud_from" and "cloud_to" filters which use the cloud_coverage field, so I think landsat-util should use that.
https://github.com/sat-utils/sat-api-lib/blob/242e11d6c1a0754638890e2781391c5920936a3f/libs/queries.js#L188

So the easiest thing to do is to update landsat-util to use cloud_coverage as the filter (which is the same value as cloudCoverFull). Alternately I can update the records, but I basically need to reingest all the landsat data to add the field.

I'm just looking for the least painless fix here, because these issues will largely be fixed in the new sat-api and associated sat-utils: https://github.com/sat-utils/ there just isn't a direct replacement for landsat-util quite yet.

[lucasvr]

The easiest thing is certainly to update landsat-util, but all your user base will be forced to update to the most recent version of the software with that approach. I'm not sure how much of a problem that is given that you're planning to release a new sat-api soon.

Let me know if you want me to proceed with the PR or if you'll reingest the data on your side.

[matthewhanson]

@lucasvr I think I can fix this on server side without reingesting, I'll just remap any cloudCoverFull param in the request to cloud_coverage. Stay tuned.

[matiaslee]

Dear @matthewhanson ,

no news with respect this problem, right? If this is the case, @lucasvr could you create the PR in order to make visible a temporal solution (PR can be not accepted, but we will have a temporal fix to the problem)

Thanks in advance!

[lucasvr]

It looks like there's another problem too: the sceneids returned by a search command cannot be retrieved with landsat download <sceneid>: LC82310652017323LGN00 is not available on AWS S3 or Google Storage

So, even though the search command has been fixed, it looks like more code changes may be needed to point the download URLs to the right location.

[matiaslee]

I think this is an old problem: https://github.com/developmentseed/landsat-util/issues/233

For solving this problem I am using this version of landsat-util: https://github.com/jmellian/landsat-util with a USGS machine-to-machine access. The solution is the conclusion of the post. :)

[matthewhanson]

@matiaslee @lucasvr sorry, I did try and fix this server side by rewriting any cloudCoverFull parameter passed in into cloud_coverage. And while it does accept it as a parameter now, it returns the whole collection and doesn't filter it.

And I haven't had the time to get back to it. I'll at least push up my changes tonight and I'll post a link here in case anyone has any ideas.

[matiaslee]

Hi @matthewhanson !

no time? the same here :) I wait for your link, maybe I can do something for fixing the problem. Good weekend!

[matthewhanson]

@matiaslee @lucasvr Sorry for the long delay, but I believe I've managed to add back in support for cloudCoverFull on the server side which should fix this issue.

Please let me know if it's working for you.

[matiaslee]

dear @matthewhanson , bad news... I've tried again with a fresh landsat-util install and no success:

$ landsat --version
landsat version 0.14.dev0

$ landsat search -p 228,81 --latest 5
{
    "limit": 25, 
    "results": [], 
    "status": "SUCCESS", 
    "total": 0, 
    "total_returned": 0
}
['Search completed!']

$ landsat search -p 23,35 --latest 5
{
    "limit": 25, 
    "results": [], 
    "status": "SUCCESS", 
    "total": 0, 
    "total_returned": 0
}
['Search completed!']

$ landsat search -p 226,84 --latest 5
{
    "limit": 25, 
    "results": [], 
    "status": "SUCCESS", 
    "total": 0, 
    "total_returned": 0
}
['Search completed!']

[matthewhanson]

@matiaslee Hey is there an way you could try and print out the actual query it's making to api.developmentseed.org/satellites/

That will help me be able to debug.

[soumyapatro]

Having a similar issue.

landsat search --lat 38.9004204 --lon -77.0237117 ===> 0 items were found { "limit": 10, "results": [], "status": "SUCCESS", "total": 0, "total_returned": 0 }

[jessetrana]

Here's what I see when I do a request. Note that I also did the "pip install requests[security]" step.

landsat search --lat 38.9004204 --lon -77.0237117 Request

https://api.developmentseed.org/satellites/landsat?search=cloudCoverFull:[-1+TO+100.0]+AND+upperLeftCornerLatitude:[38.9004204+TO+1000]+AND+lowerRightCornerLatitude:[-1000+TO+38.9004204]+AND+lowerLeftCornerLongitude:[-1000+TO+-77.0237117]+AND+upperRightCornerLongitude:[-77.0237117+TO+1000]&limit=10

Response

{"meta":{"found":0,"name":"sat-api","license":"CC0-1.0","website":"https://api.developmentseed.org/satellites/","page":1,"limit":10},"results":[]}

Raw object

<Response [200]> ===> 0 items were found { "limit": 10, "results": [], "status": "SUCCESS", "total": 0, "total_returned": 0 } ['Search completed!']

[jessetrana]

Note that removing the cloudCoverFull part of the query seems to return results.

[jessetrana]

My hacky workaround until this is corrected was this Dockerfile: FROM developmentseed/landsat-util RUN pip install requests[security] RUN sed -i 's/cloudCoverFull/cloud_coverage/g' /landsat/landsat/search.py

[matiaslee]

Super nice fix! Thanks @jessetrana! I did the replacement in search.py and the download is working :) Thanks!

[Gnonpi]

I applied the sed command of @jessetrana , it fixed the problem

[matthewhanson]

This fix is on the develop branch and will be included in the next and final release, 0.14.0