JuliaClimate / STAC.jl

SpatioTemporal Asset Catalog (STAC) julia client
MIT License
28 stars 3 forks source link

fix Earthdata example (almost?) #15

Closed gaelforget closed 4 months ago

gaelforget commented 1 year ago

It seems that requests to https://cmr.earthdata.nasa.gov/search/granules.stac now require that the url includes a collection_concept_id. The modification to src/search.jl does this and allows for an empty query specification.

In the example, the handling of url, query v FeatureCollection also needed an update after https://github.com/JuliaClimate/STAC.jl/commit/918a1a95649769b01da5c26c8f1cd6d8761bdbbc I believe.

Some problems that may remain :

codecov-commenter commented 1 year ago

Codecov Report

Patch coverage: 50.00% and project coverage change: -1.58 :warning:

Comparison is base (9f0160f) 96.62% compared to head (6236557) 95.05%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #15 +/- ## ========================================== - Coverage 96.62% 95.05% -1.58% ========================================== Files 7 7 Lines 178 182 +4 ========================================== + Hits 172 173 +1 - Misses 6 9 +3 ``` | [Impacted Files](https://app.codecov.io/gh/JuliaClimate/STAC.jl/pull/15?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaClimate) | Coverage Δ | | |---|---|---| | [src/search.jl](https://app.codecov.io/gh/JuliaClimate/STAC.jl/pull/15?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaClimate#diff-c3JjL3NlYXJjaC5qbA==) | `88.57% <50.00%> (-8.21%)` | :arrow_down: |

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Alexander-Barth commented 1 year ago

It seems that if we make a get request then the collection_concept_id parameter get propagated automatically:

using STAC, URIs, Dates, HTTP, JSON3

timerange = (DateTime(2005,1,1),DateTime(2005,1,2))
collection_concept_id= "C1299783579-LPDAAC_ECS"

url = "https://cmr.earthdata.nasa.gov/search/granules.stac"

query = query = Dict(
    "collection_concept_id" => collection_concept_id,
    "temporal" => join(string.(timerange),','),
    "pageSize" => 100,
)

function FeatureCollection2(url,query; method=:post)
    if method == :get
        url = string(URI(URI(url), query = query))
    end

    ch = Channel{STAC.Item}() do c
        while true
            if method == :post
                @debug "post $url" query
                r = HTTP.post(url,[],JSON3.write(query))
            else
                @debug "get $url"
                r = HTTP.get(url)
            end
            data = JSON3.read(String(r.body))
            for d in data[:features]
                put!(c,STAC.Item("",d,STAC._assets(d)))
            end

            # check if there is a next page
            next = filter(d -> get(d,"rel",nothing) == "next",data[:links])
            # no more next page
            if length(next) == 0
                break
            else
                url = next[1][:href]
            end
        end
    end
end

collection = collect(FeatureCollection2(url,query, method=:get))

@show length(collection)
# output 402

I could not find any collection that supports opendap:

curl "https://cmr.earthdata.nasa.gov/search/collections.json?page_size=100" | python -m json.tool

What ever I query C1996881146-POCLOUD gives always an empty collection. Currently podaac.jpl.nasa.gov is down. Maybe this is related.

Alexander-Barth commented 4 months ago

This problem should be fixed now:

https://github.com/JuliaClimate/STAC.jl/commit/e163edb7728c8e62c20c461fe775ab25f6302e63

Thank for letting me know.

gaelforget commented 4 months ago

This problem should be fixed now:

e163edb

Thank for letting me know.

Works for me now. Thanks for this!