EcoJulia / RasterDataSources.jl

Easily download and use raster data sets in Julia
MIT License
21 stars 10 forks source link

[WIP] Add the Deltares World Flood dataset #58

Open asinghvi17 opened 1 year ago

asinghvi17 commented 1 year ago

Description at https://planetarycomputer.microsoft.com/dataset/deltares-floods and https://nbviewer.org/github/microsoft/AIforEarthDataSets/blob/main/data/deltares-floods.ipynb

Basically a computation of coastal flooding with certain parameters (return time, year) on a specified DEM. This is ~1gb in size.

Needs docs and tests (if possible)

codecov[bot] commented 1 year ago

Codecov Report

Patch coverage has no change and project coverage change: -2.37 :warning:

Comparison is base (14423e0) 72.25% compared to head (648b12d) 69.89%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #58 +/- ## ========================================== - Coverage 72.25% 69.89% -2.37% ========================================== Files 19 20 +1 Lines 739 764 +25 ========================================== Hits 534 534 - Misses 205 230 +25 ``` | [Impacted Files](https://codecov.io/gh/EcoJulia/RasterDataSources.jl/pull/58?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=EcoJulia) | Coverage Δ | | |---|---|---| | [src/deltares/worldflood.jl](https://codecov.io/gh/EcoJulia/RasterDataSources.jl/pull/58?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=EcoJulia#diff-c3JjL2RlbHRhcmVzL3dvcmxkZmxvb2Quamw=) | `0.00% <0.00%> (ø)` | | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=EcoJulia). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=EcoJulia)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

rafaqz commented 1 year ago

If the file is too big you can add tests to run locally but keep them out of CI.

Also note keywords are pretty standardized here, so go for res instead of resolution. We also pass the whole string 10km because the units can be mixed and it gets confusing. The string makes it much clearer what is being specified in a script.

I would also just use year.

Thinking about this, we really need a trait interface for dimensional keywords so we can automatically make series out of them. It's a bit hacky and specific how its currently done, Rasters.jl just special cases month and date keywords. We would need to add year for this to work, but again it wont scale in the long term.

See: https://github.com/rafaqz/Rasters.jl/blob/main/src/sources/rasterdatasources.jl#L95-L112

asinghvi17 commented 1 year ago

In this case year is actually a parameter, not a time - it indicates the year of reference for sea level.

asinghvi17 commented 1 year ago
julia> Raster(Deltares{WorldFlood}, nothing; lazy = true)
[ Info: No `name` or `key` keyword provided, using first valid layer with name `:inun`
432000×216000×1 Raster{Union{Missing, Float32},3} inun with dimensions: 
  X Mapped{Float64} Float64[-180.00000003333332, -179.9991667, …, 179.99833329985603, 179.99916663318936] ForwardOrdered Regular Points crs: EPSG mappedcrs: EPSG,
  Y Mapped{Float64} Float64[-90.00000003326134, -89.99916669992801, …, 89.9983333, 89.99916663333333] ForwardOrdered Regular Points crs: EPSG mappedcrs: EPSG,
  Ti Sampled{Dates.DateTime} Dates.DateTime[Dates.DateTime("2010-01-01T00:00:00")] ForwardOrdered Irregular Points
extent: Extent(X = (-180.00000003333332, 179.99916663318936), Y = (-90.00000003326134, 89.99916663333333), Ti = (Dates.DateTime("2010-01-01T00:00:00"), Dates.DateTime("2010-01-01T00:00:00")))
missingval: missing
crs: EPSG:4326
mappedcrs: EPSG:4326

from file:
/Users/anshul/downloads/Deltares/WorldFlood/GFM_global_NASADEM90m_2050slr_rp0100_masked.nc

Not sure what to put as a layer - might end up being the year.

rafaqz commented 1 year ago

Yeah, we a squishing a wide range of datset shapes into a regular structure, and it doesnt always work so cleanly.

What are the options to be layers?

You can just have a single layer name :inun that will be the no argument default.

rafaqz commented 3 months ago

Just remembered this PR. SRTM already doesn't have a layer as there is only one. This could be the same?

We may need to add/fix dispatch in Rasters.jl to handle that.