Plotting RGB Rasters with Makie

tiemvanderdeure commented 12 months ago

I used to be able to plot RGBA rasters with Makie using heatmap, but some recent version apparently broke this. I now get an error message that says

Makie needs to convert all plot input arguments to types that can be consumed by the backends (typically Arrays with Float32 elements).
You can define a method for `Makie.convert_arguments` (a type recipe) for these types or their supertypes to make this set of arguments convertible (See http://docs.makie.org/stable/documentation/recipes/index.html).

Converting to an Array and then plotting still works. Any idea what broke this?

This triggers the error:

using Rasters, GLMakie, Colors
xdim = X(1:10); ydim = Y(1:10)
rgba_raster = Raster([RGBA(i,j,0,1) for i in 0.1:0.1:1, j in 0.1:0.1:1], (xdim, ydim))
heatmap(rgba_raster, color_map = rgba_raster)

rafaqz commented 12 months ago

Full stack trace always helps with those ideas ;)

tiemvanderdeure commented 12 months ago

Here you go!

Stacktrace:
  [1] _broadcast_getindex_evalf
    @ .\broadcast.jl:683 [inlined]
  [2] _broadcast_getindex
    @ .\broadcast.jl:656 [inlined]
  [3] getindex
    @ .\broadcast.jl:610 [inlined]
  [4] copy
    @ .\broadcast.jl:912 [inlined]
  [5] copy(bc::Base.Broadcast.Broadcasted{DimensionalData.DimensionalStyle{Base.Broadcast.DefaultArrayStyle{2}}, Tuple{DimensionalData.Dimensions.DimUnitRange{Int64, Base.OneTo{Int64}, X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, DimensionalData.Dimensions.DimUnitRange{Int64, Base.OneTo{Int64}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}}, typeof(Rasters._missing_or_float32), Tuple{Raster{Any, 2, Tuple{X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Matrix{Any}, Symbol, DimensionalData.Dimensions.LookupArrays.NoMetadata, Float32}}})
    @ DimensionalData C:\Users\tsh371\.julia\packages\DimensionalData\4TpBG\src\array\broadcast.jl:39
  [6] materialize
    @ .\broadcast.jl:873 [inlined]
  [7] _prepare_makie(A::Raster{RGBA{Float64}, 2, Tuple{X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Matrix{RGBA{Float64}}, Symbol, DimensionalData.Dimensions.LookupArrays.NoMetadata, Missing})
    @ Rasters C:\Users\tsh371\.julia\dev\Rasters\src\plotrecipes.jl:384
  [8] convert_arguments(#unused#::DiscreteSurface, raster::Raster{RGBA{Float64}, 2, Tuple{X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Matrix{RGBA{Float64}}, Symbol, DimensionalData.Dimensions.LookupArrays.NoMetadata, Missing})
    @ Rasters C:\Users\tsh371\.julia\dev\Rasters\src\plotrecipes.jl:305
  [9] convert_arguments(T::Type{Heatmap{Tuple{Raster{RGBA{Float64}, 2, Tuple{X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Matrix{RGBA{Float64}}, Symbol, DimensionalData.Dimensions.LookupArrays.NoMetadata, Missing}}}}, args::Raster{RGBA{Float64}, 2, Tuple{X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Matrix{RGBA{Float64}}, Symbol, DimensionalData.Dimensions.LookupArrays.NoMetadata, Missing}; kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Makie C:\Users\tsh371\.julia\packages\Makie\Ylh0P\src\conversions.jl:10
 [10] convert_arguments
    @ C:\Users\tsh371\.julia\packages\Makie\Ylh0P\src\conversions.jl:7 [inlined]
 [11] plot!(scene::Scene, P::Type{Heatmap}, attributes::Attributes, args::Raster{RGBA{Float64}, 2, Tuple{X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Matrix{RGBA{Float64}}, Symbol, DimensionalData.Dimensions.LookupArrays.NoMetadata, Missing}; kw_attributes::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Makie C:\Users\tsh371\.julia\packages\Makie\Ylh0P\src\interfaces.jl:292
 [12] plot!
    @ C:\Users\tsh371\.julia\packages\Makie\Ylh0P\src\interfaces.jl:275 [inlined]
 [13] plot(P::Type{Heatmap}, args::Raster{RGBA{Float64}, 2, Tuple{X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Matrix{RGBA{Float64}}, Symbol, DimensionalData.Dimensions.LookupArrays.NoMetadata, Missing}; axis::NamedTuple{(), Tuple{}}, figure::NamedTuple{(), Tuple{}}, kw_attributes::Base.Pairs{Symbol, Raster{RGBA{Float64}, 2, Tuple{X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, 
DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Matrix{RGBA{Float64}}, Symbol, DimensionalData.Dimensions.LookupArrays.NoMetadata, Missing}, Tuple{Symbol}, NamedTuple{(:color_map,), Tuple{Raster{RGBA{Float64}, 2, Tuple{X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Matrix{RGBA{Float64}}, Symbol, DimensionalData.Dimensions.LookupArrays.NoMetadata, Missing}}}})
    @ Makie C:\Users\tsh371\.julia\packages\Makie\Ylh0P\src\figureplotting.jl:48
 [14] plot
    @ C:\Users\tsh371\.julia\packages\Makie\Ylh0P\src\figureplotting.jl:31 [inlined]
 [15] #heatmap#18
    @ C:\Users\tsh371\.julia\packages\MakieCore\zgrOY\src\recipes.jl:34 [inlined]
 [16] top-level scope

rafaqz commented 12 months ago

Yeah seems we are not handling colors... this will all move to DimensionalData.jl very soon though, we can fix there

tiemvanderdeure commented 12 months ago

All right. I figured out just passing dimensions as x and y arguments fixes it, even though I don't really understand why.

heatmap(lookup(dims(rgba_raster, (X, Y)))..., rgba_raster) works.

rafaqz commented 12 months ago

Yeah its just routing around the explicit method that does that.

Although do realise that can put the pixels in the wrong place... gdal lookup values are at the start of the pixel not the middle.

felixcremer commented 10 months ago

This fails currently, because we explicitly convert the data to Float32 in https://github.com/rafaqz/Rasters.jl/blob/7eec50cce13898a58cd1f9dfcd9b0c62d4a03948/ext/RastersMakieExt/plotrecipes.jl#L350.

If I remove the Float32 there, image(rgba_raster) works and the heatmap call fails while constructing the Colorbar see error below. Maybe we could special case the _prepare dim array function for RGB elementtypes.

julia> heatmap(rgba_raster, colormap = rgba_raster)
ERROR: Neither heatmap nor any of its children use a colormap. Cannot create a Colorbar from this plot, please create it manually.
        If this is a recipe, one needs to overload `Makie.extract_colormap(::Heatmap)` to allow for the automatical creation of a Colorbar.
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] Colorbar(fig_or_scene::GridPosition, plot::Heatmap{Tuple{Vector{Float32}, Vector{Float32}, Matrix{RGBA{Float32}}}}; kwargs::Base.Pairs{Symbol, String, Tuple{Symbol}, NamedTuple{(:label,), Tuple{String}}})
   @ Makie ~/.julia/packages/Makie/NrrbP/src/makielayout/blocks/colorbar.jl:93
 [3] heatmap(A::Raster{RGBA{Float64}, 2, Tuple{X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Matrix{RGBA{Float64}}, Symbol, DimensionalData.Dimensions.LookupArrays.NoMetadata, Missing}; x::Nothing, y::Nothing, colorbarkw::NamedTuple{(), Tuple{}}, attributes::Base.Pairs{Symbol, Raster{RGBA{Float64}, 2, Tuple{X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Matrix{RGBA{Float64}}, Symbol, DimensionalData.Dimensions.LookupArrays.NoMetadata, Missing}, Tuple{Symbol}, NamedTuple{(:colormap,), Tuple{Raster{RGBA{Float64}, 2, Tuple{X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Matrix{RGBA{Float64}}, Symbol, DimensionalData.Dimensions.LookupArrays.NoMetadata, Missing}}}})
   @ DimensionalDataMakie ~/Documents/DDPlotting/dev/DimensionalData/ext/DimensionalDataMakie.jl:147
 [4] top-level scope
   @ REPL[10]:1

rafaqz commented 10 months ago

Hmm what does Makie do for missing value in RGB? transparency?

We could pass missing but then makie will have to transform that again.

I guess the current method should dispatch on Real, and we can have a fallback that just uses missing

rafaqz / Rasters.jl

Plotting RGB Rasters with Makie #518