search

fatteneder / MakieSlides.jl

MIT License
17 stars 2 forks source link

Render emojis to PDF #23

Closed asinghvi17 closed 2 years ago

asinghvi17 commented 2 years ago

Basically, this pr introduces an emojifont font in the opening to solve any potential bbox / char shape mismatch issues. All emojis are rendered with this font. It also changes a lot of the plot! signatures to convert_arguments which allows the user to update with any supported type.

This is not yet complete, but I did this in the web editor so putting it out there.

asinghvi17 commented 2 years ago

I guess this is basically what I wanted to do. The framework for the Cairo special case is there but commented out, since I tried to use Cairo to draw colored emojis and it didn't (even though the docs explicitly list colored emojis as being supported). We could still draw the SVGs using Rsvg there though.

fatteneder commented 2 years ago

I think direct rendering of emojis will also require support for variation sequences from FreeTypeAbstraction.jl to extract the code point from the font files. Without that, you don't even get passed layout_formatted_text.

Also: The font not working could be related to the note made on the release page of OpenMoji, but not 100% sure: https://github.com/hfg-gmuend/openmoji/tree/master/font

:warning: The OpenMoji fonts work in all operating systems, but will currently only show color glyphs in Mozilla Firefox and Adobe CC. This is not a limitation of the generated fonts, but of the operating systems and applications.

Don't quote me on this, but I think I remember reading somewhere that unicode standards can have different implementations and that Firefox's and Adobe's implementations did have something in common or so. Perhaps that is the reason ...

We could still draw the SVGs using Rsvg there though.

Why not use PNGs instead? We already use those for GLMakie rendering.

fatteneder commented 2 years ago

Don't quote me on this, but I think I remember reading somewhere that unicode standards can have different implementations and that Firefox's and Adobe's implementations did have something in common or so. Perhaps that is the reason ...

I think this is what I remembered:

SVG in Open Type is a standard by Adobe and Mozilla for color OpenType and Open Font Format fonts.

Taken from https://github.com/DeeDeeG/noto-color-emoji-font#examples

fatteneder commented 2 years ago

I spent some more time on trying to understand how unicode variation sequences work and how to possibly render emojis from a font file. Here is a brief summary:

Experiments

Here are my experiments with the FreeType library. Mainly adding it here for my own reference. Interesting note: the germany flag below is automatically rendered, whereas in my editor I entered it through the unicode sequence \u1f1e9\u1f1ea.

using FreeTypeAbstraction
using FreeType

# Unicode variation sequence methods imported from `FreeType.jl`.
#
# function FT_Face_GetCharVariantIndex(face, charcode, variantSelector)
#     ccall((:FT_Face_GetCharVariantIndex, libfreetype), FT_UInt, (FT_Face, FT_ULong, FT_ULong), face, charcode, variantSelector)
# end
#
# function FT_Face_GetCharVariantIsDefault(face, charcode, variantSelector)
#     ccall((:FT_Face_GetCharVariantIsDefault, libfreetype), FT_Int, (FT_Face, FT_ULong, FT_ULong), face, charcode, variantSelector)
# end
#
# function FT_Face_GetVariantSelectors(face)
#     ccall((:FT_Face_GetVariantSelectors, libfreetype), Ptr{FT_UInt32}, (FT_Face,), face)
# end
#
# function FT_Face_GetVariantsOfChar(face, charcode)
#     ccall((:FT_Face_GetVariantsOfChar, libfreetype), Ptr{FT_UInt32}, (FT_Face, FT_ULong), face, charcode)
# end
#
# function FT_Face_GetCharsOfVariant(face, variantSelector)
#     ccall((:FT_Face_GetCharsOfVariant, libfreetype), Ptr{FT_UInt32}, (FT_Face, FT_ULong), face, variantSelector)
# end

function get_variant_selectors(font::FTFont)
  selectors = UInt32[]
  ptr_selectors = FT_Face_GetVariantSelectors(font)
  ptr_selectors == Ptr{UInt32}() && return selectors
  idx = 1
  v = unsafe_load(ptr_selectors, idx)
  while v != 0
    push!(selectors, v)
    idx += 1
    v = unsafe_load(ptr_selectors, idx)
  end
  return selectors
end

function get_charcodes_of_variant(font::FTFont, variantSelector::UInt32)
  charcodes = UInt32[]
  ptr_charcodes = FT_Face_GetCharsOfVariant(font, variantSelector)
  ptr_charcodes == Ptr{UInt32}() && return charcodes
  idx = 1
  v = unsafe_load(ptr_charcodes, idx)
  while v != 0
    push!(charcodes, v)
    idx += 1
    v = unsafe_load(ptr_charcodes, idx)
  end
  return charcodes
end

function get_variantselectors_of_charcode(font::FTFont, charcode::UInt32)
  selectors = UInt32[]
  ptr_selectors = FT_Face_GetVariantsOfChar(font, charcode)
  ptr_selectors == Ptr{UInt32}() && return selectors
  idx = 1
  v = unsafe_load(ptr_selectors, idx)
  while v != 0
    push!(selectors, v)
    idx += 1
    v = unsafe_load(ptr_selectors, idx)
  end
  return selectors
end

function get_charcode_variantselector_default(font::FTFont, charcode::UInt32, variantSelector::UInt32)
  return FT_Face_GetCharVariantIsDefault(font, charcode, variantSelector)
end

function get_charcode_variantselector_index(font::FTFont, charcode::UInt32, variantSelector::UInt32)
  code = FT_Face_GetCharVariantIndex(font, charcode, variantSelector)
  return code == 0 ? nothing : code
end

# fontname = "/home/florian/.fonts/openmoji/OpenMoji-Black.ttf"
# fontname= "/home/florian/.fonts/openmoji/OpenMoji-Color.ttf"
# fontname= "/tmp/OpenMoji-Color.ttf"
fontname = "/home/florian/.fonts/NotoColorEmoji.ttf"
font = FTFont(fontname)
display(font)

selectors = get_variant_selectors(font)
# println(selectors)
charsofvariant = get_charcodes_of_variant.(Ref(font), selectors)
# println(length.(charsofvariant))
# println.(charsofvariant)
# variantsofchar = get_variantselectors_of_charcode.(Ref(font), first(charsofvariant))

### returns non-sense
# s = first(selectors)
# c = first(first(get_charcodes_of_variant.(Ref(font), selectors)))

emoji_flagde = "🇩🇪"
cp_emoji_flagde = [ convert(UInt32, codepoint(emoji_flagde[i])) for i in eachindex(emoji_flagde) ]
println(cp_emoji_flagde)
# println(cp_emoji_flagde[1] in first(charsofvariant))
# println(cp_emoji_flagde)
# println(s in selectors)
# println(get_charcode_variantselector_index.(Ref(font), cp_emoji_flagde, selectors[1]))

emoji_info = '\U2139'
cu_info = convert(UInt32, emoji_info)
# println(convert(UInt32,emoji_info))
println(cu_info in first(charsofvariant))
vs_glyphindex_info = get_charcode_variantselector_index(font, cu_info, first(selectors))
ft_face = getfield(font, :ft_ptr)
err = FreeType.FT_Load_Glyph(ft_face, vs_glyphindex_info, FreeType.FT_LOAD_NO_SCALE)
display(err)

# summary:
# it appears that glyph loading itself is not implemented in FreeTypeAbstraction,
# alhtough, this sounds wrong because how would people then load all the characters for
# text rendering? Only with FT_Load_Char?
# I think the problem is that the variation sequence methods of FreeType return
# a glyph index in the end, instead of a character code (makes sense since we started
# with a VS character code), but FT_Load_Char expects a character code.
fatteneder commented 2 years ago

Moving forward, we should settle on using PNGs and/or SVGs for now.

Ideally, I would like to only use one format at all, but from what I understand right now GLMakie prefers PNGs and CairoMakie SVGs, correct?

If it is possible to use PNGs also with CairoMakie, then we should do so for the moment. Reasons are that 1) frankly we have more important features to implement for a release (global slide elements, performance issues, etc.) and 2) we have PNGs already working for GLMakie.

asinghvi17 commented 2 years ago

I think that last commit might be unnecessary - CairoMakie should be able to render this as is without any external intervention. It's only if you want SVG output in vector graphics that you need this kind of overload.

fatteneder commented 2 years ago

Using only

function CairoMakie.draw_plot(scene::Scene, screen::CairoMakie.CairoScreen, fmttxt::T) where T <: FormattedText

    txt = fmttxt.plots[1]
    CairoMakie.draw_plot(scene, screen, txt)

    if length(fmttxt.plots) > 1
        scttr = fmttxt.plots[2]
        CairoMakie.draw_plot(scene, screen, scttr)
    end
end

gives me the following error:

Stacktrace ```julia julia> include("examples/mwe.jl") ERROR: LoadError: MethodError: no method matching draw_marker(::Cairo.CairoContext, ::Matrix{ColorTypes.RGBA{Float32}}, ::Vec{2, Float32}, ::Vec2{Float64}, ::ColorTypes.RGBA {Float32}, ::Float32, ::Vec2{Float64}, ::Quaternionf) Closest candidates are: draw_marker(::Any, ::Circle, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any) at ~/.julia/packages/CairoMakie/gU5me/src/primitives.jl:281 draw_marker(::Any, ::GeometryBasics.HyperRectangle, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any) at ~/.julia/packages/CairoMakie/gU5me/src/primitives.jl:305 draw_marker(::Any, ::Char, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any) at ~/.julia/packages/CairoMakie/gU5me/src/primitives.jl:231 Stacktrace: [1] (::CairoMakie.var"#30#31"{Scene, Cairo.CairoContext, typeof(identity), StaticArrays.SMatrix{4, 4, Float32, 16}, StaticArrays.SMatrix{4, 4, Float32, 16}, FreeTypeAbstra ction.FTFont, Symbol, Symbol})(point::Point{2, Float32}, col::ColorTypes.RGBA{Float32}, markersize::Vec{2, Float32}, strokecolor::ColorTypes.RGBA{Float32}, strokewidth::Floa t32, m::Matrix{ColorTypes.RGBA{Float32}}, mo::Vec{2, Float32}, rotation::Quaternionf) @ CairoMakie ~/.julia/packages/CairoMakie/gU5me/src/primitives.jl:224 [2] macro expansion @ ~/.julia/packages/Makie/bJ9rD/src/utilities/utilities.jl:198 [inlined] [3] broadcast_foreach(::CairoMakie.var"#30#31"{Scene, Cairo.CairoContext, typeof(identity), StaticArrays.SMatrix{4, 4, Float32, 16}, StaticArrays.SMatrix{4, 4, Float32, 16 }, FreeTypeAbstraction.FTFont, Symbol, Symbol}, ::Vector{Point{2, Float32}}, ::ColorTypes.RGBA{Float32}, ::Vector{Vec{2, Float32}}, ::ColorTypes.RGBA{Float32}, ::Float32, :: Vector{Matrix{ColorTypes.RGBA{Float32}}}, ::Vector{Vec{2, Float32}}, ::Quaternionf) @ Makie ~/.julia/packages/Makie/bJ9rD/src/utilities/utilities.jl:184 [4] draw_atomic_scatter(scene::Scene, ctx::Cairo.CairoContext, transfunc::Function, colors::ColorTypes.RGBA{Float32}, markersize::Vector{Vec{2, Float32}}, strokecolor::Col orTypes.RGBA{Float32}, strokewidth::Float32, marker::Vector{Matrix{ColorTypes.RGBA{Float32}}}, marker_offset::Vector{Vec{2, Float32}}, rotations::Quaternionf, model::StaticA rrays.SMatrix{4, 4, Float32, 16}, positions::Vector{Point{2, Float32}}, size_model::StaticArrays.SMatrix{4, 4, Float32, 16}, font::FreeTypeAbstraction.FTFont, markerspace::S ymbol, space::Symbol) @ CairoMakie ~/.julia/packages/CairoMakie/gU5me/src/primitives.jl:209 [5] draw_atomic(scene::Scene, screen::CairoMakie.CairoScreen{Cairo.CairoSurfaceBase{UInt32}}, primitive::Scatter) @ CairoMakie ~/.julia/packages/CairoMakie/gU5me/src/primitives.jl:201 [6] draw_plot(scene::Scene, screen::CairoMakie.CairoScreen{Cairo.CairoSurfaceBase{UInt32}}, primitive::Scatter{Tuple{Vector{Point{2, Float32}}}}) @ CairoMakie ~/.julia/packages/CairoMakie/gU5me/src/infrastructure.jl:251 [7] draw_plot(scene::Scene, screen::CairoMakie.CairoScreen{Cairo.CairoSurfaceBase{UInt32}}, fmttxt::Combined{MakieSlides.formattedtext, Tuple{Markdown.Paragraph}}) @ MakieSlides ~/wd/MakieSlides.jl/src/formattedtext.jl:262 [8] cairo_draw(screen::CairoMakie.CairoScreen{Cairo.CairoSurfaceBase{UInt32}}, scene::Scene) @ CairoMakie ~/.julia/packages/CairoMakie/gU5me/src/infrastructure.jl:192 [9] save(name::String, presentation::Presentation; aspect::Tuple{Int64, Int64}) @ MakieSlides ~/wd/MakieSlides.jl/src/MakieSlides.jl:234 [10] save(name::String, presentation::Presentation) @ MakieSlides ~/wd/MakieSlides.jl/src/MakieSlides.jl:219 [11] top-level scope @ ~/wd/MakieSlides.jl/examples/mwe.jl:159 [12] include(fname::String) @ Base.MainInclude ./client.jl:451 [13] top-level scope @ REPL[20]:1 in expression starting at /home/florian/wd/MakieSlides.jl/examples/mwe.jl:159 ```

I thought that was the problem you mentioned here https://github.com/fatteneder/MakieSlides.jl/pull/15#issuecomment-1137875735.

But now that you say it, it might be enough to just implement the draw_marker method for Matrix{RGBAf} overload.

asinghvi17 commented 2 years ago

Yeah that's what would have to happen. We could implement it here for now and upstream it to CairoMakie...see the implementation of the fast path in the heatmap draw_atomic for how you need to do it.

fatteneder commented 2 years ago

Got it to work. Only thing is that I now convert the Matrix{RGBAf} pixels first to Matrix{ARGB32} and then create a CairoSurface with it. I tried to do avoid this by using Cairo.move_to and Cairo.set_source_rgba but could not get it work. Do you know of a more efficient solution?

asinghvi17 commented 2 years ago

AFAIK that solution is optimal since Cairo wants UInt32 color. It would have been converted either way.

Overall it looks good to me. Thanks for pushing it this far! I am on vacation now but feel free to upstream the method you implemented to CairoMakie.