Closed wsshin closed 3 months ago
Hi @wsshin,
I'm afraid, this isn't really a problem restricted to JLD2 but one more generally with Unicode. There appear to be two different unicode representations of the "same" visual symbol and julia generates a different version depending on how you create it.
julia> :â
:â
julia> string(:â)
"â"
julia> string(:â) == "â"
false
julia> codeunits(string(:â))
2-element Base.CodeUnits{UInt8, String}:
0xc3
0xa2
julia> codeunits("â")
3-element Base.CodeUnits{UInt8, String}:
0x61
0xcc
0x82
julia> Symbol(string(:â)) == :â
true
julia> string(Symbol("â")) == "â"
true
julia> Symbol("â") == :â
false
julia> string(:â) == "â"
false
Thanks @JonasIsensee. I reported the issue to JuliaLang/julia.
The correct solution here is for JLD2 to apply normalization before doing the lookup.
The place to edit is here, I think: https://github.com/JuliaIO/JLD2.jl/blob/30dd57839159945fd3d17886891fe30b19367703/src/groups.jl#L25
The correct function to compare strings is Unicode.is_equal_normalized()
and one could consider always adding a normalization step prior to saving (or loading the whole thing) with Unicode.normalize()
I hit this with a variable named G\bar on Julia 1.9.0. I will rename variable to G for now. Thanks for documenting this issue.
@JonasIsensee, is there any reason why your https://github.com/JuliaIO/JLD2.jl/issues/466#issuecomment-1491471559 cannot be implemented? I am experiencing this problem a year from my initial report, and I find that it hasn't been resolved yet.
Oh, it should be relatively easy to fix. You are welcome to submit a PR.
Ensuring unicode normalization at every step, as required for the behavior added in #561, caused a significant loss in performance for working with files independent of whether they used unicode or not.
Due to this, this fix was reverted in v0.4.51
.
I would recommend that you add the normalization to your code or not rely on unnormalized unicode.
I can save variables named with modifiers (like
\hat
) with no problem:However, if I try to load the saved variable, an error is generated complaining that it cannot find the variable name:
Strangely, if I simply load the file without specifying the variable name, the result shows that the variable is actually loaded:
It turns out that the loaded
"â"
has a different byte representation than the saved"â"
:So, somewhere during
jldsave()
seems to change the byte representation of"â"
.Here is the version info: