search

JuliaLang / JuliaSyntax.jl

The Julia compiler frontend
Other
270 stars 33 forks source link

Error: JuliaSyntax parser failed — falling back to flisp! #505

Open resmaeilbeigi opened 3 days ago

resmaeilbeigi commented 3 days ago

I got this error in a large Julia codebase and cannot reproduce the issue on a small instance, but I think it is still useful to share what piece of code failed and how to work around it.

So calling the function

function timeToFloat(time)
  s = split(string(time), ":")
  Meta.parse(s[1])+Meta.parse(s[2])/60.0
end

raised the error

┌ Error: JuliaSyntax parser failed — falling back to flisp!
│ This is not your fault. Please submit a bug report to https://github.com/JuliaLang/JuliaSyntax.jl/issues
│   exception =
│    MethodError: no method matching pointer(::SubString{InlineStrings.String15})
│    
│    Closest candidates are:
│      pointer(::LaTeXStrings.LaTeXString) (method too new to be called from this world context.)
│       @ LaTeXStrings ~/tmp/julia-packages/1.10.5/packages/LaTeXStrings/ZtSdh/src/LaTeXStrings.jl:129
│      pointer(::CSV.ReversedBuf) (method too new to be called from this world context.)
│       @ CSV ~/tmp/julia-packages/1.10.5/packages/CSV/cwX2w/src/utils.jl:572
│      pointer(::CSV.ReversedBuf, ::Integer) (method too new to be called from this world context.)
│       @ CSV ~/tmp/julia-packages/1.10.5/packages/CSV/cwX2w/src/utils.jl:572
│      ...
│    
│    Stacktrace:
│      [1] Base.JuliaSyntax.ParseStream(text::SubString{InlineStrings.String15}, index::Int64; version::VersionNumber)
│        @ Base.JuliaSyntax /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/base/JuliaSyntax/src/parse_stream.jl:309
│      [2] Base.JuliaSyntax.ParseStream(text::SubString{InlineStrings.String15}, index::Int64)
│        @ Base.JuliaSyntax /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/base/JuliaSyntax/src/parse_stream.jl:307
│      [3] core_parser_hook(code::SubString{InlineStrings.String15}, filename::String, lineno::Int64, offset::Int64, options::Symbol)
│        @ Base.JuliaSyntax /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/base/JuliaSyntax/src/hooks.jl:162
│      [4] invoke_in_world(::UInt64, ::Any, ::Any, ::Vararg{Any}; kwargs::@Kwargs{})
│        @ Base ./essentials.jl:926
│      [5] invoke_in_world(::UInt64, ::Any, ::Any, ::Vararg{Any})
│        @ Base ./essentials.jl:923
│      [6] (::Base.JuliaSyntax.var"#invoke_fixedworld#120"{Base.JuliaSyntax.var"#invoke_fixedworld#117#121"{typeof(Base.JuliaSyntax.core_parser_hook), UInt64}})(::SubString{InlineStrings.String15}, ::Vararg{Any}; kws::@Kwargs{})
│        @ Base.JuliaSyntax /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/base/JuliaSyntax/src/hooks.jl:118
│      [7] (::Base.JuliaSyntax.var"#invoke_fixedworld#120"{Base.JuliaSyntax.var"#invoke_fixedworld#117#121"{typeof(Base.JuliaSyntax.core_parser_hook), UInt64}})(::SubString{InlineStrings.String15}, ::Vararg{Any})
│        @ Base.JuliaSyntax /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/base/JuliaSyntax/src/hooks.jl:117
│      [8] _parse_string(text::SubString{InlineStrings.String15}, filename::String, lineno::Int64, index::Int64, options::Symbol)
│        @ Base.Meta ./meta.jl:200
│      [9] #parse#3
│        @ ./meta.jl:238 [inlined]
│     [10] parse
│        @ ./meta.jl:236 [inlined]
│     [11] parse(str::SubString{InlineStrings.String15}; filename::String, raise::Bool, depwarn::Bool)
│        @ Base.Meta ./meta.jl:278
│     [12] parse
│        @ ./meta.jl:276 [inlined]
│     [13] timeToFloat(time::InlineStrings.String15)

at the line containing Meta.parse(s[1])+Meta.parse(s[2])/60.0. The error goes away if s is a list of Strings:

function timeToFloat(time)
  s = map(String, split(string(time), ":"))
  Meta.parse(s[1])+Meta.parse(s[2])/60.0
end

I should point out that this error happens after several calls to the function with varying arguments (the function argument time can be a String or Dates.Time object).

Version Info:

Julia Version 1.10.5
Commit 6f3fdf7b362 (2024-08-27 14:19 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 16 × 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, tigerlake)
Threads: 1 default, 0 interactive, 1 GC (on 16 virtual cores)
Environment:
  LD_LIBRARY_PATH = :/opt/gurobi1001/linux64/lib
  JULIA_PARALLEL = false
  JULIA_VERSION = 1.10.5
  JULIA_LOAD_PATH = ./src/:
  JULIA_DEPOT_PATH = /home/rasul/tmp/julia-packages/1.10.5
LilithHafner commented 2 days ago

Thanks for reporting this issue!

This is a MWE reproducer for Julia:

Meta.parse(split(InlineString("1 2"))[1])

And this is a MWE reproducer for JuliaSyntax:

julia> JuliaSyntax.parsestmt(SyntaxNode, split(InlineString("1 2"))[1])
ERROR: MethodError: no method matching pointer(::SubString{String3})

Closest candidates are:
  pointer(::Cwstring)
   @ Base c.jl:187
  pointer(::Base64.Buffer)
   @ Base64 ~/.julia/juliaup/julia-1.10.5+0.aarch64.linux.gnu/share/julia/stdlib/v1.10/Base64/src/buffer.jl:20
  pointer(::Cstring)
   @ Base c.jl:186
  ...

Stacktrace:
 [1] JuliaSyntax.ParseStream(text::SubString{String3}, index::Int64; version::VersionNumber)
   @ JuliaSyntax ~/.julia/packages/JuliaSyntax/BHOG8/src/parse_stream.jl:309
 [2] _parse(rule::Symbol, need_eof::Bool, ::Type{…}, text::SubString{…}, index::Int64; version::VersionNumber, ignore_trivia::Bool, filename::Nothing, first_line::Int64, ignore_errors::Bool, ignore_warnings::Bool, kws::@Kwargs{})
   @ JuliaSyntax ~/.julia/packages/JuliaSyntax/BHOG8/src/parser_api.jl:80
 [3] _parse(rule::Symbol, need_eof::Bool, ::Type{SyntaxNode}, text::SubString{String3}, index::Int64)
   @ JuliaSyntax ~/.julia/packages/JuliaSyntax/BHOG8/src/parser_api.jl:77
 [4] _parse(rule::Symbol, need_eof::Bool, ::Type{SyntaxNode}, text::SubString{String3})
   @ JuliaSyntax ~/.julia/packages/JuliaSyntax/BHOG8/src/parser_api.jl:77
 [5] parsestmt(::Type{SyntaxNode}, text::SubString{String3}; kws::@Kwargs{})
   @ JuliaSyntax ~/.julia/packages/JuliaSyntax/BHOG8/src/parser_api.jl:140
 [6] parsestmt(::Type{SyntaxNode}, text::SubString{String3})
   @ JuliaSyntax ~/.julia/packages/JuliaSyntax/BHOG8/src/parser_api.jl:140
 [7] top-level scope
   @ REPL[4]:1
Some type information was truncated. Use `show(err)` to see complete types.

The issue is that JuliaSyntax's ParseStream type uses a raw Vector{UInt8} for performance rather than using the abstract string API and that we assume (falsely) that all SubStrings are backed by such an array rather than explicitly stringifying them.

# Buffers originating from strings
function ParseStream(text::String, index::Integer=1; version=VERSION)
    ParseStream(unsafe_wrap(Vector{UInt8}, text),
                text, index, version)
end
function ParseStream(text::SubString, index::Integer=1; version=VERSION) # This should be ::SubString{String}, not ::SubString.
    # See also IOBuffer(SubString("x"))
    ParseStream(unsafe_wrap(Vector{UInt8}, pointer(text), sizeof(text)),
                text, index, version)
end
function ParseStream(text::AbstractString, index::Integer=1; version=VERSION)
    ParseStream(String(text), index; version=version)
end