Closed terasakisatoshi closed 2 years ago
tokenize(Tokenizer(), "アイス") fails:
tokenize(Tokenizer(), "アイス")
_ _ _ _(_)_ | Documentation: https://docs.julialang.org (_) | (_) (_) | _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help. | | | | | | |/ _` | | | | |_| | | | (_| | | Version 1.7.0 (2021-11-30) _/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release |__/ | julia> versioninfo() Julia Version 1.7.0 Commit 3bf9d17731 (2021-11-30 12:12 UTC) Platform Info: OS: macOS (x86_64-apple-darwin19.5.0) CPU: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-12.0.1 (ORCJIT, skylake) Environment: JULIA_EDITOR = subl JULIA_PROJECT = @. julia> using Awabi julia> tokenize(Tokenizer(), "アイス") ERROR: UndefVarError: matrix not defined Stacktrace: [1] build_lattice(tokenizer::Tokenizer, sentence::String) @ Awabi ~/.julia/packages/Awabi/ec4gr/src/tokenizer.jl:93 [2] tokenize(tokenizer::Tokenizer, s::String) @ Awabi ~/.julia/packages/Awabi/ec4gr/src/tokenizer.jl:110 [3] top-level scope @ REPL[3]:1 julia>
I think this line causes our issue.
https://github.com/nakagami/Awabi.jl/blob/be7757f95bbc0f2febdbed98ac9989637df325da/src/tokenizer.jl#L93
tokenize(Tokenizer(), "アイス")
fails: