Multibyte characters in title and journal name

[hongyuanjia]

Thanks for this great plugin!

citation.vim works seamlessly on English items, but failed to display titles or journal names if there are multibyte characters. Please below. Also note that the citation keys do not have this issue. BTW, I am using the Zotero source.

Vim version info

``` VIM - Vi IMproved 8.1 (2018 May 18, compiled Aug 25 2018 18:34:30) MS-Windows 64-bit GUI version with OLE support Included patches: 1-327 Compiled by stop_@LAPTOP-E7OHISR4 Huge version with GUI. Features included (+) or not (-): +acl +byte_offset +conceal -dnd +float +keymap +menu +num64 +python/dyn +startuptime +terminal +vertsplit +windows +arabic +channel +cryptv -ebcdic +folding +lambda +mksession +ole +python3/dyn +statusline -tgetent +virtualedit +writebackup +autocmd +cindent +cscope +emacs_tags -footer +langmap +modify_fname +packages +quickfix -sun_workshop -termresponse +visual -xfontset +autochdir +clientserver +cursorbind +eval +gettext/dyn +libcall +mouse +path_extra +reltime +syntax +textobjects +visualextra -xim +autoservername +clipboard +cursorshape +ex_extra -hangul_input +linebreak +mouseshape +perl/dyn +rightleft +tag_binary +timers +viminfo +xpm_w32 +balloon_eval +cmdline_compl +dialog_con_gui +extra_search +iconv/dyn +lispindent +multi_byte_ime/dyn +persistent_undo +ruby/dyn +tag_old_static +title +vreplace -xterm_save -balloon_eval_term +cmdline_hist +diff +farsi +insert_expand +listcmds +multi_lang -postscript +scrollbind -tag_any_white +toolbar -vtp +browse +cmdline_info +digraphs +file_in_path +job +localmap +mzscheme/dyn +printer +signs +tcl/dyn +user_commands +wildignore ++builtin_terms +comments +directx +find_in_path +jumplist +lua/dyn +netbeans_intg +profile +smartindent -termguicolors +vartabs +wildmenu system vimrc file: "$VIM\vimrc" user vimrc file: "$HOME\_vimrc" 2nd user vimrc file: "$HOME\vimfiles\vimrc" 3rd user vimrc file: "$VIM\_vimrc" user exrc file: "$HOME\_exrc" 2nd user exrc file: "$VIM\_exrc" system gvimrc file: "$VIM\gvimrc" user gvimrc file: "$HOME\_gvimrc" 2nd user gvimrc file: "$HOME\vimfiles\gvimrc" 3rd user gvimrc file: "$VIM\_gvimrc" defaults file: "$VIMRUNTIME\defaults.vim" system menu file: "$VIMRUNTIME\menu.vim" Compilation: cl -c /W3 /nologo -I. -Iproto -DHAVE_PATHDEF -DWIN32 -DFEAT_CSCOPE -DFEAT_TERMINAL -DFEAT_NETBEANS_INTG -DFEAT_JOB_CHANNEL -DFEAT_XPM_W32 -DWINVER=0x0501 -D_WIN32_WINNT=0x0501 /MP -DHAVE_STDINT_H /Ox /GL -DNDEBUG /MD -DFEAT_OLE -DFEAT_MBYTE_IME -DDYNAMIC_IME -DGLOBAL_IME -DFEAT_MBYTE -DFEAT_GUI_W32 -DFEAT_DIRECTX -DDYNAMIC_DIRECTX -DFEAT_DIRECTX_COLOR_EMOJI -DDYNAMIC_ICONV -DDYNAMIC_GETTEXT -DFEAT_TCL -DDYNAMIC_TCL -DDYNAMIC_TCL_DLL=\"tcl86.dll\" -DDYNAMIC_TCL_VER=\"8.6\" -DFEAT_LUA -DDYNAMIC_LUA -DDYNAMIC_LUA_DLL=\"lua53.dll\" -DFEAT_PYTHON -DDYNAMIC_PYTHON -DDYNAMIC_PYTHON_DLL=\"python27.dll\" -DFEAT_PYTHON3 -DDYNAMIC_PYTHON3 -DDYNAMIC_PYTHON3_DLL=\"python37.dll\" -DFEAT_MZSCHEME -I "..\..\libs\racket-6.12\include" -DDYNAMIC_MZSCHEME -DDYNAMIC_MZSCH_DLL=\"libracketxxxxxxx.dll\" -DDYNAMIC_MZGC_DLL=\"libmzgcxxxxxxx.dll\" -DFEAT_PERL -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DDYNAMIC_PERL -DDYNAMIC_PERL_DLL=\"perl528.dll\" -DFEAT_RUBY -DDYNAMIC_RUBY -DDYNAMIC_RUBY_VER=25 -DDYNAMIC_RUBY_DLL=\"x64-msvcrt-ruby250.dll\" -DFEAT_HUGE /Fd.\ObjGXOULYHTRZAMD64/ /Zi Linking: link /nologo /subsystem:windows /opt:ref /LTCG:STATUS /HIGHENTROPYVA:NO oldnames.lib kernel32.lib advapi32.lib shell32.lib gdi32.lib comdlg32.lib ole32.lib netapi32.lib uuid.lib /machine:AMD64 gdi32.lib version.lib winspool.lib comctl32.lib advapi32.lib shell32.lib netapi32.lib /machine:AMD64 msvcrt.lib oleaut32.lib user32.lib /nodefaultlib:lua53.lib /STACK:8388608 /nodefaultlib:python27.lib /nodefaultlib:python37.lib "C:\ActiveTcl\lib\tclstub86.lib" WSock32.lib .\xpm\x64\lib-vc14\libXpm.lib /PDB:gvim.pdb -debug ```

My Zotero version: 5.0.58

Thanks!

[rafaqz]

Thanks for the detailed error report.

This plugin does need a serious UTF8 overhaul, see also #20 . But my citations are 99.99% english language so I rarely experience any of these problems, and I've become very busy writing other code in another language and jumping back into python/vimscript takes me some time. Honestly it's unlikely I will fix this in the next 8 months (when my current thesis is finished), so if you feel like having a go at fixing it that would be the best way of getting it done soon.

[hongyuanjia]

Thanks, @rafaqz

Seems like it's time for some pythoning...

[hongyuanjia]

I know only little python.

For a dirty workaround, I just modified parser.py to directly return string without any cleaning.

i.e. changing below

https://github.com/rafaqz/citation.vim/blob/918bd039b058b6cf94e0f2a59d3ea47139c2f47c/python/citation_vim/zotero/parser.py#L71

to

return string

At least it works for me. I hope I could find a better way when I get to know more about python.

Thanks for this great plugin again.

[rafaqz]

It will break if you have {} or some other chars in the string anywhere. But that's a bad reason to strip non english chars... I'll look at a fix for it.

[rafaqz]

Try the latest update. It was doing some heavy handed character stripping that mostly wasn't needed.

[hongyuanjia]

Thanks a lot @rafaqz ! I can confirm that the latest version can display multibyte characters correctly.

[rafaqz]

Great! Thanks for finding the source of the problem, that made it a really obvious fix.