Encoding problems on win7 32/64 with hg backends

[GoogleCodeExporter]

What steps will reproduce the problem?
1. in _vimrc :
set encoding=utf-8
lang messages zh_CN.utf-8

2.intall mercurial-1.7.1 on win7 32/64, locale CH 中文(简体 中国)

3.use gvim.exe (7.3) to edit some source,
:VCSCommit
and call 
:VCSLog
:VCSReview

The chinese characters of vcslog and vcsreview is irrecognizable.
Maybe hg.exe is in locale chinese (cp936), vcscommand should convert it into 
the encoding which vim used internally(it's utf-8, here). Or it's just a bug of 
gvim 7.3 or hg.exe. I have changed the encoding of gvim into locale 
chinese(cp936), and some outputs of vcslog and vcsreview was correct, but the 
hg commit messages in chinese were still irrecognizable. Maybe hg store 
messages in utf-8, there need some conversion. It's better to set encoding of 
vim to utf-8.

What is the expected output? What do you see instead?
Correct locale characters.  Locale characters irrecognizable.

What version of the product are you using? On what operating system?
win7 32/64, both.

Please provide any additional information below.
The attach file is output of :VCSLog(left pic) and :VCSReview.
if I use this settings in _vimrc:
lang messages en
then the output of :VCSLog is correct english characters, but the chinese 
characters in the output of :VCSReview is still wrong.

Original issue reported on code.google.com by tux...@gmail.com on 18 Nov 2010 at 6:11

Attachments:

• screen.jpg

[GoogleCodeExporter]

It's a bug of communication between gvim7.3.exe and hg.exe. 

I use "hg serve" and web browser to check the repository and find the Chinese 
description of hg commits are irrecognizable too. Then I have some test like 
this:

Test 1:
In windows cmd.exe, 
hg ci -m "中文提交, commit in Chinese"
The command runs correctly, and check the repository with web browser, the 
Chinese description is ok too.

Test 2:
In gvim.exe,
:!hg ci -m "中文提交，commit in Chinese"
The command fails in gvim.exe because of encoding problems, and if I use 
vcscommand :VCSCommit, the commit will succeed and check the repository with 
web browser, the Chinese description is corrupt.

Maybe when the gvim7.3.exe and hg.exe communicates, there needs some encoding 
conversion between them.

Original comment by tux...@gmail.com on 18 Nov 2010 at 8:32

[GoogleCodeExporter]

Does this version of the plugin help?

http://repo.or.cz/w/vcscommand.git/snapshot/029f92d52fd5bb5d3f8fbbcfc0f32dca660a
a1e9.zip

Original comment by bob.hies...@gmail.com on 23 Feb 2011 at 5:11

[GoogleCodeExporter]

I have tested the fixed version and found it was still not work.

In windows cmd.exe:
     hg ci -m "中文, Chinese"
Everything is ok in the hg.exe, and in gvim.exe I used the command "\cl" to 
check the hg log. The Chinese word in the commit message is ok too.

But if I tested the command "\cc" in gvim.exe and used some Chinese words in 
the commit message, the gvim.exe reported the error message like this:

VCSCommand: Version control command failed: ... transaction abort! rollback 
completed abort: decoding near '(some irrecognizable words)...' : 'gbk' codec 
can't decode bytes in position 2-3: illegal multibyte sequence!

And more I have tested the command "\cr" in gvim.exe and found that the text in 
the HG Review window was still irrecognizable too.

Original comment by tux...@gmail.com on 24 Feb 2011 at 7:04

[GoogleCodeExporter]

I've described same problem here:

http://vim.1045645.n5.nabble.com/VCSCommand-and-file-encoding-td1179412.html

It's not "win7" or "hg" problem.
It's vcscommand and source code not in latin (russian, chinese, anything else).
vcscommand can handle only latin and nothing more.

Original comment by dtesle...@gmail.com on 19 Sep 2011 at 1:33

[GoogleCodeExporter]

Dmitri,

As apparently other users have no issue, vcscommand seems to handle more than 
latin.

Are commands other than commit working?

Please provide a complete description of the error, including files which 
trigger any issue, as I requested in that very thread.  Please all information 
regarding your particular configuration, including terminal encoding, file 
encoding, operating system, vim version, locale, and the VCS type, version, and 
locale.

Original comment by bob.hies...@gmail.com on 23 Sep 2011 at 2:55

[GoogleCodeExporter]

I have commit and everything else working. But way it works with 
anything but latin is buggy. That's what I mean by "can handle only latin".

Here's my example:
1) Latest gvim 7.3 with latest vcscommand (by the way I use vcscommand 
since long ago, more that three years for sure and all this time it works
that way so peculiar version number doesn't matter)
cat ~/.vimrc:
set enc=utf-8
set fenc=cp1251
set fencs=ucs-bom,utf-8,cp1251
...
2) Files I edit are in cp1251 and under cvs or git. 
3) Platforms I work on includes win32 and linux. I think it doesn't matter.

When I edit file in cp1251 it looks fine in vim.
When I issue VCSDiff on file text in diff window is garbled. If I issue
set enc=cp1251 text in diff buffer would become fine but text in any 
other buffer would become garbled.

Here's easy test case to show problem:
1) get vim with vcscommand and git
2) set this in ~/.vimrc
set enc=utf-8
set fenc=cp1251
set fencs=ucs-bom,utf-8,cp1251
set keymap=russian-jcukenwin
3) mkdir /tmp/test && cd /tmp/test && git init && touch test && git add test && 
git commit . -m test && vim test
4) enter a<C-6>;jgf<Esc>:set fenc=cp1251<cr>:w<cr>:VCSDiff<cr>
5) note that text in diff window on line 6 differs from what you have
in test file buffer

Original comment by dtesle...@gmail.com on 23 Sep 2011 at 7:28

[GoogleCodeExporter]

Setting:

set tenc=cp1251

fixes it for me, with that test case.  That is assuming your terminal displays 
cp1251 correctly outside of vim via:

cat test

Original comment by bob.hies...@gmail.com on 23 Sep 2011 at 8:53

[GoogleCodeExporter]

It looks like I'm running into a question of how to support people who are 
using conflicting configurations.  The patch I previously accepted attempts to 
translate based on 'tenc'.  In your case, I think it would work to translate 
based on 'fenc'.  This is, of course, not guaranteed to work, as the encoding 
of the file is not guaranteed to remain static throughout time.

I could change the current code to check whether tenc is set, or fenc is set, 
and apply the translation depending on which is set.  However, I don't have a 
feel for how people typically use the plugin.  I will probably try a patch like 
this and run it by some of the folks who saw issues previously.

I strongly recommend storing and accessing everything as utf-8 :)

Original comment by bob.hies...@gmail.com on 26 Sep 2011 at 2:59

[GoogleCodeExporter]

Dmitri,

  In your configuration, what encoding would you expect log messages to use?  UTF-8?

Original comment by bob.hies...@gmail.com on 26 Sep 2011 at 8:01

[GoogleCodeExporter]

Bob,
I've tried set tenc=cp1251 and didn't help much:
1) now text in vim and gvim looks different: gvim is ok but text in vim is 
garbled
2) it "solves" problem for vim: text in file buffer and vcs diff buffer looks 
the same. there's quotes because text is garbled in both buffers
3) for gvim it does nothing

> In your configuration, what encoding would you expect log messages to use?  
UTF-8?
I guess I expect utf-8 because I have "set enc=utf-8" in ~/.vimrc

Original comment by dtesle...@gmail.com on 27 Sep 2011 at 6:35

[GoogleCodeExporter]

Could you please try the version of vcscommand.vim at:

http://repo.or.cz/w/vcscommand.git/blob_plain/30b5f24fce3cd2ff3e70a9aed2335bd817
57c7ef:/plugin/vcscommand.vim

I try to work around mixed-mode setups like you describe.

Original comment by bob.hies...@gmail.com on 27 Sep 2011 at 3:21

[GoogleCodeExporter]

Thanks!!! It works!

Original comment by dtesle...@gmail.com on 28 Sep 2011 at 6:34

[GoogleCodeExporter]

Thank you for testing.  This will be released soon, as long as it doesn't break 
another tester's configuration.

Original comment by bob.hies...@gmail.com on 29 Sep 2011 at 7:18

[GoogleCodeExporter]

This change is released.

Original comment by bob.hies...@gmail.com on 3 Oct 2011 at 5:25

• Changed state: Fixed
• Added labels: Type-Enhancement
• Removed labels: Type-Defect