utf-8 decomposed characters in path to the working copy prevents tree view from working

GoogleCodeExporter commented 9 years ago

What steps will reproduce the problem?
1. Check out a working copy into a path with UTF-8 NFD character.
2. Open the WC with tree mode.

What is the expected output? What do you see instead?
The folders and files are expected to be shown.
Instead, svnX seems to be looking for something forever and the content will 
not be shown.
If the number of NFD character changes, the result also varies. Sometimes, the 
folders are partially shown.
Doesn't seem to affect the flat view mode.

What version of the product are you using? On what operating system?
SvnX 1.3.4
MacPorts subversion @1.8.8_0+unicode_path
Tried on two Macs; one running Mavericks, the other Lion.

Please provide any additional information below.
Thank you for the patch you provided to subversion, and eventually to MacPorts, 
for svn 1.7.
http://subversion.tigris.org/issues/show_bug.cgi?id=2464

MacPorts got a contribution to make the patch work on 1.8, but seems to need 
more fix to be used with svnX.

related to issue:152

Regards,
Takeshi

Original issue reported on code.google.com by kim.take...@gmail.com on 3 May 2014 at 6:27

GoogleCodeExporter commented 9 years ago

I found a message in the Console.

14/05/04 8:25:15.521 svnX: *** -[NSPathStore2 substringFromIndex:]: Range or 
index out of bounds
14/05/04 8:25:15.522 svnX: (
    0   CoreFoundation                      0x90fafa67 __raiseError + 231
    1   libobjc.A.dylib                     0x97c7c149 objc_exception_throw + 155
    2   CoreFoundation                      0x90f17289 +[NSException raise:format:arguments:] + 137
    3   CoreFoundation                      0x90f171f9 +[NSException raise:format:] + 57
    4   Foundation                          0x9968706c -[NSString substringFromIndex:] + 111
    5   svnX                                0x0001d59b svnX + 116123
    6   svnX                                0x0001c398 svnX + 111512
    7   CoreFoundation                      0x90f07a9d __invoking___ + 29
    8   CoreFoundation                      0x90f079d9 -[NSInvocation invoke] + 137
    9   svnX                                0x00024999 svnX + 145817
    10  svnX                                0x0002346a svnX + 140394
    11  svnX                                0x000234ad svnX + 140461
    12  Foundation                          0x99656df1 __-[NSNotificationCenter addObserver:selector:name:object:]_block_invoke_1 + 49
    13  CoreFoundation                      0x90eef903 ___CFXNotificationPost_block_invoke_1 + 275
    14  CoreFoundation                      0x90eba688 _CFXNotificationPost + 2776
    15  Foundation                          0x99641fde -[NSNotificationCenter postNotificationName:object:userInfo:] + 92
    16  Foundation                          0x9971e99a _performFileHandleSource + 1159
    17  CoreFoundation                      0x90e7c13f __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 15
    18  CoreFoundation                      0x90e7baf6 __CFRunLoopDoSources0 + 246
    19  CoreFoundation                      0x90ea59c8 __CFRunLoopRun + 1112
    20  CoreFoundation                      0x90ea51dc CFRunLoopRunSpecific + 332
    21  CoreFoundation                      0x90ea5088 CFRunLoopRunInMode + 120
    22  HIToolbox                           0x9ad0e543 RunCurrentEventLoopInMode + 318
    23  HIToolbox                           0x9ad158ab ReceiveNextEventCommon + 381
    24  HIToolbox                           0x9ad1571a BlockUntilNextEventMatchingListInMode + 88
    25  AppKit                              0x9145eee8 _DPSNextEvent + 678
    26  AppKit                              0x9145e752 -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] + 113
    27  AppKit                              0x9145aac1 -[NSApplication run] + 911
    28  AppKit                              0x916ebac5 NSApplicationMain + 1054
    29  svnX                                0x00002296 svnX + 4758
    30  svnX                                0x000021bd svnX + 4541
)

Original comment by kim.take...@gmail.com on 3 May 2014 at 11:26

GoogleCodeExporter commented 9 years ago

Hello,

The old patch was not written for Subversion 1.8.x.
The modification to it appears not to work correctly even when doing a simple 
`svn st`.
[I see duplicated items with NFC characters, one marked as missing & one as 
unversioned.]

I’ve written and tested a new patch.  [Attached.]
It’s for Subversion 1.8.9.
NOTE: It requires that NFC characters be stored in the WCs database for correct 
operation.
This means you MUST checkout a new WC once this patch has been applied.
[If included in a future release of Subversion then the `upgrade` command could 
be extended to fix the database.]

I hope this will fix your problem.

Please test the patch and let me know how you get on.

Regards,

CHRIS

Original comment by chris...@gmail.com on 21 May 2014 at 12:41

Attachments:

subversion-1.8.9-osx_precompose.patch

GoogleCodeExporter commented 9 years ago

Thank you for the patch!

Unfortunately, some test didn't pass with the build.
I have attached the tests.log.

I replaced the macports patch file and made the build.

The build from original source code also shows similar (if not identical) 
result.
It might be linking to some macports libraries, but i am not sure if that is 
the cause.

Tried on Mavericks and Lion.

Original comment by kim.take...@gmail.com on 27 May 2014 at 10:38

Attachments:

tests.log

GoogleCodeExporter commented 9 years ago

I just reran `make check` (from the 1.8.9 obj dir).  I get:
    Summary of test results:
      1921 tests PASSED
      57 tests SKIPPED
      32 tests XFAILED (1 WORK-IN-PROGRESS)
    SUMMARY: All tests successful.

This looks good to me.

All your FAILs appear to be crashes.  Are you sure the patch applied cleanly?  
Are you using any other svn patches?
Please zip & attach the (5) patched files.

I’m running on OSX 10.8.5.  I built Subversion 1.8.9 with SQLite 3.8.4.3, 
serf 1.3.5 and the following config:
    configure --prefix=/opt/subversion_1.8.9 --target=i386-apple-darwin9 --disable-static --disable-javahl \
              --disable-mod-activation --without-berkeley-db --without-jdk --without-swig --without-apxs \
              --with-serf=<path-to-obj-dir>  --with-sysroot=/Developer/SDKs/MacOSX10.5.sdk \
              "CFLAGS=-mmacosx-version-min=10.5 -march=i386 -DSVN_SQLITE_INLINE -Os"
All built with GCC 4.2.1.

> The build from original source code also shows similar (if not identical) 
result.

If you are seeing failures without my patch then it is likely that you have 
some other problem.

Original comment by chris...@gmail.com on 27 May 2014 at 3:22

GoogleCodeExporter commented 9 years ago

Attached is the files patched, and additionally, files altered by macports or 
configure.

SQLite and serf are installed by macports.
The following ports are currently installed:
  serf1 @1.3.4_0 (active)
  sqlite3 @3.8.4.3_0 (active)

According to the config.log;
It was created by subversion configure 1.8.9, which was
generated by GNU Autoconf 2.69.  Invocation command line was

  $ ./configure --prefix=/opt/local --with-berkeley-db=:/opt/local/include/db46:/opt/local/lib/db46:db-4.6 --with-apr=/opt/local/bin/apr-1-config --with-apr-util=/opt/local/bin/apu-1-config --without-apxs --mandir=${prefix}/share/man --with-serf=/opt/local --with-sasl=/opt/local --with-libmagic=/opt/local --without-gnome-keyring

and the compiler is;
configure:2755: checking for gcc
configure:2782: result: /usr/bin/clang
configure:3011: checking for C compiler version
configure:3020: /usr/bin/clang --version >&5
Apple LLVM version 5.1 (clang-503.0.40) (based on LLVM 3.4svn)
Target: x86_64-apple-darwin13.2.0
Thread model: posix

Without the patch, there are two test failed.

FAIL:  svnlook_tests.py 12: test 'svnlook * -t'
FAIL:  trans_tests.py 1: commit new files with keywords active from birth
Summary of test results:
  1944 tests PASSED
  57 tests SKIPPED
  32 tests XFAILED (1 WORK-IN-PROGRESS)
  2 tests FAILED
SUMMARY: Some tests failed.

Original comment by kim.take...@gmail.com on 27 May 2014 at 5:43

Attachments:

svn.zip

GoogleCodeExporter commented 9 years ago

The patched files all look correct.
Your config is quite different from mine.
Notably yours has: ENABLE_NLS=1, HAVE_BIND_TEXTDOMAIN_CODESET=1, 
SVN_HAS_ATOMIC_BUILTINS=1
Mine has SVN_SQLITE_INLINE=1

I don’t think that your config will use the correct sqlite3.
Which sqlite3 does `otool -L /opt/local/bin/svn | grep sql` report?
[If it’s /usr/lib/libsqlite3.dylib then it’s probably not what you want.]

Assuming it’s not a clang problem, I suggest you try 
`--target=i386-apple-darwin9`, `--without-berkeley-db`, `"CFLAGS=-march=i386 
-DSVN_SQLITE_INLINE"` and an appropriate sqlite setting/amalgamation/lib.

> Without the patch, there are two test failed.
> FAIL:  svnlook_tests.py 12: test 'svnlook * -t'
> FAIL:  trans_tests.py 1: commit new files with keywords active from birth

I checked my tests.log and both those tests passed.

The bottom line is that it should be possible for you to build/configure a 
working version and then change settings one-by-one until you find where the 
problem is.

Original comment by chris...@gmail.com on 27 May 2014 at 10:56

GoogleCodeExporter commented 9 years ago

Thank you for encouraging!

`otool -L svn | grep sql` returned
    /opt/local/lib/libsqlite3.0.dylib (compatibility version 9.0.0, current version 9.6.0)

Which matches what's expected.

Now I have to find some time to try the various configurations.

Original comment by kim.take...@gmail.com on 27 May 2014 at 11:42

GoogleCodeExporter commented 9 years ago

OK.  [I wasn’t expecting it to find that sqlite without a `--with-sqlite=…` 
config.]

Actually, `/opt/local/bin/svn` is not used by `make check`.  [And it’s 
unrelated unless you ran `make install`.]
Make check uses `./subversion/svn/.libs/svn`.

Another thought - on my machine `locale` reports "en_GB.UTF-8" for all vars.
It’s conceivable that a non-English setting could cause test failures.

After that I’d suggest that you try an i386 build (rather than x84_64) next.

Original comment by chris...@gmail.com on 28 May 2014 at 1:45

GoogleCodeExporter commented 9 years ago

Now I start understanding how `make check` works with subversion project.

Working with macports working file starts to get confusing, so I tried with the 
original source codes.

I made no flags for configure and there were no error with testing.
Summary of test results:
  1945 tests PASSED
  58 tests SKIPPED
  32 tests XFAILED (1 WORK-IN-PROGRESS)
SUMMARY: All tests successful.

And with the patch, I tried;
  ./configure --target=i386-apple-darwin9 --disable-static --disable-javahl --disable-mod-activation --without-berkeley-db --without-jdk --without-swig --without-apxs

with CFLAGS set;
DEV-KIM-PRO-2:subversion-1.8.9-new kim$ set
CFLAGS='-mmacosx-version-min=10.5 -march=i386 -DSVN_SQLITE_INLINE -Os'

but x86_64 objects seems to be built:
DEV-KIM-PRO-2:subversion-1.8.9-new kim$ lipo -info ./subversion/svn/.libs/svn
Non-fat file: ./subversion/svn/.libs/svn is architecture: x86_64

And the tests are crashing.
Maybe a call to svn_dirent_join is crashing.

Original comment by kim.take...@gmail.com on 29 May 2014 at 10:20

GoogleCodeExporter commented 9 years ago

The `CFLAGS=…` should be an option passed to `./configure`.
You probably also need to execute `make clean` before executing `make` again.

However, if you have a build that passes all the tests then you could simply 
apply the patch & try `make && make check`.

Original comment by chris...@gmail.com on 29 May 2014 at 2:27

GoogleCodeExporter commented 9 years ago

The previous result (crashes) were the result with the patch applied to the 
'test passed' files.
I am surprised that the patch makes such a difference. I still can't find out 
what I am doing wrong.

I have set the CFLAGS as environmental values since ./configure complained 
about it.
In the configure.log it says;
configure:2782: result: gcc
configure:3011: checking for C compiler version
configure:3020: gcc --version >&5
Apple LLVM version 5.1 (clang-503.0.40) (based on LLVM 3.4svn)
Target: x86_64-apple-darwin13.2.0
Thread model: posix
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr 
--with-gxx-include-dir=/usr/include/c++/4.2.1
configure:3031: $? = 0
configure:3020: gcc -v >&5
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr 
--with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 5.1 (clang-503.0.40) (based on LLVM 3.4svn)
Target: x86_64-apple-darwin13.2.0
Thread model: posix
configure:3031: $? = 0
configure:3020: gcc -V >&5
clang: error: argument to '-V' is missing (expected 1 value)
clang: error: no input files
configure:3031: $? = 1
configure:3020: gcc -qversion >&5
clang: error: unknown argument: '-qversion' 
[-Wunused-command-line-argument-hard-error-in-future]
clang: note: this will be a hard error (cannot be downgraded to a warning) in 
the future
clang: error: no input files
configure:3031: $? = 1
configure:3051: checking whether the C compiler works
configure:3073: gcc -mmacosx-version-min=10.5 -march=i386 -DSVN_SQLITE_INLINE 
-Os   conftest.c  >&5
error: unknown target CPU 'i386'
configure:3077: $? = 1
configure:3115: result: no
configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME "subversion"
| #define PACKAGE_TARNAME "subversion"
....

Maybe Xcode 5.1 is not compatible with i386?

I have also tried using lldb for the first time, and if I am doing right, it is 
showing the Sqlite library from macports distribution linked can be a problem;
(lldb) 
Process 61737 stopped
* thread #1: tid = 0x4f8bf7, 0x0000000100002e94 
client-test`create_greek_repos(repos_url=0x00007fff5fbff4f8, 
name=0x0000000100003aa8, opts=<unavailable>, pool=0x0000000101006028) + 68 at 
client-test.c:63, queue = 'com.apple.main-thread', stop reason = step out
Return value: (svn_error_t *) $169 = 0x0000000000000000

    frame #0: 0x0000000100002e94 client-test`create_greek_repos(repos_url=0x00007fff5fbff4f8, name=0x0000000100003aa8, opts=<unavailable>, pool=0x0000000101006028) + 68 at client-test.c:63
   60     SVN_ERR(svn_test__create_repos(&repos, name, opts, pool));
   61   
   62     /* Prepare and commit a txn containing the Greek tree. */
-> 63     SVN_ERR(svn_fs_begin_txn2(&txn, svn_repos_fs(repos), 0 /* rev */,
   64                               0 /* flags */, pool));
   65     SVN_ERR(svn_fs_txn_root(&txn_root, txn, pool));
   66     SVN_ERR(svn_test__create_greek_tree(txn_root, pool));
(lldb) 
Process 61737 stopped
* thread #1: tid = 0x4f8bf7, 0x00000001002de029 
libsqlite3.0.dylib`sqlite3VdbeMemSetStr + 89, queue = 'com.apple.main-thread', 
stop reason = EXC_BAD_ACCESS (code=1, address=0x1021330)
    frame #0: 0x00000001002de029 libsqlite3.0.dylib`sqlite3VdbeMemSetStr + 89
libsqlite3.0.dylib`sqlite3VdbeMemSetStr + 89:
-> 0x1002de029:  cmpb   $0x0, (%rax)
   0x1002de02c:  je     0x1002de038               ; sqlite3VdbeMemSetStr + 104
   0x1002de02e:  incl   %ebx
   0x1002de030:  incq   %rax

I might loose connection to the computer for a while, but try to catch up.

Original comment by kim.take...@gmail.com on 30 May 2014 at 12:35

GoogleCodeExporter commented 9 years ago

I am afraid I am start making some confusion here.
I should get more time to settle down and tackle the issue more 
systematically...

Original comment by kim.take...@gmail.com on 30 May 2014 at 12:51

spricity / svnx

utf-8 decomposed characters in path to the working copy prevents tree view from working #211