make check-all failing with latest LLVM

[Quuxplusone]

Bugzilla Link	PR12196
Status	RESOLVED FIXED
Importance	P enhancement
Reported by	Brendan (bkirby@mips.com)
Reported on	2012-03-06 14:09:14 -0800
Last modified on	2012-03-07 17:54:38 -0800
Version	trunk
Hardware	PC Linux
CC	bob.wilson@apple.com, efriedma@quicinc.com, geek4civic@gmail.com, llvm-bugs@lists.llvm.org
Fixed by commit(s)
Attachments
Blocks
Blocked by
See also

I'm able to build both LLVM and Clang with the latest trunk. But when I run
make check-all, I'm seeing the following tests fail:
    Clang :: Index/complete-modules.m
    Clang :: Modules/auto-module-import.m
    Clang :: Modules/compiler_builtins.m
    Clang :: Modules/cstd.m
    Clang :: Modules/cycles.c
    Clang :: Modules/decldef.mm
    Clang :: Modules/diamond-pch.c
    Clang :: Modules/diamond.c
    Clang :: Modules/header-import.m
    Clang :: Modules/inferred-submodules.m
    Clang :: Modules/irgen.c
    Clang :: Modules/lookup.cpp
    Clang :: Modules/lookup.m
    Clang :: Modules/macros.c
    Clang :: Modules/method_pool.m
    Clang :: Modules/module-private.cpp
    Clang :: Modules/namespaces.cpp
    Clang :: Modules/normal-module-map.cpp
    Clang :: Modules/objc-categories.m
    Clang :: Modules/on-demand-build-warnings.m
    Clang :: Modules/on-demand-build.m
    Clang :: Modules/on-demand-macros.m
    Clang :: Modules/redecl-merge.m
    Clang :: Modules/redeclarations.m
    Clang :: Modules/requires.m
    Clang :: Modules/subframeworks.m
    Clang :: Modules/submodules-preprocess.cpp
    Clang :: Modules/submodules.cpp
    Clang :: Modules/submodules.m
    Clang :: Modules/wildcard-submodule-exports.cpp

[Quuxplusone]

According to the buildbots, make check-all should be passing on Linux. Can you give more details about your configuration?

[Quuxplusone]

Some of the language syntax recently changed with modules, which would break these tests if you aren't using a clean build. Is this a completely clean build?

[Quuxplusone]

I'm building on Ubuntu:
Ubuntu 10.04.3 LTS

I'm using the GCC that comes with it to compile LLVM:
gcc (Ubuntu 4.4.3-4ubuntu5) 4.4.3

I start from a clean slate every time.  So, I'm checking out the latest version
of both LLVM and Clang into source and source/tools/clang.  Then, my build
process is:
mkdir build
cd build
../source/configure --prefix=$(dirname $PWD)/install
make
make check-all

[Quuxplusone]

Fariborz and I have both seen these same problems on some of our internal branches. I was just about to look into it. I'll let you know if I get anywhere.

[Quuxplusone]

I can't reproduce it on trunk.

[Quuxplusone]

I'm testing against trunk now. I'll let you know what I find.

[Quuxplusone]

I've been investigating this a bit.  I randomly picked the Modules/requires.m
test and ran it in lldb.  It is failing for me when the sys::fs::unique_file
call in LockFileManager() fails to create the lock file.  I think that is
happening because the temporary directory for the module cache doesn't exist at
the time.

The lock file is being created at the beginning of the compileModule() function
in CompilerInstance.cpp.

The non-existent path within the module cache is being returned by this call:

    ModuleFileName = PP->getHeaderSearchInfo().getModuleFileName(Module);

That's in CompilerInstance::loadModule at about line 880.

[Quuxplusone]

Another problem that I noticed.  Later in that code, there is this:

    if (!ModuleFile) {
      getDiagnostics().Report(ModuleNameLoc,
                              BuildingModule? diag::err_module_not_built
                                            : diag::err_module_not_found)
        << ModuleName
        << SourceRange(ImportLoc, ModuleNameLoc);
      return 0;
    }

The err_module_not_built diagnostic is being created but is never reported.  I
just see:

2 errors generated.

[Quuxplusone]

The last problem is due to the use of -verify.  I should have noticed that.
Without -verify, it reports this:

requires.m:4:32: fatal error: could not build module 'DependsOnModule'
@__experimental_modules_import DependsOnModule.CXX; // expected-error{{module
'DependsOnModule.CXX' requires feature 'cplusplus'}}
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~
1 error generated.

It looks like there is code inside that unique_file function that is supposed
to create the directory.  I'll debug that next.

[Quuxplusone]

The unique_file() function has this code:

  // Try to open + create the file.
rety_open_create:
  int RandomFD = ::open(RandomPath.c_str(), O_RDWR | O_CREAT | O_EXCL, 0600);
  if (RandomFD == -1) {
    // If the file existed, try again, otherwise, error.
    if (errno == errc::file_exists)
      goto retry_random_path;
    // The path prefix doesn't exist.
    if (errno == errc::no_such_file_or_directory) {

It's failing because the hash value for the directory in the module cache is
much longer than in my other build, and errno is set to ENAMETOOLONG.  That
case isn't handled here.

[Quuxplusone]

I just built against trunk, and make check-all is still failing for me.  Here's
a link to the output of make check-all:
https://dmz-portal.mips.com/bb/builders/LLVM%20with%20direct%20object%20emitter%20and%20static/builds/158/steps/make%20check-all/logs/stdio/text

[Quuxplusone]

CompilerInvocation::getModuleHash appears to be the culprit, but I'm not sure yet what exactly is going wrong.

[Quuxplusone]

With my build from a git-svn repo of trunk, getClangFullRepositoryVersion returns an empty string.

With my build from an internal git branch, getClangFullRepositoryVersion returns a 200 character string. This is the build that fails. I believe the length of that string was exacerbated by svn r151910, which added separate info for clang and llvm repos.

That 200 character string is translated to 1600 bits in the Signature value in CompilerInvocation::getModuleHash(). That accounts for about 44 characters in the hash. That's only a partial explanation, though, since in the failing case, the hash is much longer than that.

[Quuxplusone]

I'm pulling from an internal git server as well that is sync'd up every night with the git-svn repository. So, I'm probably seeing the same thing as you. This had been working find until last Friday. Could this be from the r151910 revision in Clang? The commit message is "clang -v support for separate clang.git and llvm.git, patch by Andrew Trick."

[Quuxplusone]

Oops. My calculation was off. Each character in the hash consumes 5+ bits, not 36 bits. Thus, the revision info is sufficient to explain the problem. I'm going to look into computing the module signature as a big string and then hashing it to a fixed-length hash value for the directory name.

[Quuxplusone]

I decided not to attempt a real fix. As a temporary workaround, I've just removed the clang version info from the module hash. svn r152266.

I filed rdar://problem/11004569 to track a proper fix.