search

conan-io / conan

Conan - The open-source C and C++ package manager
https://conan.io
MIT License
8.17k stars 974 forks source link

CMake always rerun with Ninja generator #556

Closed piponazo closed 7 years ago

piponazo commented 7 years ago

Hi,

I am trying to use conan in our enterprise project and I am having problems when I use it in a CMake project in which I want to use the Ninja generator. This project has many external dependencies, and it has been working normally with any kind of generator (Make, Ninja, MSVC) before trying to use conan.

The first step we tried in our migration is to handle the boost dependency. When we use the Make generator everything works like a charm. However when I try to use the Ninja generator, every time I run the ninja command, CMake is rerunning the configuration many times.

I tried to run Ninja with "-d explain" to determine the origin of the problem. For some reason many of the cmake files are considered as dirty. 

# Configure project
mkdir build
conan install .. && cmake -GNinja ..

# Run ninja with debugging capabilities
luis@p4dDesktop:~/projects/pix4d/master-conan/pix4dmapper/build$ ninja -d explain
ninja explain: output /home/luis/.conan/data/zlib/1.2.8/lasote/stable/package/21ace02f4960dd0c1d50bd3abe1537054de08157/FindZLIB.cmake of phony edge with no inputs doesn't exist
ninja explain: /home/luis/.conan/data/Boost/1.60.0/piponazo/testing/package/ed2b408ce34ce36caef16f74181d3bc588210ba6/FindBoost.cmake is dirty
ninja explain: /home/luis/.conan/data/zlib/1.2.8/lasote/stable/package/21ace02f4960dd0c1d50bd3abe1537054de08157/FindZLIB.cmake is dirty
...
ninja explain: /home/luis/projects/Pix4DMapper-Master/master-conan/pix4dmapper/src/apps/CMakeLists.txt is dirty
# A bunch of other project cmake files
...
ninja explain: /usr/local/share/cmake-3.6/Modules/AutogenInfo.cmake.in is dirty
ninja explain: /usr/local/share/cmake-3.6/Modules/CMakeCCompiler.cmake.in is dirty
ninja explain: /usr/local/share/cmake-3.6/Modules/CMakeCCompilerABI.c is dirty
# Many other cmake files
...
ninja explain: CMakeCache.txt is dirty
ninja explain: CMakeFiles/3.6.2/CMakeCCompiler.cmake is dirty
ninja explain: CMakeFiles/3.6.2/CMakeCXXCompiler.cmake is dirty
ninja explain: CMakeFiles/3.6.2/CMakeSystem.cmake is dirty
ninja explain: CMakeFiles/feature_tests.c is dirty
ninja explain: CMakeFiles/feature_tests.cxx is dirty
ninja explain: conanbuildinfo.cmake is dirty
[0/1] Re-running CMake...

Note that when I remove the following lines from my CMakeLists.txt file, the ninja generator starts to work again:

#include(${CMAKE_BINARY_DIR}/conanbuildinfo.cmake)
#conan_basic_setup()

Any of you has experienced similar issues?

lasote commented 7 years ago

Could be an issue with the conan file's timestamps, but we need to debug it, could be other thing. No similar problems detected previously. Thanks for reporting.

memsharded commented 7 years ago

I have been trying for a while, Ubuntu 14, with dependencies to Boost & Poco. First, everything worked fine, I thought that it was because I was not using the find_package(Boost feature. I added it, and still the same, it build fast, not "dirty" files problem.

So unable to reproduce it yet :(

memsharded commented 7 years ago

As info, also using CMake 3.6 (from the conan cmake_installer/0.1@lasote/stable package). Maybe cmake issue?

Update: Tried with CMake 3.1, the same.

Using Ninja version 1.3.4 Indeed it could be an issue with the dates of the files, https://github.com/ninja-build/ninja/issues/1120, but don't know why it happens to you and not me.

piponazo commented 7 years ago

I will try later with different versions of ninja and CMake. Right now I am using the latest version of both (1.7.1 and 3.6.2 respectively)

piponazo commented 7 years ago

Interesting! I tried with the ninja package provided by Ubuntu 16.04 (version 1.5.1) and the situation has improved. Instead of rerunning CMake forever now it only do it once, and then the compilation starts. However, each time that I run ninja, is doing re-running cmake again (only once).

Then I tried with the latest version of ninja (from their github repository) and I fell into the same problem (re-running cmake forever).

What is strange for me is that this situation only happens if I include the conan file.

piponazo commented 7 years ago

@memsharded I tried with the same ninja version than you (1.3.4) and It is also re-running once cmake.

The strange thing is that I also tried to reproduce the same issue with a hello-world project and it is not happening :weary:

I will let you know if I discover the problem.

lasote commented 7 years ago

@piponazo Cannot reproduce the error either. :weary: I've uploaded a new conan installer recipe with virtualenv (ninja_installer/0.1@lasote/testing) to ease trying several versions but it works just fine. Could you paste the failing conanfile to see the dependency tree?

piponazo commented 7 years ago

Sure. I am just depending on my personal boost package. This is the conanfile:

[requires]
Boost/1.60.0@piponazo/testing

[generators]
cmake

[options]
Boost:shared=True
Boost:fPIC=True
Boost:with_date_time=True
Boost:with_filesystem=True
Boost:with_iostreams=True
Boost:with_program_options=True
Boost:with_serialization=True
Boost:with_system=True
Boost:with_thread=True

As I told before I tried to reproduce the issue in the same machine (office desktop) with a smaller project with no luck. However, yesterday I tried at home with other small project and I could reproduce it. I will try again in that machine this evening ;)

memsharded commented 7 years ago

When you say with a "smaller project", you mean not depending on your personal boost package? Thanks!

piponazo commented 7 years ago

No, I am sorry for the confusion. Out enterprise project has many software dependencies, and right now I only removed boost from our previous "package manager" and I started to use the boost package provided by conan.

Since this project is so big (many other dependencies, a lot of complex cmake stuff, and so on) I wonder if this issue was triggered by something else. I tried to reproduce the issue with a simpler hello-world project making use of boost. So in that smaller project, I use the same conanfile, then I include the cmake file generated with conan install and I included the boost folders and libraries in the same way that I do in the big project.

I could not spend much more time on this thing at work, but I will try to investigate it in my spare time :)

piponazo commented 7 years ago

I was able to reproduce the issue in my personal laptop.

Here you can find some "relevant?" system details:

H/W path        Device     Class          Description
=====================================================
                           system         W740SU (Not Applicable)
/0                         bus            W740SU
/0/0                       memory         64KiB BIOS
/0/14                      processor      Intel(R) Core(TM) i7-4760HQ CPU @ 2.10GHz
/0/14/15                   memory         1MiB L2 cache
/0/14/16                   memory         256KiB L1 cache
/0/14/17                   memory         6MiB L3 cache
/0/18                      memory         16GiB System Memory
/0/18/0                    memory         8GiB SODIMM DDR3 Synchronous 1600 MHz (0,6 ns)
...
/0/2/0.0.0      /dev/sdb   disk           240GB Crucial_CT240M50
/0/2/0.0.0/1    /dev/sdb1  volume         125GiB Windows NTFS volume
/0/2/0.0.0/3    /dev/sdb3  volume         97GiB Extended partition
/0/2/0.0.0/3/5  /dev/sdb5  volume         23GiB Linux filesystem partition
/0/2/0.0.0/3/6  /dev/sdb6  volume         74GiB Linux filesystem partition

# I am running the experiments on the /dev/sdb6 partition (SSD Disk)

This is exactly the sequence of commands that I run to reproduce the issue:

# Open a terminal
1) source env/env1/bin/activate # enter in the python environment where I installed conan
2) cd conanExample
3) mkdir build && cd build
4) conan install .. && cmake -GNinja ..
5) ninja

Conan config file:

[storage]
path: ~/.conan/data

[proxies]

[settings_defaults]
arch=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++
compiler.version=5.4

Resulting packages in the .conan/data/ directory: Boost bzip2 zlib.

If you need any other detail, please, do not hesitate to ask me.

memsharded commented 7 years ago

Yes, please, exact versions. 16.04 comes with gcc 5.4, but conan doesn't have binaries for some deps as bzip2. So: gcc: cmake: 3.6? Installed from? Ninja?

Thanks!

memsharded commented 7 years ago

Did a test with your repo and:

Still not able to reproduce :(

piponazo commented 7 years ago

I just updated the previous comment to include all the missing information you asked for ;)

memsharded commented 7 years ago

As a side observation, you might want to change your default libstdc++ in conan.conf to libstdc++11, which would be the default in Ubuntu 16 with gcc 5.4

memsharded commented 7 years ago

@piponazo Could you please list with details the contents of: /home/luis/.conan/data/zlib/1.2.8/lasote/stable/package/21ace02f4960dd0c1d50bd3abe1537054de08157/ ?

I am especially interested in FindZLib.cmake, could you please run stat on it? Thanks!

piponazo commented 7 years ago

aha! I think that you have caught the problem!

luis@luis-W740SU:~$ ls /home/luis/.conan/data/zlib/1.2.8/lasote/stable/package/21ace02f4960dd0c1d50bd3abe1537054de08157/ -lha
total 28K
drwxrwxr-x 4 luis luis 4,0K oct 11 22:39 .
drwxrwxr-x 3 luis luis 4,0K oct 11 22:39 ..
-rw-rw-r-- 1 luis luis  314 oct 11 22:39 conaninfo.txt
-rw-rw-r-- 1 luis luis 1,6K oct 11 22:39 conanmanifest.txt
-rw-rw-r-- 1 luis luis 2,1K ene  1  1970 FindZLIB.cmake
drwxrwxr-x 2 luis luis 4,0K oct 11 22:39 include
drwxrwxr-x 2 luis luis 4,0K oct 11 22:39 lib

luis@luis-W740SU:~$ stat /home/luis/.conan/data/zlib/1.2.8/lasote/stable/package/21ace02f4960dd0c1d50bd3abe1537054de08157/FindZLIB.cmake 
  File: '/home/luis/.conan/data/zlib/1.2.8/lasote/stable/package/21ace02f4960dd0c1d50bd3abe1537054de08157/FindZLIB.cmake'
  Size: 2093        Blocks: 8          IO Block: 4096   regular file
Device: 816h/2070d  Inode: 787762      Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/    luis)   Gid: ( 1000/    luis)
Access: 1970-01-01 01:00:00.000000000 +0100
Modify: 1970-01-01 01:00:00.000000000 +0100
Change: 2016-10-11 22:39:21.829907608 +0200
 Birth: -

Does this means that the FindZLIB.cmake file in the recipe has a bad date?

By the way, yes I know about the libstdc++11 thing (I read carefully the conan.io documentation hehe). I am using that in my desktop machine in the office but the file I sent you was generated in my personal laptop were I did not spend time configuring this file ;). Anyway thanks!

piponazo commented 7 years ago

Note that the file in the source folder already has that date:

luis@luis-W740SU:~$ ls /home/luis/.conan/data/zlib/1.2.8/lasote/stable/source/ -lha
total 44K
drwxrwxr-x  3 luis luis 4,0K oct 11 22:39 .
drwxrwxr-x  6 luis luis 4,0K oct 11 22:39 ..
-rw-rw-r--  1 luis luis  190 ene  1  1970 CMakeLists.txt
-rw-rw-r--  1 luis luis 3,2K oct 11 22:39 conan_export.tgz
-rw-rw-r--  1 luis luis 5,1K oct 11 22:39 conanfile.py
-rw-rw-r--  1 luis luis 4,8K oct 11 22:39 conanfile.pyc
-rw-rw-r--  1 luis luis  155 oct 11 22:39 conanmanifest.txt
-rw-rw-r--  1 luis luis 2,1K ene  1  1970 FindZLIB.cmake
drwxrwxr-x 14 luis luis 4,0K oct 11 22:39 zlib-1.2.8
memsharded commented 7 years ago

Yes, it is possible. The .tgz used to transfer files doesn't use file dates, so always compress to the same checksum. What it is weird is why it happens in your system and not in ours. Also, I have been trying to manually change the mtime date to the failing one both for Ninja 1.5 and 1.7, without success.

Also, still don't know why other files such as those of cmake are marked as dirty, definitely conan has nothing to do there, and it doesn't modify or touch them in any way.

In any case, a small step forward, lets keep working on it, and we will find that damn bug! :)

piponazo commented 7 years ago

At home I have two computers (desktop + laptop). I could reproduce the issue in both of them. In the laptop I run Ubuntu 16.04 and in the desktop Xubuntu 16.04. Both pcs have SSDs. At first I was suspecting about this and the fact that I use the option noatime when mounting the /home partition. So I tried to remove that option, remove all the packages from the conan/data folder and repeat the process. The result is the same.

I also tried to move the conan/data folder to partition mounted in a non-SDD hard-drive and repeat the previous steps. No luck here neither.

piponazo commented 7 years ago

I have been trying some things and there is something that I found very weird. We have some files in the source folders of conan having EPOCH in their dates. We don't know why but that's fine for the moment.

However if I take one of those files and I copy it to the tmp folder this is what happen:

# Checking that the original file is in EPOCH
ls -lh ~/.conan/data/Boost/1.60.0/lasote/stable/source/FindBoost.cmake 
-rw-r--r-- 1 luis luis 82K ene  1  1970 /home/luis/.conan/data/Boost/1.60.0/lasote/stable/source/FindBoost.cmake

# I copy that file to other place
cp ~/.conan/data/Boost/1.60.0/lasote/stable/source/FindBoost.cmake /tmp/

# I check its time in the new place
ls /tmp/FindBoost.cmake  -lh
-rw-r--r-- 1 luis luis 82K oct 15 18:37 /tmp/FindBoost.cmake

Why is not happening the same when conan is copying that file to the package folders? Is it a limitation in python? Is there any option to configure this?

When I compile the boost package I can find these lines in the log:

Boost/1.60.0@lasote/stable: Generating the package
Boost/1.60.0@lasote/stable: Package folder /home/luis/.conan/data/Boost/1.60.0/lasote/stable/package/8b111290ce72aaeabe5c7ccfbf008ab6905362aa
Boost/1.60.0@lasote/stable package(): Copied 136 '.h' files
Boost/1.60.0@lasote/stable package(): Copied 3 '.pl' files: gen_function_N.pl, gen_maybe_include.pl, gen_signal_N.pl
Boost/1.60.0@lasote/stable package(): Copied 3 '.m4' files: for.m4, pool_construct_simple.m4, pool_construct.m4
Boost/1.60.0@lasote/stable package(): Copied 1 '.cmake' files: FindBoost.cmake
...

But strangely, the FindBoost.cmake file still have the EPOCH time in the package folder.

memsharded commented 7 years ago

That is a very good investigation into the causes. I will also try a few things with your ideas, thanks!

memsharded commented 7 years ago

I have done a few further steps, I'd say we are very close:

I have tracked down the issues in this order:

Lets kill this bug!! :D :D

piponazo commented 7 years ago

I just tried the two things you proposed.

In the first case, I deleted my conan installation (I installed using the linux installer) and installed again using pythonenv + pip. Then I modified the file mentioned in #566 and I tried to recompile the packages again. Sadly, I am still falling in the same situation. The file FindBoost.cmake is still en epoch in the package folder.

However, in the second test, I did this:

luis@p4dDesktop:~/.conan/data/Boost/1.60.0/piponazo/testing/package/ed2b408ce34ce36caef16f74181d3bc588210ba6$ ls -lh
total 996K
-rw-rw-r-- 1 luis luis 1.1K Oct 17 09:08 conaninfo.txt
-rw-rw-r-- 1 luis luis 900K Oct 17 09:08 conanmanifest.txt
-rw-rw-r-- 1 luis luis  82K Jan  1  1970 FindBoost.cmake
drwxrwxr-x 3 luis luis 4.0K Oct 17 09:08 include
drwxrwxr-x 2 luis luis 4.0K Oct 17 09:08 lib
luis@p4dDesktop:~/.conan/data/Boost/1.60.0/piponazo/testing/package/ed2b408ce34ce36caef16f74181d3bc588210ba6$ python
Python 2.7.12 (default, Jul  1 2016, 15:12:24) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.utime("FindBoost.cmake", None)
>>> 
luis@p4dDesktop:~/.conan/data/Boost/1.60.0/piponazo/testing/package/ed2b408ce34ce36caef16f74181d3bc588210ba6$ ls -lh
total 996K
-rw-rw-r-- 1 luis luis 1.1K Oct 17 09:08 conaninfo.txt
-rw-rw-r-- 1 luis luis 900K Oct 17 09:08 conanmanifest.txt
-rw-rw-r-- 1 luis luis  82K Oct 17 09:13 FindBoost.cmake
drwxrwxr-x 3 luis luis 4.0K Oct 17 09:08 include
drwxrwxr-x 2 luis luis 4.0K Oct 17 09:08 lib

As you can see, it's updating the file correctly.

memsharded commented 7 years ago

Please, with the modified installation try to remove conan remove "*" -f your local cache, then install it again, then rebuild.

The touch is done at untargz time, that is done at install if not already installed.

memsharded commented 7 years ago

Another test you could do if that doesn't work is to insert a os.system("stat " + fname) before and after the os.utime, to check if it is changing the file mtime or not.

piponazo commented 7 years ago

I tried both things you mentioned and I ended up in the same situation. The FindBoost.cmake package in the package folder is still in epoch. That's very weird. Are you sure that you are calling that touch() function when the package is installed after its compilation?

memsharded commented 7 years ago

What was the output of putting such os.system before and after the os.time, did they execute showing the same output, or maybe they didn't execute? If you want to share screen, we may try to do a pair-programming debug session.

memsharded commented 7 years ago

I have another idea:

for a, b, c in os.walk("afolder"):
    print a, b, c

What does this prints if "afolder" does not exist? NOTHING. That could be the case for:

def get_package(self, package_reference, dest_folder, remote):
        """
        Read the conans package from remotes
        Will iterate the remotes to find the conans unless remote was specified

        returns (dict relative_filepath:abs_path , remote_name)"""
        zipped_files = self._call_remote(remote, "get_package", package_reference, dest_folder)
        files = unzip_and_get_files(zipped_files, dest_folder, PACKAGE_TGZ_NAME)
        # Issue #214 https://github.com/conan-io/conan/issues/214
        for dirname, _, files in os.walk(dest_folder):
            for fname in files:
                touch(os.path.join(dirname, fname))

        return files

It could be a case of fast execution, that the dest_folder is still not created, even if it has been commanded to create previously, so the iteration of os.walk does nothing.

memsharded commented 7 years ago

I know it is very difficult, as it is running a extract of the tgz before. What it is clear, is that it is important to monitor that touch is indeed being called.

memsharded commented 7 years ago

Fixed by #568. Will be released in conan 0.14

piponazo commented 7 years ago

Great! Thanks for the support guys ;)

memsharded commented 7 years ago

You are welcome! Better leave this issue open, we close them once they are released.