conanmanifest.txt not updated with conan install --update

piponazo commented 7 years ago

Hi all,

I recently discovered something I think it could be a bug. In order to illustrate the case, let's imagine that I have a project with 3 dependencies (X, Y, Z).

I was checking the functionality of the --update argument in the conan install command and I ran the following commands for testing it:

clone the project that depends on X, Y, Z. Run conan install without the --update argument. Conan reports that all my packages are "Already installed".
Go to the repository of the conan recipe for X and run conan test_package (the manifest file for that package is updated at this point replacing the timestamp).
Go to the project and run again conan install. Conan reports that all my packages are "Already installed".
Go to the project and run conan install --update. Now for the package X I get: "WARN: Current package is newer than remote upstream one".
Edit manually the conanmanifest.txt file for the package X and replace the timestamp with an older date.
Go to the project and run conan install --update. Now for the package X I get:

X: WARN: Current package is older than remote upstream one
X: WARN: Removing it to retrieve or build an updated one
X: Looking for package b1c1bba64e5b990636f4bcca97c3f5313c24bc3c in remote 'conan-server' 
Downloading conanmanifest.txt
[==================================================]                  
Downloading conaninfo.txt
[==================================================]                  
Downloading conan_package.tgz
[==================================================]                  
X: Package installed b1c1bba64e5b990636f4bcca97c3f5313c24bc3c

Everything is fine up to this point. However when I take a look to the conanmanifest.txt file that I modified , I still see the date I modified in the step 5, instead of the one that comes from the server.

Is this a bug or there is something I am missing ?

memsharded commented 7 years ago

Yes, seems a bug. Label as that, and will have a look. Thanks for such detailed report!

memsharded commented 7 years ago

Hi @piponazo ,

I am checking this issue, but I have just realized I don't understand at all what you are trying to accomplish:

You have created a new binary with conan test_package. The local package is then newer than the one in the remote
When you try to update it with --update it will refuse, printing an error and saying that that local is newer.
It also says, if you want to get the remote older one, first remove the local one.

All this is a protection to avoid accidental overwriting of newer binaries with old ones from remotes. So what you are doing manually editing the manifests, is not clear to me. They should never be edited manually. Can you please tell me a bit more about this, please?

memsharded commented 7 years ago

Hi again @piponazo ,

I have add a test with the process you describe, you can read it in #958, but it doesn't fail as you describe. Could you please check it?

In any case, I am still interested in my previous question, understanding what you tried to do.

Thanks very much!

piponazo commented 7 years ago

Hi @memsharded,

First of all, thanks you for taking a look to this issue. I was doing these experiments because we are currently having some issues in our Continuous Integration system and I was trying to understand what could be happening. I am not in the office right now and I cannot paste here the detailed information, I will try to give you report the logs tomorrow.

The main problem we are having is that we compile a package in a CI node and when we try to upload it to our private conan server we get something like this:

10:57:18 Conan wrapper: running conan: upload new packages to the conan-server
10:57:18 ERROR: Remote recipe is newer than local recipe: 
10:57:19  Remote date: 1486018746
10:57:19  Local date: 1484319002
10:57:19 Traceback (most recent call last):
10:57:19   File "J:/jenkins/workspace/master-debug/compiler/msvc/label/win/mode/debug/pix4dmapper/dev_tools/jenkins/pix4dmapper/build.d/04_conan.py", line 124, in <module>
10:57:19     main()
10:57:19   File "J:/jenkins/workspace/master-debug/compiler/msvc/label/win/mode/debug/pix4dmapper/dev_tools/jenkins/pix4dmapper/build.d/04_conan.py", line 118, in main
10:57:19     conan_commands()
10:57:19   File "J:/jenkins/workspace/master-debug/compiler/msvc/label/win/mode/debug/pix4dmapper/dev_tools/jenkins/pix4dmapper/build.d/04_conan.py", line 97, in conan_commands
10:57:19     check_call(['conan', 'upload', '-r', 'conan-server', req, '--all'])
10:57:19   File "C:\Python27\lib\subprocess.py", line 540, in check_call
10:57:19     raise CalledProcessError(retcode, cmd)
10:57:19 subprocess.CalledProcessError: Command '['conan', 'upload', '-r', 'conan-server', 'Mustache/1.0.1-0@pix4d/stable', '--all']' returned non-zero exit status 1

That is why I started to try crazy things with the --update argument and the timestamps inside the manifest files.

I hope this helps you to understand why I was doing such crazy things :P. I will take a look to the test code and continue with the investigation on my side.

memsharded commented 7 years ago

I see. But that is a different problem I think. It was complaining about the remote having a newer date than the local build (could happen if the remote and the CI slaves not datetime synced, for example). I think you can just use the --force argument to make sure your build slaves really upload the packages to the remote. Could that be a solution? Thanks for your feedback! (no hurries :)

piponazo commented 7 years ago

We did not want to use the --force option because we already have many dependencies and it takes some time to upload all the packages to the server. Take into account that in our organisation we are more than 25 c++ developers, working on feature branches, and every second/minute in our CI system is very valuable :wink: .

I can give you more details now about the problematic case. We were running a python script in our CI that has this logic for the conan usage:

call conan install $PROJECT_PATH --update -r conan-server for downloading all the available packages from our conan server. If this command fails, it means we need to compile a package for the current configuration (OS, compiler, build_type).
call the previous command appending --build missing. In that we we compile the missing packages for the current configuration.
Parse the generated conaninfo.txt file (that tell us all the dependencies of the project) and run conan upload -r conan-server $DEPENDENCY --all. This should upload the new generated packages to the server.

However this script we causing problems from time to time. I will copy & paste the jenkins log we obtained in one of those cases. Note that I have added some comments in the log (#) and skipped some non-interesting parts (...)

# Checking project dependencies (Note that Mustache is already installed)
Conan wrapper: running conan: verify if we need dependencies. An ERROR from the execution of conan is expected.
10:55:59 Exiv2/0.25-1@pix4d/stable: Not found, retrieving from server 'conan-server' 
10:55:59 Exiv2/0.25-1@pix4d/stable: Trying with 'conan-server'...
10:55:59 Downloading conanmanifest.txt
10:55:59 
10:55:59 Downloading conanfile.py
10:55:59 
10:55:59 Downloading conan_export.tgz
10:55:59 
...
10:55:59 Requirements
10:55:59     Archive/3.1.2-0@pix4d/stable from conan-server
10:55:59     CGAL/4.9-0@pix4d/stable from conan-server
10:55:59     Ceres/1.11.0-0@pix4d/stable from conan-server
10:55:59     Exiv2/0.25-1@pix4d/stable from conan-server
10:55:59     Expat/2.2.0-0@pix4d/stable from conan-server
10:55:59     GDAL/2.1.1-0@pix4d/stable from conan-server
10:55:59     LibSURE/1.3.0.490-0@pix4d/stable from conan-server
10:55:59     Libastro/3.7.7-0@pix4d/stable from conan-server
10:55:59     Liblas/1.8.1-1@pix4d/stable from conan-server
10:55:59     LightGBM/0.0.1-1@pix4d/stable from conan-server
10:55:59     Mustache/1.0.1-0@pix4d/stable from conan-server
10:55:59     OpenMesh/4.1.1-0@pix4d/stable from conan-server
10:55:59     OpenSceneGraph/3.4.0-0@pix4d/stable from conan-server
10:55:59     proj/4.9.2-0@pix4d/testing from conan-server
10:55:59     zlib/1.2.8@pix4d/stable from conan-server
10:55:59 Packages
10:55:59     Archive/3.1.2-0@pix4d/stable:9024ea81a84d4c512108194627f750ed0bdf8225
10:55:59     CGAL/4.9-0@pix4d/stable:2c0843cc59ff2d07e33c808b3398bc624e6b54e4
10:55:59     Ceres/1.11.0-0@pix4d/stable:2c0843cc59ff2d07e33c808b3398bc624e6b54e4
10:55:59     Exiv2/0.25-1@pix4d/stable:cd372d94b3acd012fbbed238d2ec573fb8edaf11
10:55:59     Expat/2.2.0-0@pix4d/stable:2c0843cc59ff2d07e33c808b3398bc624e6b54e4
10:55:59     GDAL/2.1.1-0@pix4d/stable:dec3fd3faf7b1ff625c6b6fd19737a1603b160ea
10:55:59     LibSURE/1.3.0.490-0@pix4d/stable:2c0843cc59ff2d07e33c808b3398bc624e6b54e4
10:55:59     Libastro/3.7.7-0@pix4d/stable:2c0843cc59ff2d07e33c808b3398bc624e6b54e4
10:55:59     Liblas/1.8.1-1@pix4d/stable:09170587ecd626998070c1e903a638ed383652d9
10:55:59     LightGBM/0.0.1-1@pix4d/stable:2c0843cc59ff2d07e33c808b3398bc624e6b54e4
10:55:59     Mustache/1.0.1-0@pix4d/stable:2c0843cc59ff2d07e33c808b3398bc624e6b54e4
10:55:59     OpenMesh/4.1.1-0@pix4d/stable:2c0843cc59ff2d07e33c808b3398bc624e6b54e4
10:55:59     OpenSceneGraph/3.4.0-0@pix4d/stable:2c0843cc59ff2d07e33c808b3398bc624e6b54e4
10:55:59     proj/4.9.2-0@pix4d/testing:2c0843cc59ff2d07e33c808b3398bc624e6b54e4
10:55:59     zlib/1.2.8@pix4d/stable:2c0843cc59ff2d07e33c808b3398bc624e6b54e4
10:55:59 
10:55:59 CGAL/4.9-0@pix4d/stable: Already installed!
10:55:59 Ceres/1.11.0-0@pix4d/stable: Already installed!
10:55:59 Expat/2.2.0-0@pix4d/stable: Installing package 2c0843cc59ff2d07e33c808b3398bc624e6b54e4
10:55:59 Expat/2.2.0-0@pix4d/stable: Looking for package 2c0843cc59ff2d07e33c808b3398bc624e6b54e4 in remote 'conan-server' 
10:55:59 Downloading conanmanifest.txt
10:55:59 
10:55:59 Downloading conaninfo.txt
10:55:59 
10:55:59 Downloading conan_package.tgz
10:55:59 
10:55:59 Expat/2.2.0-0@pix4d/stable: Package installed 2c0843cc59ff2d07e33c808b3398bc624e6b54e4
10:55:59 LibSURE/1.3.0.490-0@pix4d/stable: Already installed!
10:55:59 Libastro/3.7.7-0@pix4d/stable: Already installed!
10:55:59 LightGBM/0.0.1-1@pix4d/stable: Already installed!
10:55:59 Mustache/1.0.1-0@pix4d/stable: Already installed!
10:55:59 OpenMesh/4.1.1-0@pix4d/stable: Already installed!
10:56:00 OpenSceneGraph/3.4.0-0@pix4d/stable: Already installed!
10:56:00 ERROR: Can't find a 'proj/4.9.2-0@pix4d/testing' package for the specified options and settings
10:56:00 
10:56:00 - Try to build from sources with "--build proj" parameter
10:56:00 - If it fails, you could try to contact the package author  at 'https://bitbucket.org/pix4d/conan-proj', report your configuration and try to collaborate to support it.
10:56:00 
10:56:00 Package configuration:
10:56:00 - Settings: arch=x86_64, build_type=Debug, compiler=Visual Studio, compiler.runtime=MD, compiler.version=12, os=Windows
10:56:00 - Options: 
10:56:00 Conan wrapper: some of the Conan dependencies need to be compiled on this machine
10:56:00 Conan wrapper: running conan: build missing dependencies
...
...
...
# Here we are compiling the missing dependency (At this point we were migrating PROJ from our previous build_depends system to conan)
10:56:00 proj/4.9.2-0@pix4d/testing: Building package from source as defined by build_policy='missing'
10:56:00 proj/4.9.2-0@pix4d/testing: Building your package in C:\Users\pix4d\.conan\data\proj\4.9.2-0\pix4d\testing\build\2c0843cc59ff2d07e33c808b3398bc624e6b54e4
10:56:00 proj/4.9.2-0@pix4d/testing: Copying sources to build folder
...
...
# The package is correctly generated
10:56:29 proj/4.9.2-0@pix4d/testing: Package '2c0843cc59ff2d07e33c808b3398bc624e6b54e4' built
10:56:29 proj/4.9.2-0@pix4d/testing: Build folder C:\Users\pix4d\.conan\data\proj\4.9.2-0\pix4d\testing\build\2c0843cc59ff2d07e33c808b3398bc624e6b54e4
10:56:29 proj/4.9.2-0@pix4d/testing: Generated conaninfo.txt
10:56:29 proj/4.9.2-0@pix4d/testing: Generated conanbuildinfo.txt
10:56:29 proj/4.9.2-0@pix4d/testing: Generated conanenv.txt
10:56:29 proj/4.9.2-0@pix4d/testing: Generating the package
10:56:29 proj/4.9.2-0@pix4d/testing: Package folder C:\Users\pix4d\.conan\data\proj\4.9.2-0\pix4d\testing\package\2c0843cc59ff2d07e33c808b3398bc624e6b54e4
10:56:29 proj/4.9.2-0@pix4d/testing: WARN: This conanfile has no package step
10:56:29 proj/4.9.2-0@pix4d/testing package(): WARN: No files copied!
10:56:29 proj/4.9.2-0@pix4d/testing: Package '2c0843cc59ff2d07e33c808b3398bc624e6b54e4' created
...
...
...
# Now we try to upload all the packages 
10:56:34 Conan wrapper: running conan: upload new packages to the conan-server
10:56:34 Uploading Archive/3.1.2-0@pix4d/stable
10:56:34 Uploaded conan recipe 'Archive/3.1.2-0@pix4d/stable' to 'conan-server': http://conan-server:9300
10:56:34 Uploading package 1/2: 71216f8a3ce0e621eed5c053c417fdcab8134684
10:56:34 
Checking package integrity...                                         
Package integrity OK!                                                 
10:56:34 
Package is up to date.                                                
10:56:34 Uploading package 2/2: 9024ea81a84d4c512108194627f750ed0bdf8225
10:56:35 
Checking package integrity...                                         
Package integrity OK!                                                 
10:56:35 
Package is up to date.                                                
10:56:35 Conan wrapper: running conan: cleanup artifacts
...
...
...
# But when we try to upload mustache ... it fails (it the remote recipe was newer than the local, why it did not download it in the first steps of the script?)
10:57:18 Conan wrapper: running conan: upload new packages to the conan-server
10:57:18 ERROR: Remote recipe is newer than local recipe: 
10:57:19  Remote date: 1486018746
10:57:19  Local date: 1484319002
10:57:19 Traceback (most recent call last):
10:57:19   File "J:/jenkins/workspace/master-debug/compiler/msvc/label/win/mode/debug/pix4dmapper/dev_tools/jenkins/pix4dmapper/build.d/04_conan.py", line 124, in <module>
10:57:19     main()
10:57:19   File "J:/jenkins/workspace/master-debug/compiler/msvc/label/win/mode/debug/pix4dmapper/dev_tools/jenkins/pix4dmapper/build.d/04_conan.py", line 118, in main
10:57:19     conan_commands()
10:57:19   File "J:/jenkins/workspace/master-debug/compiler/msvc/label/win/mode/debug/pix4dmapper/dev_tools/jenkins/pix4dmapper/build.d/04_conan.py", line 97, in conan_commands
10:57:19     check_call(['conan', 'upload', '-r', 'conan-server', req, '--all'])
10:57:19   File "C:\Python27\lib\subprocess.py", line 540, in check_call
10:57:19     raise CalledProcessError(retcode, cmd)
10:57:19 subprocess.CalledProcessError: Command '['conan', 'upload', '-r', 'conan-server', 'Mustache/1.0.1-0@pix4d/stable', '--all']' returned non-zero exit status 1

memsharded commented 7 years ago

Just a couple of quick tips while I review the full report:

If you remove the conan-io remote, or your conan-server is the first one in the list of remotes, it will be the default. Maybe you want to use a "conan remote remove conan.io" & "conan remote add conan-server " at the beginning of your process, to avoid adding "-r conan-server" to every command.
For your step 3, the command "conan info --only" would give a stripped list of dependencies, maybe easier to parse. Also, if the job is building many different binaries, it could be useful to get the full package reference from conaninfo.txt and don't upload --all, but -p ID of the package. Even if packages are checked (md5) before re-uploading, that take some time for big packages.

memsharded commented 7 years ago

A couple of questions:

10:56:00 ERROR: Can't find a 'proj/4.9.2-0@pix4d/testing' package for the specified options and settings ... Here we are compiling the missing dependency (At this point we were migrating PROJ from our previous build_depends system to conan) 10:56:00 proj/4.9.2-0@pix4d/testing: Building package from source as defined by build_policy='missing'

This is a bit weird, it might be a bug, can you please verify this is correct? If the package recipe has declared a build_policy=missing, then the installation shouldn't fail, it should build the packages from sources without complaining there is no pre-compiled binary.

About the failure while uploading "Mustache", I guess it could come from concurrent building. I remember you were running concurrent CI jobs, so is it possible that another job has uploaded its packages for Mustache before this job tried to upload its binaries? I think this could be the issue, as the recipe is always uploaded when uploading binaries, for consistency, and we probably have to think about something to address this use case.

I would need to know also the procedure for first upload of package recipes, how are recipes uploaded initially? Because this process describes the flow for creating binaries from them, but not where the recipes come from.

Please tell me, I am starting to investigate possible approaches to fix this issue.

piponazo commented 7 years ago

Thank you for your quick response :).

About this tips:

I already knew what you mentioned about not using the -r conan-server but it makes more readable the script ;).
I did not consider to upload just the packages that were compiled for the current configuration. I did not know from where to extract that information. It is a good Idea and I will try to implement it :+1: . We were not using before conan info but we just started to use it trying to solve the issue that I just reported. However we realised that we need #940 for this approach.

About the question:

Take into account that the first time that we run conan install we are not specifying the build policy. We just want to know at that point if we have all the project dependencies in the current node or we need to run additional conan steps. In the normal scenario (compilation of our master branch) the first command will detect that the project dependencies are satisfied and we can continue with the compilation of our project. Only when somebody is working on a branch doing conan stuff (migrate one project dependency to conan, update one of the existing conan-recipes, etc) is when we need to run the other steps.

I also think that the problem might come from the concurrent configuration. Each time we trigger a jenkins compilation job we execute in parallel the following configurations:

linux / release
linux / debug
mac / release
mac / debug
win / release
win / debug

Actually for linux & mac (debug), we do not specify the build_type and we just use the Release version of the packages (normally we are not interested in the debug symbols of 3rd party libraries). Therefore, it could be that some nodes are compiling the packages more or less at the same time and trying to push the packages to the server. I thought that this case was controlled, but you mentioned that it could be an issue. I am happy if this could help you to find new issues in your software ;)

Finally, our procedure for uploading recipes to the conan server is the following:

A developer work on a recipe in a independent repository, i.e. conan-mustache. He/She uses conan test_package locally to test that everything is fine.
Once the recipe is working for his/her main platform (normally linux) we upload the recipe to our conan-server (to the testing channel).
Now we make our project depend on this new recipe. Jenkins will trigger a build job for our project and it should find the new recipe in the conan-server. The packages will be generated for each platform and uploaded to the conan-server.

If you need more information, do not hesitate to ask :smiley:

memsharded commented 7 years ago

Yes, all of this makes perfect sense. I am trying to figure out something, doesn't seem simple, but we will keep working on it. Thanks again for your excellent detailed feedback.

memsharded commented 7 years ago

I am working on this. It could be possible, but only if the creation of the "Mustache" package is being executed concurrently. The only moment that the timestamp of the package recipe, that is the one which is causing the difference, is generated at conan export (also at conan test_package, as this does an export too).

If the creation of Mustache is clearly done (and finished) before launching the build of the project, I am not sure where the difference in the recipe timestamps could come from.

memsharded commented 7 years ago

Hi @piponazo Sorry we didn't manage to address this for 0.20. The next release is already huge and still a lot of work to do before it is ready. As this is an important issue that needs time, we have had to move it forward for next 0.21.

memsharded commented 7 years ago

This should be addressed by #1074, so marking it as fixed, will be available in next 0.21

conan-io / conan

conanmanifest.txt not updated with conan install --update #949