Closed drussel closed 5 years ago
Conan is trying to build another package other than boost as part of this run.
I never seen that someone executes Conan this way. If you have installed Conan with PIP then you automatically get an executable conan.exe
in your Python search path.
pip install -U conan
c:\tmp> conan install "path/to/parent/of/conanfile.py" "version@user/channel" -g cmake -if "path/to/working/dir" --build=missing -s compiler="Visual Studio" -s compiler.version=15 -o tbbmalloc=True -s build_type=Release
The complaint about a missing reference is correct according to the docs. If you do not supply the name and/or version in the recipe you have to provide this information at Conan invocation.
We use a virtual environment so we can control the Conan version as part of our build process. This is only partly successful since we are caught between Conan changes that impact the ~/.conan files and those that impact recipes, but seems to work better than trying to trust the system Conan to be an appropriate version on all systems where we try to build (and that all projects agree on an appropriate version) or enforcing the Conan version as part of the build.
I'm not sure invoking it with python -m conans.conan
does anything useful any more. It was a work around to deal with path/pythonpath variability prior to switching to the virtual env since getting a consistent behavior across various developers (windows) machines and Jenkins machines had proved tricky.
I'll see if calling venv/[bin,Scripts]/conan
directly makes any difference.
I don't think to call conan that way makes any difference. I'm trying to reproduce or figure out, but it is indeed a bug. Thanks for reporting!
@drussel Could you please paste here the contents (if present) of the file in your conan cache at:
~/.conan/data/Boost/1.68.0-16/velo3d/thirdparty/metadata.json
?
To confirm, you haven't removed that Boost package from the cache after trying to upgrade to 1.14.1
, right?
EDIT: By the way, about the version@user/channel
, the error is confusing but that has nothing to do with the real issue. We have to improve that to raise more meaningful errors.
We had a similar issue today on one of our mac nodes, using conan 1.14.0, with this error message:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/conans/model/ref.py", line 111, in loads
_, name, version, user, channel, revision, _ = ConanFileReference.sep_pattern.split(text)
ValueError: not enough values to unpack (expected 7, got 1)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/conans/client/command.py", line 376, in install
ref = ConanFileReference.loads(args.path_or_reference)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/conans/model/ref.py", line 114, in loads
"OpenCV/1.0.6@user/stable" % text)
conans.errors.ConanException: Wrong package recipe reference pix4d-magma
Write something like OpenCV/1.0.6@user/stable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/conans/client/command.py", line 1578, in run
method(args[0][1:])
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/conans/client/command.py", line 389, in install
install_folder=args.install_folder)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/conans/client/conan_api.py", line 93, in wrapper
return f(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/conans/client/conan_api.py", line 580, in install
no_imports=no_imports)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/conans/client/manager.py", line 80, in install
installer.install(deps_graph, keep_build)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/conans/client/installer.py", line 302, in install
self._build(nodes_by_level, keep_build, root_node, graph_info)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/conans/client/installer.py", line 324, in _build
self._handle_node_cache(node, keep_build, processed_package_refs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/conans/client/installer.py", line 404, in _handle_node_cache
assert node.prev, "PREV for %s is None" % str(pref)
AssertionError: PREV for proj/4.9.3-6@pix4d/stable:f8bda7f0751e4bc3beaa6c3b2eb02d455291c8a2 is None
The content of ~/.conan/data/proj/4.9.3-6/pix4d/stable/metadata.json is:
{"recipe": {"revision": "0", "properties": {}}, "packages": {}}
@david-sinuela-pix4d-dev Could you say if that package was also created with a previous conan version?
I've been able to reproduce the issue. You have the package f8bda7f0751e4bc3beaa6c3b2eb02d455291c8a2
but the metadata.json
file is not containing that package ID (in your case none of them).
We could patch it but the goal is to guess why it happened.
Could you tell us how that package was generated?
conan create
?conan export-pkg
?conan copy
?
... any other information you could consider relevant?@lasote The package was generated in a separate CI job that builds the dependencies and uploaded to a local conan server using conan package tools. Both the builder and the server were using conan 1.12.3 until a few days ago when we upgraded to version 1.14.0. Actually I just checked, the packages were produced in February, at that time the builder node had conan 1.9.1 and conan package tools 0.20.
The node that failed today consuming the packages had conan 1.12.3 before and has 1.14.0 now.
I don't know if I will be able to reproduce the issue again as the nodes will be recycled tonight.
Thanks for the info. I'm trying changing conan version and I'm not able to reproduce the issue. I think we might consider a package as "corrupted" or "invalid" if at the local cache it has no entry in the "packages" at the metadata.json. Will talk with the team.
Hi @drussel @david-sinuela-pix4d-dev
I have been investigating this, without much success. I have introduced a new assert in https://github.com/conan-io/conan/pull/4927, which might bring some new data. It is already in develop, so if you could please try it from the test package:
https://test.pypi.org/project/conan/1.15.0.dev1554729202/
$ pip install -i https://test.pypi.org/simple/ conan==1.15.0.dev1554729202
And run it against your project, that would help. Many thanks!
The metadata printed out is with the dev version is {"recipe": {"revision": "0", "properties": {}}, "packages": {"b16eb78376abcccbbb95c0ccfd7fc1c01657972d": {"revision": "0", "recipe_revision": "0", "properties": {}}, "3ec8b5bd6b1c2f5e8ca4facd1a389bef6866cd83": {"revision": "0", "recipe_revision": "0", "properties": {}}, "df18e90df1fedfd9db3e0852ce38a27c76c5a54b": {"revision": null, "recipe_revision": null, "properties" : {}}}}
otherwise things look the same
conan install path/to/dir/with/conanfile.py repo@velo3d/stable -g cmake -if builddir --build missing -s "compiler=Visual Studio" -s compiler.version=15 -o tbbmalloc=True -s build_type=Release
Configuration:
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=Visual Studio
compiler.runtime=MD
compiler.version=15
os=Windows
os_build=Windows
[options]
tbbmalloc=True
[build_requires]
[env]
WARN: ZipLib/5d0ad773801-12@velo3d/thirdparty requirement Boost/1.68.0-15@velo3d/thirdparty overridden by Flow/repo@velo3d/stable to Boost/1.68.0-16@velo3d/thirdparty
WARN: avro-cpp/137a5212-13@velo3d/thirdparty requirement Boost/1.68.0-15@velo3d/thirdparty overridden by Flow/repo@velo3d/stable to Boost/1.68.0-16@velo3d/thirdparty
WARN: aws-sdk-cpp/1.7.73-5@velo3d/thirdparty requirement zlib/1.2.11@velo3d/thirdparty overridden by Flow/repo@velo3d/stable to zlib/1.2.11-4@velo3d/thirdparty
WARN: embree3/14630235779-4@velo3d/thirdparty requirement IntelTBB/2019_U1-5@velo3d/thirdparty overridden by Flow/repo@velo3d/stable to IntelTBB/2019_U3-6@velo3d/thirdparty
WARN: QtAutoUpdater/1.0@velo3d/thirdparty requirement Qt/5.10.1-11@velo3d/thirdparty overridden by Flow/repo@velo3d/stable to Qt/5.10.1-14@velo3d/thirdparty
Traceback (most recent call last):
File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\model\ref.py", line 111, in loads
_, name, version, user, channel, revision, _ = ConanFileReference.sep_pattern.split(text)
ValueError: not enough values to unpack (expected 7, got 1)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\client\command.py", line 377, in install
ref = ConanFileReference.loads(args.path_or_reference)
File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\model\ref.py", line 114, in loads
"OpenCV/1.0.6@user/stable" % text)
conans.errors.ConanException: Wrong package recipe reference C:/Users/daniel/src/flow-3
Write something like OpenCV/1.0.6@user/stable
During handling of the above exception, another exception occurred:
File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\client\command.py", line 1581, in run
method(args[0][1:])
File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\client\command.py", line 390, in install
install_folder=args.install_folder)
File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\client\conan_api.py", line 93, in wrapper
return f(*args, **kwargs)
File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\client\conan_api.py", line 580, in install
no_imports=no_imports)
File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\client\manager.py", line 60, in install
self._recorder)
File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\client\graph\graph_manager.py", line 149, i
n load_graph
apply_build_requires=apply_build_requires)
File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\client\graph\graph_manager.py", line 241, i
apply_build_requires=apply_build_requires)
File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\client\graph\graph_manager.py", line 179, i
n _recurse_build_requires
binaries_analyzer.evaluate_graph(graph, build_mode, update, remote_name)
File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\client\graph\graph_binaries.py", line 217,
in evaluate_graph
self._evaluate_node(node, build_mode, update, evaluated, remote_name)
File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\client\graph\graph_binaries.py", line 115,
in _evaluate_node
assert node.prev, "PREV for %s is None: %s" % (str(pref), metadata.dumps())
AssertionError: PREV for Boost/1.68.0-16@velo3d/thirdparty:df18e90df1fedfd9db3e0852ce38a27c76c5a54b is None: {"recipe":
{"revision": "0", "properties": {}}, "packages": {"b16eb78376abcccbbb95c0ccfd7fc1c01657972d": {"revision": "0", "recipe_
revision": "0", "properties": {}}, "3ec8b5bd6b1c2f5e8ca4facd1a389bef6866cd83": {"revision": "0", "recipe_revision": "0",
"properties": {}}, "df18e90df1fedfd9db3e0852ce38a27c76c5a54b": {"revision": null, "recipe_revision": null, "properties"
: {}}}}
ERROR: PREV for Boost/1.68.0-16@velo3d/thirdparty:df18e90df1fedfd9db3e0852ce38a27c76c5a54b is None: {"recipe": {"revisio
n": "0", "properties": {}}, "packages": {"b16eb78376abcccbbb95c0ccfd7fc1c01657972d": {"revision": "0", "recipe_revision"
: "0", "properties": {}}, "3ec8b5bd6b1c2f5e8ca4facd1a389bef6866cd83": {"revision": "0", "recipe_revision": "0", "propert
ies": {}}, "df18e90df1fedfd9db3e0852ce38a27c76c5a54b": {"revision": null, "recipe_revision": null, "properties": {}}}}
Looking some more. I have 9 subdirs in my packages folder for that recipe, but it looks like only 3 in the metadata. The df18 one (will the null revision) looks to be the main one that gets used. If I conan search for that recipe, I get 7 hits, one of which lists no settings out options (that one is not in the metadata).
A lot of the initial package development and building occurs on the machine in question. And there have likely been failures and aborts of builds along the way.
The one with no options listed looks to be partial (no lib dir in the package).
Thanks very much for the feedback. It seems that some null
revisions slipped into the metadata even with the migrations. We are going to prepare and release a new patch version with a more thorough migration asap.
Migration merged, will be released in 1.14.2
Thanks for the quick fix. Just one note: we are still using 1.14.0 and since the nodes were recycled we have not seen the problem again.
We are trying to update from 1.13.1 to 1.14.1. However, when we run
path/to/python.exe -m conans.conan install path/to/parent/of/conanfile.py version@user/channel -g cmake -if path/to/working/dir --build missing -s "compiler=Visual Studio" -s compiler.version=15 -o tbbmalloc=True -s build_type=Release
we get an errorI'm at a bit of a loss as to what is going on.
The
version@user/channel
in the command is something we were not previously using, but we get a complaint about a missing reference if it isn't there.The boost package should already be on the system, if that matters.
Thanks.