search

conan-io / conan

Conan - The open-source C and C++ package manager
https://conan.io
MIT License
8.11k stars 963 forks source link

Issue updating to 1.14.1: "PREV for xxxx is None" #4898

Closed drussel closed 5 years ago

drussel commented 5 years ago

We are trying to update from 1.13.1 to 1.14.1. However, when we run path/to/python.exe -m conans.conan install path/to/parent/of/conanfile.py version@user/channel -g cmake -if path/to/working/dir --build missing -s "compiler=Visual Studio" -s compiler.version=15 -o tbbmalloc=True -s build_type=Release we get an error

Traceback (most recent call last):
  File "build\venv\lib\site-packages\conans\model\ref.py", line 111, in loads
    _, name, version, user, channel, revision, _ = ConanFileReference.sep_pattern.split(text)
ValueError: not enough values to unpack (expected 7, got 1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "build\venv\lib\site-packages\conans\client\command.py", line 377, in install
    ref = ConanFileReference.loads(args.path_or_reference)
  File "build\venv\lib\site-packages\conans\model\ref.py", line 114, in loads
    "OpenCV/1.0.6@user/stable" % text)
conans.errors.ConanException: Wrong package recipe reference C:/Users/daniel/src/flow
Write something like OpenCV/1.0.6@user/stable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "build\venv\lib\site-packages\conans\client\command.py", line 1579, in run
    method(args[0][1:])
  File "build\venv\lib\site-packages\conans\client\command.py", line 390, in install
    install_folder=args.install_folder)
  File "build\venv\lib\site-packages\conans\client\conan_api.py", line 93, in wrapper
    return f(*args, **kwargs)
  File "build\venv\lib\site-packages\conans\client\conan_api.py", line 580, in install
    no_imports=no_imports)
  File "build\venv\lib\site-packages\conans\client\manager.py", line 80, in install
    installer.install(deps_graph, keep_build)
  File "build\venv\lib\site-packages\conans\client\installer.py", line 302, in install
    self._build(nodes_by_level, keep_build, root_node, graph_info)
  File "build\venv\lib\site-packages\conans\client\installer.py", line 324, in _build
    self._handle_node_cache(node, keep_build, processed_package_refs)
  File "build\venv\lib\site-packages\conans\client\installer.py", line 404, in _handle_node_cache
    assert node.prev, "PREV for %s is None" % str(pref)
AssertionError: PREV for Boost/1.68.0-16@velo3d/thirdparty:df18e90df1fedfd9db3e0852ce38a27c76c5a54b is None

ERROR: PREV for Boost/1.68.0-16@velo3d/thirdparty:df18e90df1fedfd9db3e0852ce38a27c76c5a54b is None

I'm at a bit of a loss as to what is going on.

The version@user/channel in the command is something we were not previously using, but we get a complaint about a missing reference if it isn't there.

The boost package should already be on the system, if that matters.

Thanks.

drussel commented 5 years ago

Conan is trying to build another package other than boost as part of this run.

Johnnyxy commented 5 years ago

I never seen that someone executes Conan this way. If you have installed Conan with PIP then you automatically get an executable conan.exe in your Python search path.

pip install -U conan
c:\tmp> conan install "path/to/parent/of/conanfile.py" "version@user/channel" -g cmake -if "path/to/working/dir" --build=missing -s compiler="Visual Studio" -s compiler.version=15 -o tbbmalloc=True -s build_type=Release

The complaint about a missing reference is correct according to the docs. If you do not supply the name and/or version in the recipe you have to provide this information at Conan invocation.

drussel commented 5 years ago

We use a virtual environment so we can control the Conan version as part of our build process. This is only partly successful since we are caught between Conan changes that impact the ~/.conan files and those that impact recipes, but seems to work better than trying to trust the system Conan to be an appropriate version on all systems where we try to build (and that all projects agree on an appropriate version) or enforcing the Conan version as part of the build.

I'm not sure invoking it with python -m conans.conan does anything useful any more. It was a work around to deal with path/pythonpath variability prior to switching to the virtual env since getting a consistent behavior across various developers (windows) machines and Jenkins machines had proved tricky.

I'll see if calling venv/[bin,Scripts]/conan directly makes any difference.

lasote commented 5 years ago

I don't think to call conan that way makes any difference. I'm trying to reproduce or figure out, but it is indeed a bug. Thanks for reporting!

lasote commented 5 years ago

@drussel Could you please paste here the contents (if present) of the file in your conan cache at: ~/.conan/data/Boost/1.68.0-16/velo3d/thirdparty/metadata.json? To confirm, you haven't removed that Boost package from the cache after trying to upgrade to 1.14.1, right?

EDIT: By the way, about the version@user/channel, the error is confusing but that has nothing to do with the real issue. We have to improve that to raise more meaningful errors.

david-sinuela-pix4d commented 5 years ago

We had a similar issue today on one of our mac nodes, using conan 1.14.0, with this error message:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/conans/model/ref.py", line 111, in loads
    _, name, version, user, channel, revision, _ = ConanFileReference.sep_pattern.split(text)
ValueError: not enough values to unpack (expected 7, got 1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/conans/client/command.py", line 376, in install
    ref = ConanFileReference.loads(args.path_or_reference)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/conans/model/ref.py", line 114, in loads
    "OpenCV/1.0.6@user/stable" % text)
conans.errors.ConanException: Wrong package recipe reference pix4d-magma
Write something like OpenCV/1.0.6@user/stable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/conans/client/command.py", line 1578, in run
    method(args[0][1:])
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/conans/client/command.py", line 389, in install
    install_folder=args.install_folder)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/conans/client/conan_api.py", line 93, in wrapper
    return f(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/conans/client/conan_api.py", line 580, in install
    no_imports=no_imports)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/conans/client/manager.py", line 80, in install
    installer.install(deps_graph, keep_build)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/conans/client/installer.py", line 302, in install
    self._build(nodes_by_level, keep_build, root_node, graph_info)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/conans/client/installer.py", line 324, in _build
    self._handle_node_cache(node, keep_build, processed_package_refs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/conans/client/installer.py", line 404, in _handle_node_cache
    assert node.prev, "PREV for %s is None" % str(pref)
AssertionError: PREV for proj/4.9.3-6@pix4d/stable:f8bda7f0751e4bc3beaa6c3b2eb02d455291c8a2 is None

The content of ~/.conan/data/proj/4.9.3-6/pix4d/stable/metadata.json is:

{"recipe": {"revision": "0", "properties": {}}, "packages": {}}
lasote commented 5 years ago

@david-sinuela-pix4d-dev Could you say if that package was also created with a previous conan version?

lasote commented 5 years ago

I've been able to reproduce the issue. You have the package f8bda7f0751e4bc3beaa6c3b2eb02d455291c8a2 but the metadata.json file is not containing that package ID (in your case none of them). We could patch it but the goal is to guess why it happened. Could you tell us how that package was generated?

david-sinuela-pix4d commented 5 years ago

@lasote The package was generated in a separate CI job that builds the dependencies and uploaded to a local conan server using conan package tools. Both the builder and the server were using conan 1.12.3 until a few days ago when we upgraded to version 1.14.0. Actually I just checked, the packages were produced in February, at that time the builder node had conan 1.9.1 and conan package tools 0.20.

The node that failed today consuming the packages had conan 1.12.3 before and has 1.14.0 now.

I don't know if I will be able to reproduce the issue again as the nodes will be recycled tonight.

lasote commented 5 years ago

Thanks for the info. I'm trying changing conan version and I'm not able to reproduce the issue. I think we might consider a package as "corrupted" or "invalid" if at the local cache it has no entry in the "packages" at the metadata.json. Will talk with the team.

memsharded commented 5 years ago

Hi @drussel @david-sinuela-pix4d-dev

I have been investigating this, without much success. I have introduced a new assert in https://github.com/conan-io/conan/pull/4927, which might bring some new data. It is already in develop, so if you could please try it from the test package:

https://test.pypi.org/project/conan/1.15.0.dev1554729202/

$ pip install -i https://test.pypi.org/simple/ conan==1.15.0.dev1554729202

And run it against your project, that would help. Many thanks!

drussel commented 5 years ago

The metadata printed out is with the dev version is {"recipe": {"revision": "0", "properties": {}}, "packages": {"b16eb78376abcccbbb95c0ccfd7fc1c01657972d": {"revision": "0", "recipe_revision": "0", "properties": {}}, "3ec8b5bd6b1c2f5e8ca4facd1a389bef6866cd83": {"revision": "0", "recipe_revision": "0", "properties": {}}, "df18e90df1fedfd9db3e0852ce38a27c76c5a54b": {"revision": null, "recipe_revision": null, "properties" : {}}}}

otherwise things look the same

conan install path/to/dir/with/conanfile.py repo@velo3d/stable -g cmake -if builddir --build missing -s "compiler=Visual Studio" -s compiler.version=15 -o tbbmalloc=True -s build_type=Release
Configuration:
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=Visual Studio
compiler.runtime=MD
compiler.version=15
os=Windows
os_build=Windows
[options]
tbbmalloc=True
[build_requires]
[env]

WARN: ZipLib/5d0ad773801-12@velo3d/thirdparty requirement Boost/1.68.0-15@velo3d/thirdparty overridden by Flow/repo@velo3d/stable to Boost/1.68.0-16@velo3d/thirdparty
WARN: avro-cpp/137a5212-13@velo3d/thirdparty requirement Boost/1.68.0-15@velo3d/thirdparty overridden by Flow/repo@velo3d/stable to Boost/1.68.0-16@velo3d/thirdparty
WARN: aws-sdk-cpp/1.7.73-5@velo3d/thirdparty requirement zlib/1.2.11@velo3d/thirdparty overridden by Flow/repo@velo3d/stable to zlib/1.2.11-4@velo3d/thirdparty
WARN: embree3/14630235779-4@velo3d/thirdparty requirement IntelTBB/2019_U1-5@velo3d/thirdparty overridden by Flow/repo@velo3d/stable to IntelTBB/2019_U3-6@velo3d/thirdparty
WARN: QtAutoUpdater/1.0@velo3d/thirdparty requirement Qt/5.10.1-11@velo3d/thirdparty overridden by Flow/repo@velo3d/stable to Qt/5.10.1-14@velo3d/thirdparty
Traceback (most recent call last):
  File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\model\ref.py", line 111, in loads
    _, name, version, user, channel, revision, _ = ConanFileReference.sep_pattern.split(text)
ValueError: not enough values to unpack (expected 7, got 1)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\client\command.py", line 377, in install
    ref = ConanFileReference.loads(args.path_or_reference)
  File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\model\ref.py", line 114, in loads
    "OpenCV/1.0.6@user/stable" % text)
conans.errors.ConanException: Wrong package recipe reference C:/Users/daniel/src/flow-3
Write something like OpenCV/1.0.6@user/stable

During handling of the above exception, another exception occurred:

  File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\client\command.py", line 1581, in run
    method(args[0][1:])
  File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\client\command.py", line 390, in install
    install_folder=args.install_folder)
  File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\client\conan_api.py", line 93, in wrapper
    return f(*args, **kwargs)
  File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\client\conan_api.py", line 580, in install
    no_imports=no_imports)
  File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\client\manager.py", line 60, in install
    self._recorder)
  File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\client\graph\graph_manager.py", line 149, i
n load_graph
    apply_build_requires=apply_build_requires)
  File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\client\graph\graph_manager.py", line 241, i
    apply_build_requires=apply_build_requires)
  File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\client\graph\graph_manager.py", line 179, i
n _recurse_build_requires
    binaries_analyzer.evaluate_graph(graph, build_mode, update, remote_name)
  File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\client\graph\graph_binaries.py", line 217,
in evaluate_graph
    self._evaluate_node(node, build_mode, update, evaluated, remote_name)
  File "C:\Users\daniel\AppData\Roaming\Python\Python37\site-packages\conans\client\graph\graph_binaries.py", line 115,
in _evaluate_node
    assert node.prev, "PREV for %s is None: %s" % (str(pref), metadata.dumps())
AssertionError: PREV for Boost/1.68.0-16@velo3d/thirdparty:df18e90df1fedfd9db3e0852ce38a27c76c5a54b is None: {"recipe":
{"revision": "0", "properties": {}}, "packages": {"b16eb78376abcccbbb95c0ccfd7fc1c01657972d": {"revision": "0", "recipe_
revision": "0", "properties": {}}, "3ec8b5bd6b1c2f5e8ca4facd1a389bef6866cd83": {"revision": "0", "recipe_revision": "0",
 "properties": {}}, "df18e90df1fedfd9db3e0852ce38a27c76c5a54b": {"revision": null, "recipe_revision": null, "properties"
: {}}}}

ERROR: PREV for Boost/1.68.0-16@velo3d/thirdparty:df18e90df1fedfd9db3e0852ce38a27c76c5a54b is None: {"recipe": {"revisio
n": "0", "properties": {}}, "packages": {"b16eb78376abcccbbb95c0ccfd7fc1c01657972d": {"revision": "0", "recipe_revision"
: "0", "properties": {}}, "3ec8b5bd6b1c2f5e8ca4facd1a389bef6866cd83": {"revision": "0", "recipe_revision": "0", "propert
ies": {}}, "df18e90df1fedfd9db3e0852ce38a27c76c5a54b": {"revision": null, "recipe_revision": null, "properties": {}}}}
drussel commented 5 years ago

Looking some more. I have 9 subdirs in my packages folder for that recipe, but it looks like only 3 in the metadata. The df18 one (will the null revision) looks to be the main one that gets used. If I conan search for that recipe, I get 7 hits, one of which lists no settings out options (that one is not in the metadata).

A lot of the initial package development and building occurs on the machine in question. And there have likely been failures and aborts of builds along the way.

The one with no options listed looks to be partial (no lib dir in the package).

memsharded commented 5 years ago

Thanks very much for the feedback. It seems that some null revisions slipped into the metadata even with the migrations. We are going to prepare and release a new patch version with a more thorough migration asap.

memsharded commented 5 years ago

Migration merged, will be released in 1.14.2

david-sinuela-pix4d commented 5 years ago

Thanks for the quick fix. Just one note: we are still using 1.14.0 and since the nodes were recycled we have not seen the problem again.