BuildInfo Create crashes on header-only dependency

yarcod-zpt commented 1 month ago

Hi!

I've been troubleshooting an issue from our CI, where it was first running: conan create . -pr x86_64-gcc-9 -pr:b default -s '&:build_type=Debug' --format json > graph_local.json followed by conan art:build-info create graph_local.json test_build 1 conan --server artifactory --with-dependencies. This results in a crash with a stacktrace like this:

> conan art:build-info create graph_local.json test_build 1 conan --server artifactory --
with-dependencies
ERROR: Traceback (most recent call last):
  File "/home/jenkins/.local/lib/python3.8/site-packages/conan/cli/cli.py", line 294, in main
    cli.run(args)
  File "/home/jenkins/.local/lib/python3.8/site-packages/conan/cli/cli.py", line 193, in run
    command.run(self._conan_api, args[0][1:])
  File "/home/jenkins/.local/lib/python3.8/site-packages/conan/cli/command.py", line 187, in run
    sub.run(conan_api, parser, *args)
  File "/home/jenkins/.local/lib/python3.8/site-packages/conan/cli/command.py", line 205, in run
    info = self._method(conan_api, parent_parser, self._parser, *args)
  File "/var/jenkins_home/workspace/test_build_branch/.conan2/extensions/commands/art/cmd_build_info.py", line 429, in build_info_create
    cli_out_write(bi.create())
  File "/var/jenkins_home/workspace/test_build_branch/.conan2/extensions/commands/art/cmd_build_info.py", line 342, in create
    bi.update({"modules": self.get_modules()})
  File "/var/jenkins_home/workspace/test_build_branch/.conan2/extensions/commands/art/cmd_build_info.py", line 326, in get_modules
    package_module = self.create_module(node, "package", transitive_dependencies)
  File "/var/jenkins_home/workspace/test_build_branch/.conan2/extensions/commands/art/cmd_build_info.py", line 286, in create_module
    deps_artifacts = self.get_artifacts(nodes.get(require_id), module_type, is_dependency=True)
  File "/var/jenkins_home/workspace/test_build_branch/.conan2/extensions/commands/art/cmd_build_info.py", line 206, in get_artifacts
    artifacts = _get_local_artifacts()
  File "/var/jenkins_home/workspace/test_build_branch/.conan2/extensions/commands/art/cmd_build_info.py", line 145, in _get_local_artifacts
    dl_folder = Path(artifacts_folder).parents[0] / "d"
  File "/usr/lib/python3.8/pathlib.py", line 1042, in __new__
    self = cls._from_parts(args, init=False)
  File "/usr/lib/python3.8/pathlib.py", line 683, in _from_parts
    drv, root, parts = self._parse_args(args)
  File "/usr/lib/python3.8/pathlib.py", line 667, in _parse_args
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType

ERROR: expected str, bytes or os.PathLike object, not NoneType

Copying the generated graph_local.json from the CI environment to my machine and performing the same conan art:build-info create ... command as above, I instead got this error message: ERROR: There are missing artifacts for the cli11/2.1.1#f536e14f30439418720dfd2a13a448cd recipe. Check that you have all the packages installed in the Conan cache when creating the Build Info.. When performing the full chain of commands leading up to this, I get the same raised exception as above -- identical apart from the paths, of course. I ensured that the same Python version (3.8) was used locally as well.

I confirmed via debugger that the crash happens because cli11 has no package_folder set, which I of course could see in the generated graph as well. The working theory is that either the graph generation does not properly handle transient header-only libraries, or the interpretation of the graph done here does not handle that specific case. I should mention, however, that the same commands work just fine when creating the package with another set of options, e.g. -s '&:build_type=Release' (there are other options as well, which I left out for brevity).

Do you have any suggestions on how I could move forward at this point?

memsharded commented 1 month ago

Hi @yarcod-zpt

Thanks for your report.

It seems you might be running an outdated command. I have checked the code, and it seems to be fixed now. Can you please make sure to update to the latest code from this repo?

yarcod-zpt commented 1 month ago

Thanks for your reply @memsharded!

I made sure to update the command to the latest version from this repo, but still have the same outcome.

Not sure if it helps, but just to clarify the context in which cli11 is used in our case:

Our internal package_a uses cli11 when building. This works fine on its own, using the exact same commands in the CI pipeline as the faulty one.
Another package, package_b, uses package_a as a dependency.
package_a is comprised of multiple components, i.e. self.cpp_info.components["package_a"].requires = [ ... , cli11::cli11 ].
I have tried removing it from the above mentioned .requires list. This does not work, even though I don't need/want to expose it to any consumer (as it is a header-only library, should be statically compiled here).

The issue could be because the graph of package_b resolves cli11 first as skip: True for the package_a dependency, but then for some reason sets cli11 as skip: False when it tries to resolve it for itself; one guess is that the header only library by default sets transitive_headers=True, which could mean that the consumer tries to resolve those headers but fails when the package path turns out empty.

While I think I have managed to work around the issue by setting self.requires(..., visible=False)" for cli11 in package_a, I still believe we might have hit an edge case here that handles the graph badly.

memsharded commented 1 month ago

Thanks for the details.

While I think I have managed to work around the issue by setting self.requires(..., visible=False)"

No, this shouldn't be necessary, we should definitely fix the error. Could you please share the stacktrace that is produced with the updated code? That can help us to reproduce and fix it. Thanks!

yarcod-zpt commented 1 month ago

The stacktrace is seemingly identical to the one I posted initially, but I will add it here in case there is some part I overlooked. This build should have been using the latest version of the extensions.

> conan art:build-info create graph_local.json test_build 5 conan --server artifactory --with-dependencies
ERROR: Traceback (most recent call last):
  File "/home/jenkins/.local/lib/python3.8/site-packages/conan/cli/cli.py", line 294, in main
    cli.run(args)
  File "/home/jenkins/.local/lib/python3.8/site-packages/conan/cli/cli.py", line 193, in run
    command.run(self._conan_api, args[0][1:])
  File "/home/jenkins/.local/lib/python3.8/site-packages/conan/cli/command.py", line 187, in run
    sub.run(conan_api, parser, *args)
  File "/home/jenkins/.local/lib/python3.8/site-packages/conan/cli/command.py", line 205, in run
    info = self._method(conan_api, parent_parser, self._parser, *args)
  File "/var/jenkins_home/workspace/test_build_branch/.conan2/extensions/commands/art/cmd_build_info.py", line 429, in build_info_create
    cli_out_write(bi.create())
  File "/var/jenkins_home/workspace/test_build_branch/.conan2/extensions/commands/art/cmd_build_info.py", line 342, in create
    bi.update({"modules": self.get_modules()})
  File "/var/jenkins_home/workspace/test_build_branch/.conan2/extensions/commands/art/cmd_build_info.py", line 326, in get_modules
    package_module = self.create_module(node, "package", transitive_dependencies)
  File "/var/jenkins_home/workspace/test_build_branch/.conan2/extensions/commands/art/cmd_build_info.py", line 286, in create_module
    deps_artifacts = self.get_artifacts(nodes.get(require_id), module_type, is_dependency=True)
  File "/var/jenkins_home/workspace/test_build_branch/.conan2/extensions/commands/art/cmd_build_info.py", line 206, in get_artifacts
    artifacts = _get_local_artifacts()
  File "/var/jenkins_home/workspace/test_build_branch/.conan2/extensions/commands/art/cmd_build_info.py", line 145, in _get_local_artifacts
    dl_folder = Path(artifacts_folder).parents[0] / "d"
  File "/usr/lib/python3.8/pathlib.py", line 1042, in __new__
    self = cls._from_parts(args, init=False)
  File "/usr/lib/python3.8/pathlib.py", line 683, in _from_parts
    drv, root, parts = self._parse_args(args)
  File "/usr/lib/python3.8/pathlib.py", line 667, in _parse_args
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType

ERROR: expected str, bytes or os.PathLike object, not NoneType

memsharded commented 1 month ago

I am not sure what is happening.

Your stacktrace is reporting in Line 145:

L145: dl_folder = Path(artifacts_folder).parents[0] / "d"

but you can see in the latest in the repo: https://github.com/conan-io/conan-extensions/blob/5656f7c858bec27786e009ac8d315365ef85cbcd/extensions/commands/art/cmd_build_info.py#L145

L145:    artifacts_folder = Path(node.get("package_folder")) if artifact_type == "package" else Path(node.get("recipe_folder"))
L146:    dl_folder = artifacts_folder.parents[0] / "d"

As you can see, it seems that the lines do not match.

yarcod-zpt commented 1 month ago

I agree that it does look weird -- it seems like the CI does not pull down the latest version of the extensions despite it having been updated. Running an entirely new job still checks out the old extensions, despite just pointing at the repository, i.e. main. This smells more like some weird caching behaviour of our CI than anything else.

I'll go another round and ensure that the latest version is actually used. I might use your suggestion to use a specific version of the extensions in the future to avoid this issue.

conan-io / conan-extensions

BuildInfo Create crashes on header-only dependency #165