Hooks sabotage conandata.yml generated in export method

jwillikers commented 1 year ago

With Conan hooks installed, it is not possible to build a recipe according to example using the new Git functionality in the export and source method: https://docs.conan.io/en/latest/migrating_to_2.0/recipes.html#the-scm-attribute

The entire conandata.yml is overwritten and becomes a file with only a {} in it. This makes it impossible to follow the example.

PTAL @prince-chrismc @uilianries

prince-chrismc commented 1 year ago

Thanks for raising this!

uilianries commented 1 year ago

@jwillikers Which hook specifically? Remember hooks repository contains several hooks.

jwillikers commented 1 year ago

I believe it's the conan-center hooks specifically.

prince-chrismc commented 1 year ago

There's 70+ of those so it will take some work to investigate, is there a sample recipe with the issue?

jwillikers commented 1 year ago

Just ensure that the conan-center hooks are enabled and use conan new to generate a project. Replace the export_sources attribute with the example from the Conan docs: https://docs.conan.io/en/latest/migrating_to_2.0/recipes.html#id15.

The steps are below.

Set up the Conan Center hooks.

conan config install https://github.com/conan-io/hooks.git -sf hooks -tf hooks
conan config set hooks.conan-center

Create a directory for the test project.

mkdir hello
cd hello

Create a new Conan template project.

conan new hello/0.1 --template=meson_lib

Using this Meson template and substituting the export_sources attribute with the Git SCM functionality in the example yields the following conanfile.py file:

import os
from conan import ConanFile
from conan.tools.meson import MesonToolchain, Meson
from conan.tools.layout import basic_layout
from conan.tools.files import copy
from conan.tools.scm import Git
from conan.tools.files import load, update_conandata

class HelloConan(ConanFile):
    name = "hello"
    version = "0.1"

    # Binary configuration
    settings = "os", "compiler", "build_type", "arch"
    options = {"shared": [True, False], "fPIC": [True, False]}
    default_options = {"shared": False, "fPIC": True}

    def export(self):
        git = Git(self, self.recipe_folder)
        scm_url, scm_commit = git.get_url_and_commit()
        update_conandata(self, {"sources": {"commit": scm_commit, "url": scm_url}})

    def source(self):
        git = Git(self)
        sources = self.conan_data["sources"]
        git.clone(url=sources["url"], target=".")
        git.checkout(commit=sources["commit"])

    def config_options(self):
        if self.settings.os == "Windows":
            del self.options.fPIC

    def layout(self):
        basic_layout(self)

    def generate(self):
        tc = MesonToolchain(self)
        tc.generate()

    def build(self):
        meson = Meson(self)
        meson.configure()
        meson.build()

    def package(self):
        meson = Meson(self)
        meson.install()

    def package_info(self):
        self.cpp_info.libs = ["hello"]

Initialize a Git repository.

git init

Stage the files.

git add -A

Commit.

git commit -am "Initial commit"

Create a Conan package.

conan create . hello/0.1@

prince-chrismc commented 1 year ago

Thanks for the repo steps!

uilianries commented 1 year ago

@jwillikers Thank you for detailing your case.

We do not allow Git or even scm feature on CCI:

KB-H039 does not allow scm attribute, which uses Git implicitly: https://github.com/conan-io/hooks/blob/master/hooks/conan-center.py#L501
KB-H010: Does not allow git commands: https://github.com/conan-io/hooks/blob/master/hooks/conan-center.py#L984

Said that, using Git is an invalid case on ConanCenterIndex, which means, conan-center hook working properly, since we are not inclined to run Git there.

However, if this situation is affecting your local development, not related to ConanCenterIndex, I would suggest disabling conan-center-index hook instead.

jwillikers commented 1 year ago

@uilianries It would be helpful if those hooks you mentioned actually presented errors. In my case, building any project using the new Git SCM attribute fails with a completely opaque failure. When switching between internal and CCI projects, I can usually just disable hook errors with the environment variable as necessary, though I find CCI's hooks helpful for non-CCI packages, too.

Here's the output that shows how this fails. It took me several hours to figure out why this was breaking since there's no hook failure related to the Git SCM functionality.

[HOOK - conan-center.py] pre_export(): [DEPRECATED GLOBAL CPPSTD (KB-H001)] OK
[HOOK - conan-center.py] pre_export(): [REFERENCE LOWERCASE (KB-H002)] OK
[HOOK - conan-center.py] pre_export(): ERROR: [RECIPE METADATA (KB-H003)] Conanfile doesn't have 'url' attribute.  (https://github.com/conan-io/conan-center-index/blob/master/docs/error_knowledge_base.md#KB-H003) 
[HOOK - conan-center.py] pre_export(): ERROR: [RECIPE METADATA (KB-H003)] Conanfile doesn't have 'license' attribute.  (https://github.com/conan-io/conan-center-index/blob/master/docs/error_knowledge_base.md#KB-H003) 
[HOOK - conan-center.py] pre_export(): ERROR: [RECIPE METADATA (KB-H003)] Conanfile doesn't have 'description' attribute.  (https://github.com/conan-io/conan-center-index/blob/master/docs/error_knowledge_base.md#KB-H003) 
[HOOK - conan-center.py] pre_export(): ERROR: [RECIPE METADATA (KB-H003)] Conanfile doesn't have 'homepage' attribute.  (https://github.com/conan-io/conan-center-index/blob/master/docs/error_knowledge_base.md#KB-H003) 
[HOOK - conan-center.py] pre_export(): ERROR: [RECIPE METADATA (KB-H003)] Conanfile doesn't have 'topics' attribute.  (https://github.com/conan-io/conan-center-index/blob/master/docs/error_knowledge_base.md#KB-H003) 
[HOOK - conan-center.py] pre_export(): [HEADER_ONLY, NO COPY SOURCE (KB-H005)] OK
[HOOK - conan-center.py] pre_export(): [FPIC OPTION (KB-H006)] OK
[HOOK - conan-center.py] pre_export(): [VERSION RANGES (KB-H008)] OK
[HOOK - conan-center.py] pre_export(): [RECIPE FOLDER SIZE (KB-H009)] Total recipe size: 10.830078125 KB
[HOOK - conan-center.py] pre_export(): [RECIPE FOLDER SIZE (KB-H009)] OK
[HOOK - conan-center.py] pre_export(): [EXPORT LICENSE (KB-H023)] exports: None
[HOOK - conan-center.py] pre_export(): [EXPORT LICENSE (KB-H023)] exports: None
[HOOK - conan-center.py] pre_export(): [EXPORT LICENSE (KB-H023)] OK
[HOOK - conan-center.py] pre_export(): [TEST PACKAGE FOLDER (KB-H024)] OK
[HOOK - conan-center.py] pre_export(): [META LINES (KB-H025)] OK
[HOOK - conan-center.py] pre_export(): [CONAN CENTER INDEX URL (KB-H027)] OK
[HOOK - conan-center.py] pre_export(): [CMAKE MINIMUM VERSION (KB-H028)] OK
[HOOK - conan-center.py] pre_export(): [TEST PACKAGE - RUN ENVIRONMENT (KB-H029)] OK
[HOOK - conan-center.py] pre_export(): [SYSTEM REQUIREMENTS (KB-H032)] OK
[HOOK - conan-center.py] pre_export(): [CONANDATA.YML FORMAT (KB-H030)] OK
[HOOK - conan-center.py] pre_export(): [TEST PACKAGE - NO IMPORTS() (KB-H034)] OK
[HOOK - conan-center.py] pre_export(): [NO AUTHOR (KB-H037)] OK
[HOOK - conan-center.py] pre_export(): [NOT ALLOWED ATTRIBUTES (KB-H039)] OK
[HOOK - conan-center.py] pre_export(): [NO TARGET NAME (KB-H040)] OK
[HOOK - conan-center.py] pre_export(): [NO FINAL ENDLINE (KB-H041)] OK
[HOOK - conan-center.py] pre_export(): [NO REQUIRES.ADD() (KB-H044)] OK
[HOOK - conan-center.py] pre_export(): [DELETE OPTIONS (KB-H045)] OK
[HOOK - conan-center.py] pre_export(): [CMAKE VERBOSE MAKEFILE (KB-H046)] OK
[HOOK - conan-center.py] pre_export(): [CMAKE VERSION REQUIRED (KB-H048)] OK
[HOOK - conan-center.py] pre_export(): [CMAKE WINDOWS EXPORT ALL SYMBOLS (KB-H049)] OK
[HOOK - conan-center.py] pre_export(): [DEFAULT OPTIONS AS DICTIONARY (KB-H051)] OK
[HOOK - conan-center.py] pre_export(): [CONFIG.YML HAS NEW VERSION (KB-H052)] OK
[HOOK - conan-center.py] pre_export(): [PRIVATE IMPORTS (KB-H053)] OK
[HOOK - conan-center.py] pre_export(): [SINGLE REQUIRES (KB-H055)] OK
[HOOK - conan-center.py] pre_export(): [TOOLS RENAME (KB-H057)] OK
[HOOK - conan-center.py] pre_export(): [ILLEGAL CHARACTERS (KB-H058)] OK
[HOOK - conan-center.py] pre_export(): [CLASS NAME (KB-H059)] OK
[HOOK - conan-center.py] pre_export(): [NO CRLF (KB-H060)] OK
[HOOK - conan-center.py] pre_export(): [NO BUILD SYSTEM FUNCTIONS (KB-H061)] OK
[HOOK - conan-center.py] pre_export(): [TOOLS CROSS BUILDING (KB-H062)] OK
[HOOK - conan-center.py] pre_export(): [INVALID TOPICS (KB-H064)] OK
[HOOK - conan-center.py] pre_export(): [NO REQUIRED_CONAN_VERSION (KB-H065)] OK
[HOOK - conan-center.py] pre_export(): [TEST_TYPE MANAGEMENT (KB-H068)] OK
[HOOK - conan-center.py] pre_export(): [TEST PACKAGE - NO DEFAULT OPTIONS (KB-H069)] OK
[HOOK - conan-center.py] pre_export(): [MANDATORY SETTINGS (KB-H070)] OK
[HOOK - conan-center.py] pre_export(): [PYLINT EXECUTION (KB-H072)] OK
[HOOK - conan-center.py] pre_export(): [TEST V1 PACKAGE FOLDER (KB-H073)] OK
[HOOK - conan-center.py] pre_export(): [REQUIREMENT OVERRIDE PARAMETER (KB-H075)] OK
Exporting package recipe
hello/0.1: Calling export()
hello/0.1: WARN: Current commit 29c2ab15040edb57ec2f62e0e26dfc2a2068a8bd doesn't exist in remote origin
This revision will not be buildable in other computer
[HOOK - conan-center.py] post_export(): [CONANDATA.YML REDUCE (KB-H031)] Saving conandata.yml: {}
[HOOK - conan-center.py] post_export(): [CONANDATA.YML REDUCE (KB-H031)] New conandata.yml contents: {}

[HOOK - conan-center.py] post_export(): [CONANDATA.YML REDUCE (KB-H031)] OK
[HOOK - conan-center.py] post_export(): [DEFAULT SHARED OPTION VALUE (KB-H050)] OK
[HOOK - conan-center.py] post_export(): [LICENSE PUBLIC DOMAIN (KB-H056)] OK
hello/0.1: The stored package has not changed
hello/0.1: Using the exported files summary hash as the recipe revision: a406eb84fd3a0f90929f900f76e3a4ef 
hello/0.1: Source folder is corrupted, forcing removal
hello/0.1: Exported revision: a406eb84fd3a0f90929f900f76e3a4ef
Configuration (profile_host):
[settings]
arch=x86_64
build_type=Debug
compiler=gcc
compiler.cppstd=17
compiler.libcxx=libstdc++11
compiler.version=12
os=Linux
[options]
[build_requires]
[env]
[conf]
tools.env.virtualenv:auto_use=True
tools.cmake.cmaketoolchain:find_package_prefer_config=True
tools.cmake.cmaketoolchain:generator=Ninja
tools.system.package_manager:mode=install
tools.system.package_manager:sudo=True

Configuration (profile_build):
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=17
compiler.libcxx=libstdc++11
compiler.version=12
os=Linux
[options]
[build_requires]
[env]
[conf]
tools.env.virtualenv:auto_use=True
tools.cmake.cmaketoolchain:find_package_prefer_config=True
tools.cmake.cmaketoolchain:generator=Ninja
tools.system.package_manager:mode=install
tools.system.package_manager:sudo=True

hello/0.1: Forced build from source
hello/0.1 (test package): Installing package
Requirements
    hello/0.1 from local cache - Cache
Packages
    hello/0.1:79e112b065cbe2a4a0edce1a8d516321483d1705 - Build

Installing (downloading, building) binaries...
hello/0.1: WARN: Build folder is dirty, removing it: /var/home/jordan/.conan/data/hello/0.1/_/_/build/79e112b065cbe2a4a0edce1a8d516321483d1705
[HOOK - conan-center.py] pre_source(): [IMMUTABLE SOURCES (KB-H010)] OK
hello/0.1: Configuring sources in /var/home/jordan/.conan/data/hello/0.1/_/_/source/.
ERROR: hello/0.1: Error in source() method, line 25
    sources = self.conan_data["sources"]
    KeyError: 'sources'

conan-io / hooks

Hooks sabotage conandata.yml generated in export method #475