FindProtobuf.cmake

[Phibedy]

I would need the PROTOBUF_GENERATE_CPP() from FindProtobuf.cmake. I am not sure if it's already added/how to add it.

[memsharded]

That function is already in the CMake module: https://github.com/Kitware/CMake/blob/master/Modules/FindProtobuf.cmake

Though, it might not work, as such cmake module is not related to the conan package and will not find it. Maybe the package root can be provided before calling the find_package().

It would help to know what is your final goal, what is that function needed for. If it is for generating code from .proto files, I think it is much easier done from a conan recipe, check for example the code inside the test_package folder of the protobuf package

The conanfile is using the imports feature, but maybe the virtualenvs could be also very convenient

[Phibedy]

Yes it is for generating code from .proto files and use it in the same package. I will have a look at the test_package :)

[memsharded]

Great, please do so and tell me, I find it much more convenient, clear and easy to change in python (conanfile recipe) than using cmake syntax :)

[Phibedy]

Somehow I can't get it working: conan file

from conans import ConanFile, CMake

class LMSConan(ConanFile):
    name = "lms"
    version = "2.0"
    settings = "os", "compiler", "build_type", "arch"
    exports = "include/*","src/*","README.md","CMakeLists.txt","cmake/*","CMakeData.txt","messages/*"
    requires = "gtest/1.8.0@lms/stable","pugixml/1.7@lms/stable","tclap/1.2.1@lms/stable","Protobuf/2.6.1@memsharded/testing"
    generators = "cmake"

    def build(self):
        self.run('./bin/protoc ../messages/messages.proto --proto_path=../messages --cpp_out="."')
        cmake = CMake(self.settings)
        self.run('cmake %s %s' % (self.conanfile_directory, cmake.command_line))
        self.run("cmake --build . %s" % cmake.build_config)

    def package(self):
        self.copy("*.h", dst="include", src="include")
        self.copy("*.so", dst="lib")
        self.copy("*/lms", dst=".")

    def package_info(self):
        self.cpp_info.libs = ["lmscore"]

    def imports(self):
        self.copy("protoc.exe", "bin", "bin") # Windows
        self.copy("protoc", "bin", "bin") # Linux / Macos
        self.copy("libproto*.9.dylib", "bin", "bin") # Macos (when Protobuf:static=False)
        self.copy("libproto*.dll", "bin", "bin") # Windows (when Protobuf:static=False)
        self.copy("zlib*.dll", "bin", "bin") # Windows (when zlib:shared=True)

error

lms/2.0@lms/stable: Building your package in /home/phibedy/.conan/data/lms/2.0/lms/stable/build/634987f926e95a03a707e2b5c566e831a946416e
lms/2.0@lms/stable: Copying sources to build folder
lms/2.0@lms/stable: Generated cmake created conanbuildinfo.cmake
lms/2.0@lms/stable imports(): Copied 1 '' files: protoc
./bin/protoc: line 202: cd: /home/conan/.conan/data/Protobuf/2.6.1/memsharded/testing/build/cb5b0409ac2512af687c07272f75b03980a5cc5a/protobuf-2.6.1/src: No such file or directory
g++: error: google/protobuf/compiler/main.o: No such file or directory
g++: error: ./.libs/libprotobuf.so: No such file or directory
g++: error: ./.libs/libprotoc.so: No such file or directoryn
lms/2.0@lms/stable: ERROR: Package '634987f926e95a03a707e2b5c566e831a946416e' build failed
lms/2.0@lms/stable: WARN: Build folder /home/phibedy/.conan/data/lms/2.0/lms/stable/build/634987f926e95a03a707e2b5c566e831a946416e
ERROR: lms: Error 256 while executing ./bin/protoc ../../messages/messages.proto --proto_path=../../messages --cpp_out="."

The there are no files in .conan/data/lms/.../build/bin, but it says that it copied something. I wonder where the file gets copied to. As conan install .. does not call build, self.run('./bin/protoc ../messages/messages.proto --proto_path=../messages --cpp_out="."') is not called if I just want to use it in a cmake project. Copying all so files it says 9 were copied causes an empty lib directory: self.copy("*.so", dst="lib")

In addition I am not able to build Protobuf from source:

phibedy@phibedy--T530:~/Documents/programming/c++/lms_conan/lms$ conan install Protobuf/2.6.1@memsharded/testing --build=Protobuf
Requirements
    Protobuf/2.6.1@memsharded/testing from conan.io
    zlib/1.2.8@lasote/stable from conan.io
Packages
    Protobuf/2.6.1@memsharded/testing:cb5b0409ac2512af687c07272f75b03980a5cc5a
    zlib/1.2.8@lasote/stable:a79ecbb5554da8510d2c9d7856daeb24de6be0d9

zlib/1.2.8@lasote/stable: Already installed!
Protobuf/2.6.1@memsharded/testing: Installing package cb5b0409ac2512af687c07272f75b03980a5cc5a
Protobuf/2.6.1@memsharded/testing: WARN: Forced build from source
Protobuf/2.6.1@memsharded/testing: Building your package in /home/phibedy/.conan/data/Protobuf/2.6.1/memsharded/testing/build/cb5b0409ac2512af687c07272f75b03980a5cc5a
Protobuf/2.6.1@memsharded/testing: Copying sources to build folder
Protobuf/2.6.1@memsharded/testing: Generated cmake created conanbuildinfo.cmake
+ sed -i -e s/RuntimeLibrary="5"/RuntimeLibrary="3"/g;
           s/RuntimeLibrary="4"/RuntimeLibrary="2"/g; gtest/msvc/gtest-md.vcproj gtest/msvc/gtest.vcproj gtest/msvc/gtest_main-md.vcproj gtest/msvc/gtest_main.vcproj gtest/msvc/gtest_prod_test-md.vcproj gtest/msvc/gtest_prod_test.vcproj gtest/msvc/gtest_unittest-md.vcproj gtest/msvc/gtest_unittest.vcproj
+ autoreconf -f -i -Wall,no-obsolete
Can't exec "libtoolize": No such file or directory at /usr/share/autoconf/Autom4te/FileUtils.pm line 345, <GEN7> line 6.
autoreconf: failed to run libtoolize: No such file or directory
autoreconf: libtoolize is needed because this package uses Libtool

Protobuf/2.6.1@memsharded/testing: ERROR: Package 'cb5b0409ac2512af687c07272f75b03980a5cc5a' build failed
Protobuf/2.6.1@memsharded/testing: WARN: Build folder /home/phibedy/.conan/data/Protobuf/2.6.1/memsharded/testing/build/cb5b0409ac2512af687c07272f75b03980a5cc5a
ERROR: Protobuf: Error 256 while executing cd protobuf-2.6.1 && ./autogen.sh

I am using conan version 0.13.3

[memsharded]

I have to give a detailed look at this, but as a quick tip, it is not building properly from source because you need to install the tool, something like (ubuntu): sudo apt-get install build-essential libtool

[Phibedy]

That is what I thought, but as I am quite used to the fact, that conan installs everything for me, I just wondered. Thank you for having a look at it :)

Edit: If I use conan install .. (build is in the src directory) and call protoc manually it works

[memsharded]

Well, maybe it shouldn't be conan responsibility to install Visual Studio.... :) But it is true that now with installer packages, like cmake_installer or mingw_installer, and virtualenvs, there is space to package some tools too. But most likely libtool as provided by apt-get is not worth doing a package for it.

So if I understood your last msg, you still have the problem of not being able to automatically generate the code from the .proto file? Do you have a full working repo, not only the conanfile.py you copied above, so I can try it? Please also tell me your setup: OS (ubuntu?), compiler/version. Thanks!

[Phibedy]

Of course it should not be conans responsibility, but for opencv for example it asked me if I want to install something. Of course you can't cover everything, I just noticed it and that's why I added it :)

I am using gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4 git repository: https://github.com/lms-org/lms/tree/conan_test

Edit: @memsharded sorry, forgot to add the remote conan remote add lms http://mineforce.de:9300

Thank you for support

[memsharded]

I have been doing some research, and made it work. First, consider there are 2 possible ways to handle the LMS:

• As a project in your user space. You don't need, name, version, package_info(), package(), source() here
• As a package, inside the local conan cache. In this case, I strongly encourage to use the facilities of conan test_package

I will start with the first one, as a project, to explain some things. The key would be the following code:

class LMSConan(ConanFile):
    name = "lms"
    version = "2.0"
    settings = "os", "compiler", "build_type", "arch"
    exports = "include/*","src/*","README.md","CMakeLists.txt","cmake/*","CMakeData.txt"
    requires = "Protobuf/2.6.1@memsharded/testing"
    generators = "cmake", "txt"

    def build(self):
    library_path = ":".join(self.deps_cpp_info.lib_paths)
    self.run('LD_LIBRARY_PATH=%s ./bin/protoc ../messages/messages.proto --proto_path=../messages --cpp_out="../include"' % library_path)
        cmake = CMake(self.settings)
        self.run('cmake %s %s' % (self.conanfile_directory, cmake.command_line))
        self.run("cmake --build . %s" % cmake.build_config)

I can run it (it will fail in the compilation, but this seems a build problem, not a protobuf issue) with:

$ mkdir build && cd build
$ conan install ..
$ conan build ..

For conan build to work properly, you need to add the txt generator. You can also do it in the command line if you dont want to add it to your recipe: $ conan install .. -g=txt

It is necessary to implement the code-generation functionality in the build() method, just before the actual build. The syntax was practically fine, but with one workaround: the Protobuf binary in linux hardcodes the RPATH to the libraries, so if you download the pre-built package, it won't find them. It is necessary to add the LD_LIBRARY_PATH to help it find its shared libraries. I have tried to figure out how to fix it in the Protobuf package, but the configure file has 19.7K lines of code, and I haven't been able to do it.

For building it as a package, we need to introduce some changes, as using self.conanfile_directory instead of relative paths like ../include, but please, try this first and when it works we will move forward. Thanks!

[Phibedy]

Thank you, I added txt generator but it still fails, the path seems to be broken.

I removed all conan packages and did the following:

conan remote add lms http://mineforce.de:9300
git clone https://github.com/lms-org/lms/tree/conan_test
cd lms
mkdir build && cd build
conan install .. --build=missing
conan build ..

the error

./bin/protoc: line 202: cd: /home/conan/.conan/data/Protobuf/2.6.1/memsharded/testing/build/cb5b0409ac2512af687c07272f75b03980a5cc5a/protobuf-2.6.1/src: No such file or directory
g++: error: google/protobuf/compiler/main.o: No such file or directory
g++: error: ./.libs/libprotobuf.so: No such file or directory
g++: error: ./.libs/libprotoc.so: No such file or directorynERROR: Error 256 while executing LD_LIBRARY_PATH=/home/phibedy/.conan/data/Protobuf/2.6.1/memsharded/testing/package/cb5b0409ac2512af687c07272f75b03980a5cc5a/lib:/home/phibedy/.conan/data/pugixml/1.7/lms/stable/package/2a33b7e8e997a8270b725821f15438eb49c1226e/lib:/home/phibedy/.conan/data/tclap/1.2.1/lms/stable/package/de9c231f84c85def9df09875e1785a1319fa8cb6/lib:/home/phibedy/.conan/data/gtest/1.8.0/lms/stable/package/a79ecbb5554da8510d2c9d7856daeb24de6be0d9/lib:/home/phibedy/.conan/data/zlib/1.2.8/lasote/stable/package/a79ecbb5554da8510d2c9d7856daeb24de6be0d9/lib ./bin/protoc ../messages/messages.proto --proto_path=../messages --cpp_out="../include"

There is no .libs directory in /home/phibedy/.conan/data/Protobuf/2.6.1/memsharded/testing/package/cb5b0409ac2512af687c07272f75b03980a5cc5a/lib just libprotobuf-lite.so libprotobuf-lite.so.9 libprotobuf-lite.so.9.0.1 libprotobuf.so libprotobuf.so.9 libprotobuf.so.9.0.1 libprotoc.so libprotoc.so.9 libprotoc.so.9.0.1.

[memsharded]

Yes, you are right, it is failing. I don't know what I did, but I made it work at some point, so I guess I messed up. I'll keep working on this.

[memsharded]

In any case, as it is something related to the paths of the pre-built packages, you should be fine running --build=Protobuf, so it builds from sources, and it doesn't take much. So at least you can keep working on your package/project while we investigate this issue.

[Phibedy]

Thank you, atm I just compile the proto files manually and export them as I would like to be able to install it without any extras. Is it possible to add --build=Protobuf inside the lms-conanfile.py? I know it is possible inside the protobuf conanfile but as not everyone uses the the protoc compiler, I don't think it's good practice to add it.

The test_package looks promising, I will give it a try :)

[memsharded]

No, it is not something that can be decided from a package, to rebuild other package. It has to be the self package (the build_policy is basically intended for header-only or latest/master/head versions), or the end user.

But in any case, the code needs the protobuf library to link with, so it needs to depend on Protobuf, which already has inside the "protoc" compiler. The only thing is that it doesn't work fine in Linux due to rpath issues.

Yes, test_package is really useful!, I add it to all packages, even simple header only packages, to ensure that files are correctly being copied to the package, the include paths are working fine, etc.

[memsharded]

Another quick tip, if you don't mind statically linking Protobuf. As it is a problem with protoc and its dynamic libraries, you can just opt for the statically linked package:

$ conan install ... -o Protobuf:shared=False

I have just tested it and it works fine, generates the source code from the .proto automatically. I keep investigating anyway for the shared one.

[pepsiman]

This works for me:

add_custom_command(
    OUTPUT
    foo.pb.cc
    foo.pb.h
    COMMAND ${CONAN_BIN_DIRS_PROTOBUF}$<$<CONFIG:Debug>:${CONAN_BIN_DIRS_PROTOBUF_DEBUG}>$<$<NOT:$<CONFIG:Debug>>:${CONAN_BIN_DIRS_PROTOBUF_RELEASE}>/protoc --proto_path ${CMAKE_CURRENT_SOURCE_DIR} ${CMAKE_CURRENT_SOURCE_DIR}/foo.proto --cpp_out=.
    DEPENDS foo.proto
    )

add_library(foo)
target_sources(
    foo
    PRIVATE
    foo.pb.cc
    foo.pb.h
    )
target_link_libraries(
    foo
    PRIVATE
    CONAN_PKG::protobuf
    )

[memsharded]

H @pepsiman

This recipe is not maintained anymore. The current efforts for community recipes is now in https://github.com/conan-io/conan-center-index. Cheers!

[pepsiman]

Indeed, just commenting so that Google has a solution to the problem.