search

Azure / azure-sdk-for-cpp

This repository is for active development of the Azure SDK for C++. For consumers of the SDK we recommend visiting our versioned developer docs at https://azure.github.io/azure-sdk-for-cpp.
MIT License
181 stars 127 forks source link

Unable to build azure-storage-blobs from version 12.8.0 #4993

Open csi-amolpawar opened 1 year ago

csi-amolpawar commented 1 year ago

Exception or Stack Trace

Observing the below error while compiling the azure-storage-blobs-12.8.0

  Found package configuration file:

    /usr/local/cmake/uamqpConfig.cmake

  but it set uamqp_FOUND to FALSE so package "uamqp" is considered to be NOT
  FOUND.  Reason given by package:

  The following imported targets are referenced, but are missing:
  aziotsharedutil

-- Configuring incomplete, errors occurred!
See also "/code/repo/thirdparty/azure-sdk-for-cpp/.build/CMakeFiles/CMakeOutput.log".

To Reproduce Build the latest or 12.8.0 azure-storage-blobs

Code Snippet Unable to find dependent packages azure-uamqp-c installed at system path /usr/local/ or custom path

Expected behavior CMake build should work w/o any configuration errors and build successful

Setup:

Additional context

  1. azure-uamqp-c is built succesfully with below 
    
[root@02f90f531dbc .build]# make install

[root@02f90f531dbc .build]# ls -l /usr/local/include/ total 16 drwxr-xr-x 2 root root 4096 Sep 28 09:40 azure_c_shared_utility drwxr-xr-x 2 root root 4096 Sep 28 09:40 azure_macro_utils drwxr-xr-x 3 root root 4096 Sep 28 09:40 azureiot drwxr-xr-x 2 root root 4096 Sep 28 09:40 umock_c

[root@02f90f531dbc .build]# ls -l /usr/local/lib64/ total 1348 drwxr-xr-x 1 root root 4096 Aug 9 2021 bpf drwxr-xr-x 3 root root 4096 Sep 28 09:40 cmake -rw-r--r-- 1 root root 533914 Sep 28 09:40 libaziotsharedutil.a -rw-r--r-- 1 root root 706972 Sep 28 09:40 libuamqp.a -rw-r--r-- 1 root root 126358 Sep 28 09:40 libumock_c.a

2. `azure-sdk-for-cpp` is bulid with

[root@02f90f531dbc .build]#cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_STANDARD=20 -DBUILD_TESTING=OFF -DBUILD_SHARED_LIBS=ON -DCURL_INCLUDE_DIR="${AZG_HOME}/tools/include" -DCURL_LIBRARY="${AZG_HOME}/tools/lib/libcurl.so" -DDISABLE_AZURE_CORE_OPENTELEMETRY=ON ..



I tried to pass CMake flag `CMAKE_PREFIX_PATH:STRING=${AZURE_UAMQP_INSTALL_PREFIX}` with custom path for installed `azure-uamqp-c` targets in step (2)
RickWinter commented 1 year ago

Are you building using vcpkg or are you building from the repo?

csi-amolpawar commented 1 year ago

@RickWinter

I'm building from the repo and has the issue since it is dependent on the azure-uamq-c.

ahsonkhan commented 1 year ago

@csi-amolpawar, this is a bit surprising since we successfully build this repo from the root in our CI for Windows, Mac, and Linux distro (Ubuntu), and storage-blobs doesn't itself have any package dependency on azure-uamqp-c (from an isolated package perspective).

Either the declaration and setup of the dependency graph of azure-uamqp-c and aziotsharedutil has some issue specifically on RHEL OR there seems to be some difference in how you are consuming and building the repo and how we build/test the repo (and expect customers to acquire).

Can you please share more info on how your environment is setup and where all the dependent repos and packages live (somehow you have placed them in /usr/local/include/ and expect the build to find it there)? Is azure-sdk-for-cpp a git submodule of your repo/project?

Also, is using vcpkg a feasible solution for you to manage the package dependency? We'd love to learn more about your constraints there as well, as to why you chose to git clone instead.

csi-amolpawar commented 1 year ago

@ahsonkhan,

Thanks for your comment. Please find more details below

We could see the sdk/core/azure-core-amqp depends on the azure-uamqp-c (which additionally requires azure-c-shared-utility azure-c-testrunnerswitcher azure-ctest azure-macro-utils-c umock-c) and required while building the azure-storage-blobs.

About the environment used,

  1. The RHEL 9 platform with customised GCC13.1 tool-chain and on the same platform we were able to build azure-storage-blobs successfully before the 12.8.0 version.

  2. When trying to build the latest version >= 12.8.0, a. First build and install azure-uamqp-c with CMake's default install prefix path /usr/local/include/. The generated output is placed in /usr/local/include and /usr/local/lib due to the non-system package. b. Building azure-storage-blob using below

    cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_STANDARD=20 -DBUILD_TESTING=OFF -DBUILD_SHARED_LIBS=ON -DCURL_INCLUDE_DIR="${AZG_HOME}/tools/include" -DCURL_LIBRARY="${AZG_HOME}/tools/lib/libcurl.so" -DDISABLE_AZURE_CORE_OPENTELEMETRY=ON
make install

  3. We pull azure-sdk-for-cpp as part of the build system and generate respective libraries and consume.

  4. Initially, we tried to use vcpkg, but couldn't. This could be due to we want to use a custom toolchain (GCC13.1) and the latest libcurl, libnghttp2, etc on the RHEL 9 platform at a custom path.

LarryOsterman commented 1 year ago

Silly question: Have you tried building the azure SDK with --target azure-storage-blobs?

csi-amolpawar commented 1 year ago

@LarryOsterman

We are failing in the cmake configuration step only cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_STANDARD=20 -DBUILD_TESTING=OFF -DBUILD_SHARED_LIBS=ON -DCURL_INCLUDE_DIR="${AZG_HOME}/tools/include" -DCURL_LIBRARY="${AZG_HOME}/tools/lib/libcurl.so" -DDISABLE_AZURE_CORE_OPENTELEMETRY=ON

So can't check cmake --build . --target azure-storage-blobs.

BillyONeal commented 1 year ago

5. Initially, we tried to use vcpkg, but couldn't. This could be due to we want to use a custom toolchain (GCC13.1) and the latest libcurl, libnghttp2, etc on the RHEL 9 platform at a custom path.

You should be able to do this with vcpkg by providing empty overlay ports for these dependencies. See my colleague @data-queue 's article here: https://devblogs.microsoft.com/cppblog/using-system-package-manager-dependencies-with-vcpkg/

csi-amolpawar commented 1 year ago

@BillyONeal Thanks for the input. We'll have a look and update.

csi-amolpawar commented 10 months ago

I've diligently followed the guidelines outlined in the CMake + Vcpkg manifest mode documentation, but unfortunately, I'm encountering issues.

Additionally, it seems that both solution whether it's using CMake with Vcpkg or CMake with Fetch_Content including their respective dependencies (Installed through Vcpkg or Independently) should ideally function seamlessly.

Also note that this issue, we were facing since storage has dependency on the uampq which introduced from version azure-storage-blobs_12.8.0

LarryOsterman commented 10 months ago

I'm a tiny bit confused. Azure storage vcpkg package shouldn't have a dependency on uamqp. The Azure SDK for C++ source repo has a dependency on uamqp, but the azure storage vcpkg package shouldn't have any such dependency.

csi-amolpawar commented 10 months ago

It is getting build from Azure::azure-core

...
add_subdirectory(azure-core)

if (NOT BUILD_WINDOWS_UWP)
  message(STATUS "Including AMQP library")
  add_subdirectory(azure-core-amqp)
endif()
...

While working on the RHEL9 platform, the build includes azure-core-amqp, which is directly reliant on uamqp and required by azure-messaging-eventhubs

This observation leads to the conclusion that azure-storage-blobs doesn't have a direct dependency on uamqp.

Please clarify target azure-core-amqp is intended to be built on which non universal windows platform (UWP) and how we can avoid building it on linux platforms if not required.

LarryOsterman commented 10 months ago

It is getting build from Azure::azure-core

...
add_subdirectory(azure-core)

if (NOT BUILD_WINDOWS_UWP)
  message(STATUS "Including AMQP library")
  add_subdirectory(azure-core-amqp)
endif()
...

While working on the RHEL9 platform, the build includes azure-core-amqp, which is directly reliant on uamqp and required by azure-messaging-eventhubs

This observation leads to the conclusion that azure-storage-blobs doesn't have a direct dependency on uamqp.

Please clarify target azure-core-amqp is intended to be built on which non universal windows platform (UWP) and how we can avoid building it on linux platforms if not required.

If you're using vcpkg, then you will only build azure-core-amqp if you include azure-core-amqp in your applications vcpkg manifest.

If you're attempting to build the Azure SDK from the source repo (using cmake's FETCH_CONTENT or similar mechanisms (git submodules), then you are implicitly building the entire API surface for the repo, including all the other packages needed.

Have you tried following @BillyONeal's suggestion about creating a custom overlay port for your build environment that includes your specific requirements?

ahsonkhan commented 10 months ago

I've diligently followed the guidelines outlined in the CMake + Vcpkg manifest mode documentation, but unfortunately, I'm encountering issues.

@csi-amolpawar what issues are you encountering? Can you share logs or other info.

It might be worth it to try the basic vcpkg install azure-storage-blob-cpp command to acquire the SDK. Is that a possible solution for you?

csi-amolpawar commented 10 months ago

@ahsonkhan I encountered below error 

[root@1e643bcdf477 .build]# cmake ..
-- The C compiler identification is GNU 11.4.1
-- The CXX compiler identification is GNU 13.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/anzograph/tools/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
Vcpkg integrate step.
CMAKE_TOOLCHAIN_FILE is not defined. Define it for the user.
AZURE_SDK_DISABLE_AUTO_VCPKG is not defined. Fetch a local copy of vcpkg.
Vcpkg commit string used: 43cf47eccfbe27006cf9534a5db809798f8c37fe
Vcpkg integrate step - DONE.
CMake Error at CMakeLists.txt:60 (find_package):
  Could not find a package configuration file provided by
  "azure-storage-blobs-cpp" with any of the following names:

    azure-storage-blobs-cppConfig.cmake
    azure-storage-blobs-cpp-config.cmake

  Add the installation prefix of "azure-storage-blobs-cpp" to
  CMAKE_PREFIX_PATH or set "azure-storage-blobs-cpp_DIR" to a directory
  containing one of the above files.  If "azure-storage-blobs-cpp" provides a
  separate development package or SDK, be sure it has been installed.

-- Configuring incomplete, errors occurred!
See also "/code/repo/samples/azure_demo/.build/CMakeFiles/CMakeOutput.log"

I've not expected above error in the find_package as followed all steps from documentation

Note:

  1. I followed the steps mentioned in the README
  2. The azure-storage-blobs-cpp is installed successfully using vcpkg when it is manually setup.