search

Azure / azure-sdk-for-cpp

This repository is for active development of the Azure SDK for C++. For consumers of the SDK we recommend visiting our versioned developer docs at https://azure.github.io/azure-sdk-for-cpp.
MIT License
174 stars 125 forks source link

Cannot configure/use azure-sdk-for-cpp as per README #5222

Closed felix-kolbe closed 1 month ago

felix-kolbe commented 9 months ago

Describe the bug I followed the README.md to include azure-sdk-for-cpp in my project, but the configuring step fails.

First I used the FetchContent approach to skip any vcpkg. I created a test project for the sample to have and share a MRE: my azure-sdk-for-cpp-demo.

(I also tried to use the vcpkg way, as per README.md, but it was not working, too.)

Exception or Stack Trace Excerpt from build.log:

-- Using vendored uamqp
CMake Error at build/_deps/azuresdk-src/sdk/core/azure-core-amqp/vendor/azure-uamqp-c/CMakeLists.txt:56 (find_package):
   Could not find a package configuration file provided by
   "azure_macro_utils_c" with any of the following names:

     azure_macro_utils_cConfig.cmake
     azure_macro_utils_c-config.cmake

To Reproduce Steps to reproduce the behavior:

rm -rf build && cmake -S . -B build 2>&1 | tee logs/build.log

Code Snippet The CMakeLists.txt I used. Includes other variants I tried, as per README.

cmake_minimum_required(VERSION 3.13)

project(azure-sdk-for-cpp-demo LANGUAGES CXX)

include(FetchContent)
FetchContent_Declare(
    azuresdk
    GIT_REPOSITORY      https://github.com/Azure/azure-sdk-for-cpp.git
    GIT_TAG             main
)
FetchContent_GetProperties(azuresdk)
if(NOT azuresdk_POPULATED)
    FetchContent_Populate(azuresdk)
    # Adding all Azure SDK libraries
    add_subdirectory(${azuresdk_SOURCE_DIR} ${azuresdk_BINARY_DIR} EXCLUDE_FROM_ALL)

    # Adding one Azure SDK Library only (Storage blobs)
    # add_subdirectory(${azuresdk_SOURCE_DIR}/sdk/storage/azure-storage-blobs ${azuresdk_BINARY_DIR} EXCLUDE_FROM_ALL)

    # Adding sub components in their own binary dirs
    # add_subdirectory(${azuresdk_SOURCE_DIR}/sdk/core/azure-core ${azuresdk_BINARY_DIR}/azure-core EXCLUDE_FROM_ALL)
    # add_subdirectory(${azuresdk_SOURCE_DIR}/sdk/storage/azure-storage-common ${azuresdk_BINARY_DIR}/azure-storage-common EXCLUDE_FROM_ALL)
    # add_subdirectory(${azuresdk_SOURCE_DIR}/sdk/storage/azure-storage-blobs ${azuresdk_BINARY_DIR}/azure-storage-blobs EXCLUDE_FROM_ALL)

    # Adding the upper level components
    # add_subdirectory(${azuresdk_SOURCE_DIR}/sdk/core EXCLUDE_FROM_ALL)
    # add_subdirectory(${azuresdk_SOURCE_DIR}/sdk/storage EXCLUDE_FROM_ALL)
endif()

add_executable(${PROJECT_NAME} sample.cpp)
target_link_libraries(${PROJECT_NAME} PRIVATE Azure::azure-storage)
# target_link_libraries(${PROJECT_NAME} PRIVATE Azure::azure-identity)

Expected behavior A successful CMake configuration pass. And eventually a built sample executable.

Setup (please complete the following information):

Additional context I suspect the issue is within the sub component sdk/core/azure-core-amqp which somehow brings its own azure-uamqp-c, but the latter is plainly copied in there, so its submodules from .gitmodules -- which include the missing azure-macro-utils-c -- cannot be initialized and found at all.

Slightly related: #4993

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

ahsonkhan commented 9 months ago

Thanks @felix-kolbe for providing a detailed description of the issue you are hitting, especially with an MRE we can use to debug. We'll take a look and get back to you, and update the getting started docs accordingly.

Until then, can you clarify what you are trying to do with the repo. Do you primarily want to use the storage SDK itself or do you want to also use the EventHubs library (which brings in amqp dependencies)?

felix-kolbe commented 9 months ago

@ahsonkhan Thank you for your fast reply.

My use case

I'm already using the azure-iot-sdk-c, incl. its 'traditional' simple blob upload functionality. Although its latest releases have new functions regarding azure storage, those apparently cannot resuming previously uploaded blob parts, so I need to add this SDK.

My update

I got it working by using an older SDK revision 3a26df28 that I got to know as working. I had to activate DISABLE_AZURE_CORE_OPENTELEMETRY though. The diff with log

Yet I don't know the problematic change since that revision, and don't see how it did pass integration -- or what the difference in my setup could be.

Jinming-Hu commented 9 months ago

I think this is similar to https://github.com/Azure/azure-sdk-for-cpp/issues/4649

I also tried to build the package without vcpkg when handling that issue, but didn't succeed. I think the conclusion we got from that issue is that we do not support building SDK without vcpkg.

ahsonkhan commented 9 months ago

The readme doc is likely outdated since we recently introduced the amqp dependency (since that isn't required to use storage anyway). @felix-kolbe an you share more details about your scenario so we can understand the actual intent better. Which language is your app written in? For example, do you primarily want to use the Storage SDK for C++ to talk to the Azure Storage service which has all the functionality exposed? Do you want to keep using the azure-iot-sdk-c in conjunction with the C++ SDK, or do you prefer to replace it wholesale?

First I used the FetchContent approach to skip any vcpkg.

We also want to understand this a bit more. What made you skip vcpkg and go for the FetchContent approach?

(I also tried to use the vcpkg way, as per README.md, but it was not working, too.)

Secondly, and more importantly, how are you using vcpkg and what issue do you run into with that? I would love to understand your experience with that, and what blockers you faced, so we know the docs and experience gap there, and fix it!

I had to activate DISABLE_AZURE_CORE_OPENTELEMETRY though.

I am surprised disabling OpenTelemetry fixed the issue (temporarily), since that shouldn't have any dependency on AMQP, and your initial build error is about missing that (and azure_macro_utils_c). That seems orthogonal. The revision you linked to that worked is quite old (1.7.0 (2022-06-30) - before amqp was introduced), so we certainly want to make sure you can have it work with latest main.

csi-amolpawar commented 8 months ago

@ahsonkhan

The readme doc is likely outdated since we recently introduced the amqp dependency (since that isn't required to use storage anyway

To provide clarity, could you share the link to the updated documentation? Additionally, regarding the amqp dependency, could you specify from which version of azure_storage_blobs it's no longer required?

LarryOsterman commented 8 months ago

@ahsonkhan

The readme doc is likely outdated since we recently introduced the amqp dependency (since that isn't required to use storage anyway

To provide clarity, could you share the link to the updated documentation? Additionally, regarding the amqp dependency, could you specify from which version of azure_storage_blobs it's no longer required?

If you're using vcpkg, then azure storage has never had a dependency on AMQP.

If you're cloning the azure core repo, then the azure core repo no longer has a dependency on uAMQP as of (at least) commit 37e1952f71a3bf0b989873d136f4a671484f5c25.

csi-amolpawar commented 8 months ago

I'm using azure-storage-blobs_12.10.0 version branch to build --target azure-storage-blobs

LarryOsterman commented 8 months ago

I'm using azure-storage-blobs_12.10.0 version branch to build --target azure-storage-blobs

Ok. If you're cloning the repo at the azure-storage-blobs 12.10 checkpoint, it means that you're cloning the entire azure SDK for C++ repo. It also means that you're building the entire repo (all 600+ files) to build storage (50-70 files).

Is vcpkg a viable solution for your application? That guarantees that you will only get the package you want without all the additional stuff that other packages require.

csi-amolpawar commented 8 months ago

@LarryOsterman I have an impression that when I use --target azure-storage-blobs flag, it should build only libazure-storage-blobs.so and its dependencies. I''m not expecting to build all. I won't build azure-storage-files-datalake, azure-storage-file-shares, azure-storage-queues and not required in my application

Yes, Initially I looked for the vcpkg solution but didn't work, see my comment https://github.com/Azure/azure-sdk-for-cpp/issues/4993#issuecomment-1882463539.

LarryOsterman commented 8 months ago

@LarryOsterman I have an impression that when I use --target azure-storage-blobs flag, it should build only libazure-storage-blobs.so and its dependencies. I''m not expecting to build all. I won't build azure-storage-files-datalake, azure-storage-file-shares, azure-storage-queues and not required in my application

Yes, Initially I looked for the vcpkg solution but didn't work, see my comment #4993 (comment).

That is the case - it will only build azure storage. The problem here is that because you're invoking cmake from the root, it needs to understand the dependency graph for every package in the repo, including ones that you don't need or want.

Once the dependency graph is understood, you can ask it to only build one component, but the problems you're seeing stem from building from the root.

Normally when you're building the entire repo manually, the cmakelists.txt use vcpkg to satisfy the dependencies, so there isn't a problem, but since you can't use vcpkg that doesn't work.

@BillyONeal, do you have any ideas why Amol can't use vcpkg in his solution? The output from #4993 seems to lead me to believe that vcpkg didn't know about the storage dependency, which is weird if they have a manifest file.

BillyONeal commented 8 months ago

@BillyONeal, do you have any ideas why Amol can't use vcpkg in his solution? The output from #4993 seems to lead me to believe that vcpkg didn't know about the storage dependency, which is weird if they have a manifest file.

Without seeing exactly what the OP did I'm not sure what happened. It could be that they aren't using vcpkg or asked vcpkg to build for a different platform than the one that they're using. It could be that they didn't declare the correct dependency in their manifest. It could be the azure storage port itself is missing dependencies so that when CMake considers configs it installs, one of the find_dependencys inside fails. etc.

csi-amolpawar commented 8 months ago

@BillyONeal / @LarryOsterman

Please refer my sample code used in public repo azure-storage-vcpkg-demo and you can see failure https://github.com/amolpawarap/azure-storage-vcpkg-demo/actions/runs/7486289707/job/20376339343

bzhou-sw commented 6 months ago

I hit the same build issue when I move from Added checks to help diagnose intermittent globalCleanUp test failure (4953) to tag azure-core_1.11.2.

I cloned the repo from github, and commented out non-needed components in top CMakeLists.txt so we only build core and storage. It worked before, but now it failed at:

-- Using vendored uamqp
CMake Error at sdk/core/azure-core-amqp/vendor/azure-uamqp-c/CMakeLists.txt:56 (find_package):
  Could not find a package configuration file provided by
  "azure_macro_utils_c" with any of the following names:

    azure_macro_utils_cConfig.cmake
    azure_macro_utils_c-config.cmake

  Add the installation prefix of "azure_macro_utils_c" to CMAKE_PREFIX_PATH
  or set "azure_macro_utils_c_DIR" to a directory containing one of the above
  files.  If "azure_macro_utils_c" provides a separate development package or
  SDK, be sure it has been installed.

-- Configuring incomplete, errors occurred!
See also "/opt/project/build/CMakeFiles/CMakeOutput.log".
ERROR: 1

The CMakeLists.txt changes:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 00c78a09..5353c04c 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -11,6 +11,10 @@ set(AZ_ALL_LIBRARIES ON)

 # Compile Options
 include(FolderList)
+set(ENV{AZURE_SDK_DISABLE_AUTO_VCPKG} 1)
+set(DISABLE_AZURE_CORE_OPENTELEMETRY ON)
+option(BUILD_SHARED_LIBS "Build shared libraries" ON)
+option(BUILD_SAMPLES "Build sampless" ON)
 SetGlobalOptions()

 if(FETCH_SOURCE_DEPS)
@@ -152,17 +156,27 @@ endif()

 # sub-projects
 add_subdirectory(sdk/core)
-add_subdirectory(sdk/attestation)
+#add_subdirectory(sdk/attestation)
 # AMQP doesn't work for UWP yet, and eventhubs depends on AMQP, so we cannot include eventhubs on UWP.
 if (NOT BUILD_WINDOWS_UWP)
-  add_subdirectory(sdk/eventhubs)
+       #add_subdirectory(sdk/eventhubs)
 endif()
-add_subdirectory(sdk/identity)
-add_subdirectory(sdk/keyvault)
+#add_subdirectory(sdk/identity)
+#add_subdirectory(sdk/keyvault)
 add_subdirectory(sdk/storage)
-add_subdirectory(sdk/template)
-add_subdirectory(sdk/tables)
+#add_subdirectory(sdk/template)
+#add_subdirectory(sdk/tables)
jepio commented 4 months ago

I configure the top-level cmake project like this and it lets me build on Linux against system libraries without vcpkg:

AZURE_SDK_DISABLE_AUTO_VCPKG=1 cmake -GNinja -DDISABLE_AZURE_CORE_OPENTELEMETRY=1 -DBUILD_WINDOWS_UWP=1 ..

This effectively skips configuring/building amqp and eventhubs.

LarryOsterman commented 4 months ago

As the contributing.md file mentions, if you enable AZURE_SDK_DISABLE_AUTO_VCPKG, it tells the Azure SDK that you take responsibility for managing all the dependencies for the repo.

That set of dependencies will change over time as the contents of the SDK change and new packages are added and/or removed.

I'd like to understand why your solution cannot take advantage of vcpkg. Could you help us understand why you've disabled all use of vcpkg in your solution?

csi-amolpawar commented 4 months ago

@LarryOsterman Yes AZURE_SDK_DISABLE_AUTO_VCPKG need to enable and need to manage all the dependencies.

To take advantage of vcpkg, please guide us to fix the issue in demo created in GitHub workflow https://github.com/amolpawarap/azure-storage-vcpkg-demo/actions/runs/8949642163/job/24584169803

Vcpkg integrate step.
CMAKE_TOOLCHAIN_FILE is not defined. Define it for the user.
AZURE_SDK_DISABLE_AUTO_VCPKG is not defined. Fetch a local copy of vcpkg.
Vcpkg commit string used: 43cf47eccfbe27006cf9534a5db809798f8c37fe
Vcpkg integrate step - DONE.
CMake Error at CMakeLists.txt:37 (find_package):
  Could not find a package configuration file provided by
  "azure-storage-blobs-cpp" with any of the following names:
    azure-storage-blobs-cppConfig.cmake
    azure-storage-blobs-cpp-config.cmake
  Add the installation prefix of "azure-storage-blobs-cpp" to
  CMAKE_PREFIX_PATH or set "azure-storage-blobs-cpp_DIR" to a directory
  containing one of the above files.  If "azure-storage-blobs-cpp" provides a
  separate development package or SDK, be sure it has been installed.
-- Configuring incomplete, errors occurred!
csi-amolpawar commented 4 months ago

In my case, we use customised tool-chain which has the dependency required for azure-sdk-for-cpp storage. Hence we have disabled vcpkg.

github-actions[bot] commented 1 month ago

Hi @felix-kolbe. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

github-actions[bot] commented 1 month ago

Hi @felix-kolbe, since you haven’t asked that we /unresolve the issue, we’ll close this out. If you believe further discussion is needed, please add a comment /unresolve to reopen the issue.