Seemingly missing libcurl.so when building azure-device-update in yocto (zeus)

[SimonSimCity]

In the past 2 days I occasionally got the following error when trying to build azure-device-update in yocto (zeus) after applying the patches described in #79:

yocto@5f0b8c0979c2:~/project/build$ bitbake azure-device-update
Loading cache: 100% |########################################################################################################################################################################| Time: 0:00:00
Loaded 3735 entries from dependency cache.
Parsing recipes: 100% |######################################################################################################################################################################| Time: 0:00:01
Parsing of 2569 .bb files complete (2562 cached, 7 parsed). 3741 targets, 134 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION           = "1.44.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "arm-poky-linux-gnueabi"
MACHINE              = "raspberrypi4"
DISTRO               = "poky"
DISTRO_VERSION       = "3.0.4"
TUNE_FEATURES        = "arm vfp cortexa7 neon vfpv4 thumb callconvention-hard"
TARGET_FPU           = "hard"
meta
meta-poky
meta-yocto-bsp       = "zeus:d88d62c20d7d8da85f02edb170dae0280624ad7e"
meta-oe
meta-python
meta-networking
meta-multimedia      = "zeus:2b5dd1eb81cd08bc065bc76125f2856e9383e98b"
meta-raspberrypi     = "HEAD:0e05098853eea77032bff9cf81955679edd2f35d"
meta-filesystems     = "zeus:2b5dd1eb81cd08bc065bc76125f2856e9383e98b"
meta-rust            = "HEAD:7ff669d8cedd83a2d3efb73073a63b0a7efffddc"
meta-virtualization  = "HEAD:7e5219669ff6f8e9c8c33ffd230e95a6b2b025f4"
meta-iotedge         = "zeus:9e2b459177f163aaeeb95d2e3fd3dc6ccfb9da44"
meta-swupdate        = "HEAD:08c05c4e63da3dd71f58d8dc7cd9464790fbceb4"
meta-azure-device-update

Initialising tasks: 100% |###################################################################################################################################################################| Time: 0:00:01
Sstate summary: Wanted 163 Found 157 Missed 6 Current 386 (96% match, 98% complete)
NOTE: Executing Tasks
NOTE: Setscene tasks completed
ERROR: azure-device-update-git-r0 do_compile: Execution of '/home/yocto/project/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/azure-device-update/git-r0/temp/run.do_compile.142569' failed with exit code 1:
ninja: error: '/home/yocto/project/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/azure-iot-sdk-c/git-r0/recipe-sysroot/usr/lib/libcurl.so', needed by 'src/agent/AducIotAgent', missing and no known rule to make it
WARNING: exit code 1 from a shell command.

ERROR: Logfile of failure stored in: /home/yocto/project/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/azure-device-update/git-r0/temp/log.do_compile.142569
Log data follows:
| DEBUG: Executing shell function do_compile
| NOTE: VERBOSE=1 cmake --build /home/yocto/project/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/azure-device-update/git-r0/build --target all -- -j 16
| ninja: error: '/home/yocto/project/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/azure-iot-sdk-c/git-r0/recipe-sysroot/usr/lib/libcurl.so', needed by 'src/agent/AducIotAgent', missing and no known rule to make it
| WARNING: exit code 1 from a shell command.
| ERROR: Execution of '/home/yocto/project/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/azure-device-update/git-r0/temp/run.do_compile.142569' failed with exit code 1:
| ninja: error: '/home/yocto/project/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/azure-iot-sdk-c/git-r0/recipe-sysroot/usr/lib/libcurl.so', needed by 'src/agent/AducIotAgent', missing and no known rule to make it
| WARNING: exit code 1 from a shell command.
|
ERROR: Task (/home/yocto/project/meta-azure-device-update/recipes-azure-device-update/azure-device-update/azure-device-update_git.bb:do_compile) failed with exit code '1'
NOTE: Tasks Summary: Attempted 2131 tasks of which 2129 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /home/yocto/project/meta-azure-device-update/recipes-azure-device-update/azure-device-update/azure-device-update_git.bb:do_compile
Summary: There was 1 ERROR message shown, returning a non-zero exit code.

It mentions a missing library in azure-iot-sdk-c whereby calling bitbake azure-iot-sdk-c runs through without any error.

After a couple of attempts, it often starts working again, but I was not yet able to reproduce it. If someone got an idea or even a solution, I'd be thrilled to hear about it.

[Voxel07]

@SimonSimCity I had the same issue. I will run into this whenever there is a change to the bblayer.conf. For example: I changed the name of the layer to match the base layers naming schema.

To fix it reliably, you can run this command:

bitbake -c cleanall meta-azure-device-update azure-device-update azure-iot-sdk-c

But I believe it should be enough to just clean the sdk. That sometimes wasn't enough for me.

[ValOlson]

Hi @SimonSimCity/ @Voxel07,

Thank you for raising this issue as well! We will investigate it.

Thanks @Voxel07 for sharing your feedback.

[pierrickcurt]

Hi @SimonSimCity, I had the same issue, this is due to a polluted cache. Try to modify your azure-iot-sdk-c_git.bb file in the function sysroot_stage_all_append

sysroot_stage_all_append () {
    sysroot_stage_dir ${D}${exec_prefix}/cmake ${SYSROOT_DESTDIR}${exec_prefix}/cmake
    sed -i "s@${WORKDIR}/recipe-sysroot@\${_WORKDIR}/recipe-sysroot@g" ${SYSROOT_DESTDIR}${exec_prefix}/cmake/azure_c_shared_utilityTargets-noconfig.cmake
}

[SimonSimCity]

@pierrickcurt Thanks for providing the patch, unfortunately I run into another error when adding the sed line to the function. To be fair, I did this after I encountered the error without cleaning anything. The error I got is:

Loading cache: 100% |##################################################################################################################################################################################################| Time: 0:00:00
Loaded 3345 entries from dependency cache.
Parsing recipes: 100% |################################################################################################################################################################################################| Time: 0:00:01
Parsing of 2281 .bb files complete (2277 cached, 4 parsed). 3355 targets, 125 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION           = "1.42.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "arm-poky-linux-gnueabi"
MACHINE              = "raspberrypi3"
DISTRO               = "poky"
DISTRO_VERSION       = "2.7.4"
TUNE_FEATURES        = "arm vfp cortexa7 neon vfpv4 thumb callconvention-hard"
TARGET_FPU           = "hard"
meta
meta-poky
meta-yocto-bsp       = "warrior:c8c383d958807b991e31d22f612ba2a81a3860a4"
meta-oe
meta-python
meta-networking
meta-multimedia      = "warrior:a24acf94d48d635eca668ea34598c6e5c857e3f8"
meta-raspberrypi     = "warrior:65b95ce742ae973b856a62974d0ec22e2caf03a3"
meta-swupdate        = "warrior:7ebe40f1a9b312476580fc74541231ac202feaa4"
meta-azure-device-update = "warrior:de5a72d48b6e8ad5ffe332a6738df1d6945d8e51"

Initialising tasks: 100% |#############################################################################################################################################################################################| Time: 0:00:01
Sstate summary: Wanted 201 Found 186 Missed 15 Current 1164 (92% match, 98% complete)
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
ERROR: azure-device-update-1.0+gitAUTOINC+743bb72f5b-r0 do_compile: Function failed: do_compile (log file is located at /home/yocto/project/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/azure-device-update/1.0+gitAUTOINC+743bb72f5b-r0/temp/log.do_compile.38451)
ERROR: Logfile of failure stored in: /home/yocto/project/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/azure-device-update/1.0+gitAUTOINC+743bb72f5b-r0/temp/log.do_compile.38451
Log data follows:
| DEBUG: Executing shell function do_compile
| NOTE: VERBOSE=1 cmake --build /home/yocto/project/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/azure-device-update/1.0+gitAUTOINC+743bb72f5b-r0/build --target all -- -j 16
| ninja: error: '/home/yocto/project/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/deliveryoptimization-sdk/1.0+gitAUTOINC+006c15f8a8-r0/recipe-sysroot/usr/lib/libboost_filesystem-mt.so', needed by 'src/agent/AducIotAgent', missing and no known rule to make it
| WARNING: exit code 1 from a shell command.
| ERROR: Function failed: do_compile (log file is located at /home/yocto/project/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/azure-device-update/1.0+gitAUTOINC+743bb72f5b-r0/temp/log.do_compile.38451)
ERROR: Task (/home/yocto/project/meta-azure-device-update/recipes-azure-device-update/azure-device-update/azure-device-update_git.bb:do_compile) failed with exit code '1'
NOTE: Tasks Summary: Attempted 3928 tasks of which 3901 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /home/yocto/project/meta-azure-device-update/recipes-azure-device-update/azure-device-update/azure-device-update_git.bb:do_compile
Summary: There was 1 ERROR message shown, returning a non-zero exit code.

It works after running the code supplied by @Voxel07 after removing the package meta-azure-device-update. Should I've done something specific before applying this line after encountering the error?

EDIT: Seems like this is something local ... I'll get back as I know more ...

[SimonSimCity]

This seems to be a much deeper problem ... I fixed mine by fixing all modules to a certain commit on each of the repos - see: https://github.com/SimonSimCity/meta-azure-device-update/commit/fb5cb2720b6ba2a4aa50c21d9e824baa5f9bfce9

[SimonSimCity]

Seems like it doesn't have to do with a particular version, but when the version changes, the sstate-cache of those packages need to be cleaned. With the release of the meta-layer for 0.8.0 there was a new package added which also requires those cleanings. Here is the new command we work with:

bitbake -c cleansstate azure-device-update azure-iot-sdk-c azure-sdk-for-cpp

EDIT: No need to call cleanall, calling cleansstate is enough. Just updated the command above.

[eshashah-msft]

Closing this issue as there has been no activity on this thread. Please feel free to open an issue if needed. Our Yocto layers published in this repo and will be maintained for our releases: https://github.com/Azure/iot-hub-device-update-yocto