libcurl dependency issue with Yocto build and pre-built .NET Core 2.0.6 arm binaries

[jamespettigrew]

Following the instructions in this repo, I've been trying to package the .NET Core 2.0.6 pre-built binaries as part of a Yocto build. All is well with the build process until I hit this error regarding the libcurl dependency :

ERROR: dotnetcore-2.0.6-r0 do_package_qa: QA Issue: /opt/dotnet/shared/Microsoft.NETCore.App/2.0.6/System.Net.Http.Native.so contained in package dotnetcore requires libcurl.so.4(CURL_OPENSSL_3), but no providers found in RDEPENDS_dotnetcore? [file-rdeps]
ERROR: dotnetcore-2.0.6-r0 do_package_qa: QA run found fatal errors. Please consider fixing them.
ERROR: dotnetcore-2.0.6-r0 do_package_qa: Function failed: do_package_qa

Curl and libcurl are specified in my DEPENDS and RDEPENDS_${PN}, and curl and libcurl.so.4 are definitely packaged into the rootfs so I'm not sure why there's an issue here.

curl -V

curl 7.54.1 (arm-poky-linux-gnueabi) libcurl/7.54.1 OpenSSL/1.0.2n zlib/1.2.11
Release-Date: 2017-06-14
Protocols: file ftp ftps http https 
Features: AsynchDNS IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets HTTPS-proxy 

I'm also hitting the same problem when trying to package a self-contained application instead of going down the shared framework route.

I'm using the 'rocko' branch of poky.

Any help would be greatly appreciated!

[Petermarcu]

@wfurt @karelz

[wfurt]

I'll take a look @jamespettigrew Last time I tried was poky-morty. It seems like curl it self got updated.

[wfurt]

BTW can you send me your local.conf and your .net packaging?

[jamespettigrew]

In order to reduces variables, I've simplified the build to this script:

#!/bin/bash

if [ ! -d poky ]; then
git clone git://git.yoctoproject.org/poky.git -b rocko
fi

cd poky

if [ ! -d meta-openembedded ]; then
git clone https://github.com/openembedded/meta-openembedded -b rocko
fi

if [ ! -d meta-sunxi ]; then
git clone https://github.com/linux-sunxi/meta-sunxi -b master
fi

if [ ! -d meta-mender ]; then
git clone git://github.com/mendersoftware/meta-mender -b rocko
fi

source ./oe-init-build-env

bitbake-layers add-layer ../meta-openembedded/meta-oe
bitbake-layers add-layer ../meta-openembedded/meta-python
bitbake-layers add-layer ../meta-openembedded/meta-networking
bitbake-layers add-layer ../meta-sunxi
bitbake-layers add-layer ../meta-mender/meta-mender-core
bitbake-layers add-layer ../meta-dotnettest

cat << EOF >> conf/local.conf

MACHINE = "nanopi-neo"
IMAGE_BOOT_FILES ?="boot.scr"
IMAGE_BOOTLOADER_FILE = "u-boot-sunxi-with-spl.bin" 
IMAGE_BOOTLOADER_BOOTSECTOR_OFFSET="16"
IMAGE_FSTYPES = "ext4"
DISTRO_FEATURES_append = " systemd"
VIRTUAL-RUNTIME_init_manager = "systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
VIRTUAL-RUNTIME_initscripts = ""

INHERIT += "mender-full buildhistory"
MENDER_ARTIFACT_NAME = "release-1"
BUILDHISTORY_COMMIT = "1"

PACKAGECONFIG_pn-curl_append = "zlib ipv6 ssl"
IMAGE_INSTALL_append = " kernel-devicetree dotnetcore" 

EOF

bitbake core-image-minimal

The 'meta-dotnettest' layer is just some unrelated u-boot and kernel patches along with the dotnetcore recipe which is as follows:

DESCRIPTION = ".NET Core 2.0.6"
HOMEPAGE = "http://dot.net/"
LICENSE = "MIT"
LIC_FILES_CHKSUM="file://LICENSE.txt;md5=42b611e7375c06a28601953626ab16cb"
SECTION = "devel"
PV = "2.0.6"
PR = "r0"

RPROVIDES_${PN} = "dotnetcore"
DEPENDS = "zlib curl"
RDEPENDS_${PN} = "libunwind icu curl libcurl openssl libgssapi-krb5 util-linux-libuuid lttng-ust"

FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI = "file://dotnet-runtime-2.0.6-linux-arm.tar.gz"
S = "${WORKDIR}/dotnet-runtime-2.0.6-linux-arm"

# Silence some QA warnings
INSANE_SKIP_${PN} += "staticdev already-stripped ldflags"

do_compile() {
}

do_install() {
    install -d ${D}/opt/dotnet
    cp -R --no-dereference --preserve=mode,links -v ${WORKDIR}/dotnet-runtime-2.0.6-linux-arm/* ${D}/opt/dotnet/
}

FILES_${PN} = "/opt/dotnet"
FILES_${PN}-dbg += "/opt/dotnet/.debug"

PACKAGES = "${PN}-dbg ${PN}"

[jamespettigrew]

I suspected the 'CURL_OPENSSL_3' part of the error message may have been the culprit. With this in mind, I ran:

objdump -T libcurl.so.4

...
00021aa8 g    DF .text  00000044  Base        curl_mfprintf

Noticing all the references to 'Base', I then made a curl bbappend to set the '--enable-versioned-symbols' flag and after recompiling objdump was returning 'CURL_OPENSSL_4' in the symbols instead of 'Base'. Finally, I patched libcurl.vers.in in the curl source to change the version flavour and after recompiling the symbol versions read 'CURL_OPENSSL_3' and now the Yocto build completes successfully. I haven't yet tested it to be sure it won't blow up at runtime.

The question is why are the prebuilt binaries expecting 'CURL_OPENSSL_3'? Curl itself doesn't appear to have used this version for around a decade, and the corefx source code has comments referencing relatively recent versions of curl so I don't think there's a genuine incompatibility here. Is this a corefx build issue or is something just lost on me? Perhaps there's a Yocto setting that will avoid this problem altogether?

[wfurt]

I'm glad you got it working @jamespettigrew. In general I think we build on oldest supported distro and that may suck in some dependencies - possibly unintentionally. Did you check https://github.com/dotnet/source-build ??? There is effort to allow building from sources and that should avoid troubles like this. As minimum you may be able to rebuild native dependencies for corefx. But I understand if you want to use whole consistent package.

[Petermarcu]

@wfurt is there anything more to track here or can the issue be closed?

[wfurt]

probably not. I was planing to update Yocto notes, but that can be done independently.

[jran-dasa]

@jamespettigrew it doesn't look like the Yocto notes ever got updated.. Could you possibly provide the bbappend that solved this issue? I cannot get it to work.

[karelz]

Note that 2.0 is not supported anymore. 2.1 has built-in new networking stack that does not use libcurl at all.

[jran-dasa]

@karelz Thank you for the information. Then it is even more strange that Yocto is throwing errors. I'm using a freshly downloaded 2.1.5

[karelz]

I think it is still build-time dependency as we have opt-out from the new networking stack to use libcurl.

[wfurt]

not necessarily. libcul is still build and runtime dependency. Even if 2.1 has default http handler implemented in c#, the old handler is still part of distribution.

However, to get it you need to target 2.1 in your project files. Simply using new runtime is not sufficient.

[jamespettigrew]

@jran-dasa I created the following patch for curl 7.54.1 and then modified the curl recipe so that the patch would be applied before curl gets compiled by Yocto.

From fd9e88d37ef3aa8c08c52a3bed8bd1ce9e0b6a43 Mon Sep 17 00:00:00 2001
From: James <james@innovum.com.au>
Date: Thu, 22 Mar 2018 14:20:29 +1100
Subject: [PATCH 1/1] Modify symbol versions

Should silence spurious libcurl dependency errors when packaging dotnetcore
---
 lib/libcurl.vers.in | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/libcurl.vers.in b/lib/libcurl.vers.in
index ae978a4..bce5633 100644
--- a/lib/libcurl.vers.in
+++ b/lib/libcurl.vers.in
@@ -6,7 +6,7 @@ HIDDEN
     _save*;
 };

-CURL_@CURL_LT_SHLIB_VERSIONED_FLAVOUR@4
+CURL_@CURL_LT_SHLIB_VERSIONED_FLAVOUR@3
 {
   global: curl_*;
   local: *;
-- 
2.7.4

If this doesn't make any sense or you need more help let me know!

[jran-dasa]

@jamespettigrew Thank you very much. I finally got it working.

The patch did not do it all on its own, but when I also added EXTRA_OECONF += "--enable-versioned-symbols" into Curl bbappend, I could finally build the rootfs without errors.

[jamespettigrew]

Forgot about that part, sorry!

Glad you got it working!