Build fails on Linux : use of undeclared type 'BIGNUM'

[glaurent]

Building on Linux fails with several of the following

/srv/datadisk01/www/VideoSaverServer/.build/checkouts/APNSwift/Sources/APNSwift/APNSwiftSigner.swift:56:31: error: use of undeclared type 'BIGNUM'
        var r : UnsafePointer<BIGNUM>? = nil

Swift version 5.1-dev Ubuntu 18

[weissi]

Hmm, I can't repro on 18.04 with 5.0.2:

git clone https://github.com/kylebrowning/APNSwift
cd APNSwift
docker run -it --rm -v "$PWD:$PWD" -w "$PWD" swift:5.0.2-bionic bash -c 'apt-get -y update && apt-get -y install openssl libssl-dev && swift test'

works.

@glaurent can you send the output of

cat /etc/lsb-release; dpkg -l libssl-dev

?

[weissi]

@glaurent also can you send your Package.resolved file? At least the following things are interesting: swift-nio-ssl version / APNSwift version or commit

[glaurent]

Here goes :

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                  Version                 Architecture            Description
+++-=====================================-=======================-=======================-================================================================================
ii  libssl-dev:amd64                      1.1.1-1ubuntu2.1~18.04. amd64                   Secure Sockets Layer toolkit - development files

From Package.resolved :

{
        "package": "swift-nio-ssl",
        "repositoryURL": "https://github.com/apple/swift-nio-ssl.git",
        "state": {
          "branch": null,
          "revision": "f5dd7a60ff56f501ff7bf9be753e4b1875bfaf20",
          "version": "2.4.0"
        }
      }

{
        "package": "apnswift",
        "repositoryURL": "https://github.com/kylebrowning/APNSwift",
        "state": {
          "branch": null,
          "revision": "94acc7be09d5e195feb3230cc4bea57cd8ae255f",
          "version": "1.1.1"
        }
      }

[weissi]

Thanks @glaurent , I repro. The following command (when invoked from an APNSwift checkout) reproduces:

docker run -it --rm -v "$PWD:$PWD" -w "$PWD" ubuntu:18.04 bash -c 'apt-get update && apt-get install -y git curl libxml2-dev build-essential libssl-dev && curl https://swift.org/builds/swift-5.1-branch/ubuntu1804/swift-5.1-DEVELOPMENT-SNAPSHOT-2019-08-26-a/swift-5.1-DEVELOPMENT-SNAPSHOT-2019-08-26-a-ubuntu18.04.tar.gz | tar -C / -xz && export PATH="$PATH:/swift-5.1-DEVELOPMENT-SNAPSHOT-2019-08-26-a-ubuntu18.04/usr/bin" && swift build'

I can see the following errors:

Fetching https://github.com/apple/swift-nio.git
Fetching https://github.com/apple/swift-nio-ssl.git
Fetching https://github.com/apple/swift-nio-http2.git
Completed resolution in 27.43s
Cloning https://github.com/apple/swift-nio-ssl.git
Resolving https://github.com/apple/swift-nio-ssl.git at 2.4.0
Cloning https://github.com/apple/swift-nio.git
Resolving https://github.com/apple/swift-nio.git at 2.7.1
Cloning https://github.com/apple/swift-nio-http2.git
Resolving https://github.com/apple/swift-nio-http2.git at 1.5.0
/Users/johannes/extsrc/APNSwift/Sources/APNSwift/APNSwiftSigner.swift:56:31: error: use of undeclared type 'BIGNUM'
        var r : UnsafePointer<BIGNUM>? = nil
                              ^~~~~~
/Users/johannes/extsrc/APNSwift/Sources/APNSwift/APNSwiftSigner.swift:57:31: error: use of undeclared type 'BIGNUM'
        var s : UnsafePointer<BIGNUM>? = nil
                              ^~~~~~
/Users/johannes/extsrc/APNSwift/Sources/APNSwift/APNSwiftSigner.swift:56:31: error: use of undeclared type 'BIGNUM'
        var r : UnsafePointer<BIGNUM>? = nil
                              ^~~~~~
/Users/johannes/extsrc/APNSwift/Sources/APNSwift/APNSwiftSigner.swift:57:31: error: use of undeclared type 'BIGNUM'
        var s : UnsafePointer<BIGNUM>? = nil
                              ^~~~~~
/Users/johannes/extsrc/APNSwift/Sources/APNSwift/APNSwiftSigner.swift:56:31: error: use of undeclared type 'BIGNUM'
        var r : UnsafePointer<BIGNUM>? = nil
                              ^~~~~~
/Users/johannes/extsrc/APNSwift/Sources/APNSwift/APNSwiftSigner.swift:57:31: error: use of undeclared type 'BIGNUM'
        var s : UnsafePointer<BIGNUM>? = nil
                              ^~~~~~

[weissi]

Interestingly, this one does now repro (which uses 5.0.2):

docker run -it --rm -v "$PWD:$PWD" -w "$PWD" ubuntu:18.04 bash -c 'apt-get update && apt-get install -y git curl libxml2-dev build-essential libssl-dev && curl https://swift.org/builds/swift-5.0.2-release/ubuntu1804/swift-5.0.2-RELEASE/swift-5.0.2-RELEASE-ubuntu18.04.tar.gz | tar -C / -xz && export PATH="$PATH:/swift-5.0.2-RELEASE-ubuntu18.04/usr/bin" && swift build'

[weissi]

ok, I filed https://bugs.swift.org/browse/SR-11382 for the change in behaviour. But honestly, currently I don't understand why it works on Swift 5.0.2? BIGNUM seems to be a forward-declared struct bignum_st and pointers to forward-declared structs are usually imported as OpaquePointer...

@kylebrowning was there a reason using UnsafePointer<BIGNUM> in Swift and not OpaquePointer using the 'pointer tricks'?

[kylebrowning]

@weissi It was suggested from @Lukasa https://github.com/kylebrowning/APNSwift/pull/38#discussion_r312460093

[kylebrowning]

Looks like it was an oversight and not ready to actually merge.

https://github.com/kylebrowning/APNSwift/pull/38/files#r312515462

[weissi]

@kylebrowning ha, so yes what @Lukasa says is exactly my sentiment: No idea why this works in 5.0.2, as @lukasa said elsewhere, we should switch everything over to OpaquePointer this was the right solution always and seems to be the only solution in 5.1.

[kylebrowning]

Yes I can take care of this, this weekend.

sorry just started a new gig so I've been swamped.

[weissi]

Thanks so much @kylebrowning!

[weissi]

any updates here @kylebrowning ? The way it’s used right now shouldn’t work on 5.0 but does work for some reason... it basically seems to see BoringSSL’s definition by accident (which it shouldn’t). That changed in 5.1. So we should really fix this before 5.1 is released.

Please reach out if you need help here. But it’s mostly using OpaquePointer instead of BIGNUM because that’s just forward declared. This is pretty similar to the supporting OpenSSL 1.1 work you did before

[kylebrowning]

Sorry @weissi. New job has me working hard! Good thing though.

I’ll make sure this happens before 5.1 which I presume is this month.

[weissi]

Don’t apologise, we all understand this isn’t your actual day job! Hope your new job is going well and is fun!

[kylebrowning]

hrmmm. @weissi Running

docker run -it --rm -v "$PWD:$PWD" -w "$PWD" ubuntu:18.04 bash -c 'apt-get update && apt-get install -y git curl libxml2-dev build-essential libssl-dev && curl https://swift.org/builds/swift-5.1-branch/ubuntu1804/swift-5.1-DEVELOPMENT-SNAPSHOT-2019-08-26-a/swift-5.1-DEVELOPMENT-SNAPSHOT-2019-08-26-a-ubuntu18.04.tar.gz | tar -C / -xz && export PATH="$PATH:/swift-5.1-DEVELOPMENT-SNAPSHOT-2019-08-26-a-ubuntu18.04/usr/bin" && swift build'

Produces a different error for me, and its in swift-nio-ssl

<unknown>:0: error: reference to 'SHA256_CTX' is ambiguous
/Users/kylebrowning/Projects/swift/swift-nio-apns/.build/checkouts/swift-nio-ssl/Sources/CNIOBoringSSL/include/CNIOBoringSSL/base.h:411:32: note: candidate found by name lookup is 'SHA256_CTX'
typedef struct sha256_state_st SHA256_CTX;
                               ^
//usr/include/openssl/sha.h:56:3: note: candidate found by name lookup is 'SHA256_CTX'
} SHA256_CTX;
  ^
Failed to reconstruct type for $sSo10SHA256_CTXaD
Original type:
(type_alias_type decl=CAPNSOpenSSL.(file).SHA256_CTX)
Stack dump:

Full output here

[kylebrowning]

Alternatively,

docker run -it --rm -v "$PWD:$PWD" -w "$PWD" ubuntu:18.04 bash -c 'apt-get update && apt-get install -y git curl libxml2-dev build-essential libssl-dev && curl https://swift.org/builds/swift-5.0.2-release/ubuntu1804/swift-5.0.2-RELEASE/swift-5.0.2-RELEASE-ubuntu18.04.tar.gz | tar -C / -xz && export PATH="$PATH:/swift-5.0.2-RELEASE-ubuntu18.04/usr/bin" && swift build'

succeeds, even though we think it shouldn't

[weissi]

@kylebrowning did you try swift package update to be sure to get the latest version of swift-nio-ssl? The Swift 5.0 version will succeed despite the fact it shouldn't because Swift 5.0 doesn't have @implementationOnly which swift-nio-ssl now uses (for Swift 5.1 only) to fully hide the C stuff.

[kylebrowning]

My local checkout is up to date yet, but just gonna add it to the command before swift build and try again

Running now.

[weissi]

@kylebrowning but the dependency versions might be older because of Package.resolved, right? You'd need swift-nio-ssl >= 2.4.0

[weissi]

Oh, and btw, if you remove the --rm from the docker run, then the container won't be deleted and you can re-use that container after that command ran and don't need to wait all the time to download & install Swift again. You can just do docker ps and then docker exec -it CONTAINER_ID /bin/bash to reconnect to an existing container.

[kylebrowning]

❯ swift package update
Updating https://github.com/apple/swift-nio-ssl.git
Updating https://github.com/apple/swift-nio.git
Updating https://github.com/apple/swift-nio-http2.git
Completed resolution in 9.61s
Resolving https://github.com/apple/swift-nio-ssl.git at 2.4.0
Resolving https://github.com/apple/swift-nio.git at 2.8.0
Resolving https://github.com/apple/swift-nio-http2.git at 1.6.1

[kylebrowning]

@glaurent I have a fix coming.

[glaurent]

@kylebrowning thank you very much for your work :)

[kylebrowning]

@glaurent Give #44 a try. Im going to be doing some more testing, but its up for you now.

[glaurent]

@kylebrowning thanks a lot, I'll check this week-end

[glaurent]

@kylebrowning it builds without errors, thanks again 👍