ion authentication fails for packaged build

[lilleyse]

ion authentication is failing for packaged builds on Linux after https://github.com/CesiumGS/cesium-omniverse/pull/634 was merged.

Notes so far:

• Only affects packaged builds i.e. those built from the AlmaLinux8 docker container
• Doesn't affect local builds
• Updating curl, OpenSSL, etc to the newest versions doesn't fix it
• A workaround would be to ship our own cert again

Testing steps:

• Switch to update-libraries which has some Docker file fixes
• Run scripts/build_package_almalinux8.sh. This produces a zip folder in the build folder.
• Extract the zip file and copy the extensions to /home/slilley/Documents/Kit/shared/exts/
• Run USD Composer and sign in to Cesium ion

[lilleyse]

It turns out that the OpenSSL search dir is hardcoded into the binary. In CentOS/RHEL/Alma that path is /etc/pki/tls whereas on Ubuntu it's /etc/ssl/certs/. So an Alma build of OpenSSL won't be able to locate certs in Ubuntu.

There are a few options

• Override OPENSSLDIR when compiling OpenSSL (but now that breaks Omniverse users on CentOS/RHEL)
• Find some portable way to locate OS certificate dir at compile time or runtime
• Ship our own cert again
• Don't statically link OpenSSL. Use the system's version.

@weegeekps any thoughts? You've dealt with these sorts of issues before...

[weegeekps]

You've dealt with these sorts of issues before...

Yup, and these are the exact issues as to why the Curl devs recommend you just bundle your own certs, including providing an mirror for the Mozilla ca-cert bundle.

The ultimate problem here is that each OS and distribution uses their own paths, and most also occasionally change the paths between major versions and even OpenSSL versions. Debian/Ubuntu is particularly bad about this. You can try to do some heuristics to check the various known paths for the major OS's and their variants, but ultimately you will inevitably run into problems.

I can't find exactly where it's kept in Omniverse but I believe Nvidia also ships the Mozilla ca-cert bundles as well.

That said, I think it's worth me going over a few of the other options that exist. This is all covered in some detail in some of the Curl docs, but I'll go through the shortcomings I've run into over time. I've kept the same ordering as the docs.

1. Not verifying the peer is sadly what a lot of application devs do. This is not a good option, and we should not do it.
2. Including a CA cert bundle and then providing it using curl_easy_setopt(curl, CURLOPT_CAINFO, cacert); is the preferred way when you ask Curl devs in discussions and GitHub issues.
3. You can also, in theory, use the default CA certificate store, but where libcurl looks is set statically at compile time. There is some autodetection that works fairly well on with the Schannel backend that is unique to Windows, but on Mac and Linux it generally does not consistently work. There is no standard place to put your CA certificate store, and so every distribution has it in a different spot. So you have to compile unique binaries for every system... or you just include your own CA cert, or heuristics to handle the most common distros. I believe the latter is what Zoom does in Linux; it works well in Red Hat, Ubuntu/Debian, and some Arch-flavored distros, but outside of that Zoom just doesn't function because there it can't get validate the TLS connections.
4. The Curl command line tool does allow you to provide an environment variable for the location of the CURL_CA_BUNDLE. I've never seen it copied, and it's really meant for when you have a Man-in-the-Middle monitoring solution in a data center with a private CA cert bundle. The Curl command line tool itself is compiled specifically for each distribution so the statically set default path is good enough for it.

My recommendation is what it always has been: we should bundle the Mozilla CA cert with the Cesium for Omniverse code. I think there's always an aspiration to get away from that extra bit of maintenance but I've never seen an approach to do so that works completely in the Linux ecosystem.

[lilleyse]

@weegeekps that sounds good - we still have the code for using bundled certs, we'll just need to revert some of the changes in https://github.com/CesiumGS/cesium-omniverse/pull/634 so that we download certs as part of the build system again.

Would you recommend this for Windows too? My only concern is that some users were seeing self-signed cert issues on Windows prior to https://github.com/CesiumGS/cesium-omniverse/pull/634, like below:

[weegeekps]

@lilleyse If we do this for one OS, we should probably do it for all of them. Its easier and it will guarantee that we'll be able to handle problems for the vast majority of our users. There is one edge case where someone is on a tightly locked down network and standard CA certificates won't work for them. If we want to support that case we probably should allow them to specify and additional CA certificate bundle path via an environment variable, but that's an increasingly rare use case because most software handles it so poorly.

[lilleyse]

Maybe there's a way to configure curl/OpenSSL to check the search directory first, and then fall back to the bundled cert?

[weegeekps]

I've attempted that before, but the issue is that waiting for OpenSSL to timeout with the failure is slow. It's not something you can do frequently, and ultimately if you go that route you need to count the number of failures and set a flag that says "we need to use the bundled cert." If failures exceed 2 or 3 or whatever you want, then you need to properly fail to the user. Since you're already bundling a cert it's just easier to always use that.

Also, the default store for OpenSSL frequently still fails to authorize for that edge case I mentioned. That's why it's becoming an increasingly uncommon solution. I think we should ignore that edge case until we actually see it in the wild.

[lilleyse]

Alright @timoore, let's just go with the @weegeekps's suggestion to unblock the release.

[timoore]

Addressed by #697