New LLVM/clang versions soon?

[lgritz]

If I understand https://hub.docker.com/r/aswf/ci-osl/tags?name=2019 correctly, there are a number of osl-2019 containers with differing clang versions -- 7, 8, 9, 10?

But osl-2020 only has clang7, I think. https://hub.docker.com/r/aswf/ci-osl/tags?name=2020

OSL is thinking of bumping our minimum LLVM requirement from 7 to 9. Looks like we can bump our VFX Platform 2019 tests to llvm9, but we wouldn't have a usable LLVM for 2020. Can we get a new osl-2020 with a newer version? It doesn't need to be a whole range of them, just llvm10 will do (that was the main llvm release in 2020).

Also, we don't have any containers with LLVM 12 (and now that it's released, 13). Maybe we could get those (at least 13) in a variety of 2022 container, since those will be the dominant versions used next year?

For next time you're touching these containers anyway. There is no rush that would necessitate building a bunch of new containers just for this right now.

[aloysbaillet]

I started adding llvm 12 and 13, and as mentioned found that both require python 3 to be present at build time, worked around that locally but then found that llvm-13 fails to build with gcc-9 complaining about a destructor that is constexpr. It seems that llvm-13 does not support gcc-9, and there won't be a way to have it working while we stick to devtoolset-9! See https://www.mail-archive.com/llvm-bugs@lists.llvm.org/msg53136.html . Any thoughts @lgritz ?

[lgritz]

This code that doesn't build with gcc9 seems to be in libc++, which is part of the llvm suite, but is not the core llvm library itself. Is it possible that we can disable just the building of libc++ (which I think none of our projects depend on), and build the rest of llvm/clang?

You could build gcc11 (I need to test against that, too, because that's what people use when they are not beholden to VFX Platform), then use gcc11 to build llvm 13.

Alternately, it seems you can build llvm 13 with clang 12. :-) I'm not sure if you consider that simply another convoluted rathole we want to avoid going down.

I do wish that VFX Platform didn't keep us perpetually multiple years behind on gcc versions.

[lgritz]

Just in terms of my own priorities... Being able to test against llvm 13 is less important to me than being able to drop support for llvm 7 & 8.

So "llvm 9 on 2020" + "llvm 12 on 2022" would be enough keep me quiet for a while. The need to have 13 available is not urgent and can wait if it looks like a burden.

[aloysbaillet]

Sounds good, thanks! I'll try to skip libc++ and get 12 out soon.

[aloysbaillet]

Looks like the issue was closed automatically, but I've just released aswftesting images with clang 12 and 13 today: aswftesting/ci-osl:2022-clang12 and aswftesting/ci-osl:2022-clang13 are now ready to try. ​

The cool thing is that with these new images you can now install any clang version anywhere using conan. I need to update the docs, but here's a quick primer if you want to give it a go:

Let's say you want to use clang-13 in a VFX2021 image (that combination doesn't exist...) you could use aswftesting/ci-osl:2021-clang11 and create the following file in any folder: conanfile.txt:

[imports]
., * -> .
[requires]
clang/13.0.0@aswftesting/ci_common2

Then run in that same folder: conan install . which will download clang 13 and install it in the folder where you have your conanfile.txt

[lgritz]

Sorry, being dense here as a conan noob.

Does that build clang 13, or does that install a pre-built version?

[aloysbaillet]

To clarify: the conan install . will download a prebuild version of clang if it is found for the current "settings" (which it should be if you run from a aswfdocker image!)

[aloysbaillet]

Should come directly from https://linuxfoundation.jfrog.io/ui/repos/tree/General/aswf-conan%2Faswf%2Fclang%2F13.0.0%2Fci_common2

[aloysbaillet]

Would you be OK with using the aswftesting images for a little bit and let me know if you find any regression with them, if you don't find any I'll update the aswf images this weekend. With a good changelog and simple conan docs...

[lgritz]

Yeah, I'll give it a try.

[lgritz]

Does this ring a bell?

clang/9.0.1@aswftesting/ci_common2: Not found in local cache, looking in remotes...
clang/9.0.1@aswftesting/ci_common2: Trying with 'aswf'...
clang/9.0.1@aswftesting/ci_common2: Trying with 'conan-central'...
clang/9.0.1@aswftesting/ci_common2: Trying with 'aswftesting'...
ERROR: Unable to find 'clang/9.0.1@aswftesting/ci_common2' in remotes

How do I know which versions of clang I can ask for?

[aloysbaillet]

clang-9 is only in ci_common1 sorry ! clang/9.0.1@aswftesting/ci_common1 should work

[aloysbaillet]

the "ci_common" label is the low-level part of the VFX platform that doesn't change every year, mostly the devtoolset version on linux!

[aloysbaillet]

The https://linuxfoundation.jfrog.io/ui/repos/tree/General/aswf-conan%2Faswf%2Fclang url should point to the clang packages but it seems there's a redirection to a search page, you can search clang on the website and find all possible versions there.

[lgritz]

I see that I can search for clang, it shows me a whole lot of versions, but I'm not sure how I can detect common1 vs common2, etc.

But in any case, testing clang/9.0.1@aswftesting/ci_common1 now, will report back soon.

[aloysbaillet]

A bit disapointing that I can't send you a direct link, but you should be able to navigate to this in the Artifacts section:

[aloysbaillet]

You can also add this section to the conanfile.txt:

[generators]
virtualenv

which will generate a activate.sh file to set up the PATH and LD_LIBRARY_PATH for you

[aloysbaillet]

Actually there's an easier way that doesn't require a conanfile.txt:

conan install clang/9.0.1@aswftesting/ci_common1 -g deploy -g virtualenv

[aloysbaillet]

FYI that single command line approach works well if you only have one package to install in one folder, but it doesn't like multiple packages unfortunately.

[lgritz]

I was just coming back here to report that the conanfile technique worked, and I was able to use llvm 9.0.1 on the 2020 image (yay! that clears the way for me to phase out llvm 7 & 8). I'll try your new technique as well, that's even simpler.

[aloysbaillet]

Great to hear! I'll do a quick pass at updating some 2022 versions and release the aswf images shortly then!

[lgritz]

Is the only difference between aswftesting and aswf docker containers just that aswftesting has new features a few days or weeks earlier than they are pushed to aswf, but then at all other times they are identical?

Should I just use aswftesting all the time for my projects so that I can identify any issues immediately, and then I don't have to do anything special or go to any extra effort to test the aswftesting when you make changes?

[aloysbaillet]

I don't currently guarantee that the aswftesting images are all released identically to aswf at this stage, but as I need to do a big batch of releases for the conan work I could start guaranteeing this... Also aswftesting could break easily next time I try something else :) If you're OK with staying on the bleeding edge then that could be a nice way to introduce new features and get more testing than I usually would be able to do by myself!

[lgritz]

Yeah, so either I stay on aswf but have to take the time to do some custom test runs with aswftesting when you have new things to roll out, or I stay on aswftesting and don't have to do anything special to let you know about problems (but take the chance that occasionally you may break something). I think it's a fair trade. I can always "downgrade" to aswf if something is broken in aswftesting, or just tolerate a few days of broken tests. My impression is that things rarely break badly in aswftesting, so I think it'll be ok.

[aloysbaillet]

Seems fair indeed :) I'll make sure that I always update aswftesting first, as I have skipped it occasionally in the past for trivial updates!