N.b. you might consider building a Docker image containing all the right components for this. Even just updating the apt database and installing components from cmake 96 times (# of jobs in the Linux job matrix) takes considerable time.
Sound sensible, I imagine that the test matrix spends far more time on apt that actually building/testing the headers themselves.
In https://github.com/KhronosGroup/OpenCL-Headers/issues/142 @oddhack says
Sound sensible, I imagine that the test matrix spends far more time on apt that actually building/testing the headers themselves.