Releasing compiled binaries for vulkan kompute

[unexploredtest]

Right now, in order to install vulkan kompute, one has to compile the whole thing from source, however packaging binaries for it would be helpful. Won't Vulkan Kompute release packaged binaries? I can volunteer for .deb and .rpm packages and an AUR PKGBUILD.

[axsaucedo]

@aliPMPAINT great idea, I think this would be very useful for sure, we had a previous contributor that created an AUR package from the repo - the discussion was at https://github.com/EthicalML/vulkan-kompute/issues/81 - it seems like the documentation and maintenance didn't really get carried forward so it would be good to do this properly and ensure that it's documented and maintained. If you are able to explore this, that could be a good addition for sure

[unexploredtest]

Will do

[unexploredtest]

So, I checked @Kezii's PKGBUILD and it's fine, as long as dependencies don't change and/or CMake options stay the same. What I want to add is, alongside with the git version(vulkan-kompute-git), a release version(vulkan-kompute) which uses the latest release instead of cloning from master, as it's more stable, and also a binary edition(vulkan-kompute-bin) so that there won't be any need to compile from source. Also, is SPDLOG needed for production?

[axsaucedo]

@aliPMPAINT good idea - I think that would make sense in regards to using the latest release. Also would make sense to have the binary edition, although that would introduce some complexities on the release process - this is also the reason why I don't currently build any wheels for the python releases (as it would require developing the release pipelines that create the binaries). None of the dependencies except Vulkan and FMT are required for production, it's possible to remove SPDLOG but it does add better debugging / diagnosing experience, although the gain is not too significant so I could look at removing it eventually. Is there a reason why you woudl like to remove it?

[unexploredtest]

@axsaucedo

it's possible to remove SPDLOG but it does add better debugging / diagnosing experience, although the gain is not too significant so I could look at removing it eventually. Is there a reason why you woudl like to remove it?

Well, because I want to make the packaged binaries as light as possible, and I don't think everyone would need SPDLOG(the part I'm not sure about), so I'm thinking that putting SPDLOG as optional would makes sense I think.

[unexploredtest]

Here is how .deb would look like(still not through): vulkan-kompute-0.6.0-amd64.deb.tar.gz

[axsaucedo]

Ok thank you for the heads up @aliPMPAINT - I think this sounds reasonable, we should go ahead with this. I guess my other question is also how we can make sure these can be managed and maintained on each release? Can it be automatically updated based on the repo releases or does a new package need to be submitted on every reelase?

[unexploredtest]

Yes, there are ways to automate package releases for each release (AFAIK) I'll do more research on this regard.

[unexploredtest]

@axsaucedo The building process can be automated via these github actions tools: https://github.com/marketplace/actions/build-linux-packages or https://github.com/bpicode/github-action-fpm for .deb and .rpm https://github.com/joerick/cibuildwheel for python wheels I think these can remove the complexities that gets introduced on the release process.

[unexploredtest]

@axsaucedo Here I made a sample workflow and a demonstration of how it will work out: https://github.com/aliPMPAINT/vulkan-kompute/releases/tag/v0.2.9 https://github.com/aliPMPAINT/vulkan-kompute/blob/master/.github/workflows/cpp_linux_x86_64.yml By this way, if we create the said workflows, it'll be possible to release packages/wheels for various OS/distors with different architectures automatically by just making a new release.

[axsaucedo]

@aliPMPAINT this is quite cool, it seems the github action would be triggered on release as well. I think given that it's also saved in the github release entry this makes it easier to maintain as well. I do notice that there is quite a lot of use of third party actions - the bpicode/github-action-fpm seems to be a very simple dockerfile so it may be easier to write down the logic as an explicit step instead of using an external action. Overall this looks quite good tho, I would be interested to add this, if you can add a PR we can try it out

[unexploredtest]

the bpicode/github-action-fpm seems to be a very simple dockerfile so it may be easier to write down the logic as an explicit step instead of using an external action

Noted

Overall this looks quite good tho, I would be interested to add this, if you can add a PR we can try it out

Yeah sure! I'll keep you updated here

[unexploredtest]

@axsaucedo Sorry for the tardiness, I had huge issues with glslang, and it still remains(will explain). Anyways, with workarounds, here is the current state: https://github.com/aliPMPAINT/vulkan-kompute/releases/tag/0.0.5 The workflows: https://github.com/aliPMPAINT/vulkan-kompute/tree/master/.github/workflows

[unexploredtest]

Older versions of glslang are not compatible, as one of the TBuiltInResource struct variables was removed. This creates problem with Debian and Debian-based distros, as they have out-dated packages. Not only that, older debian(older than buster) don't have glslang at all, and there is no PPA for it afaik. Ubuntu, as it's debian based, doesn't have glslang in version older than 20.04 and with 20.04, glslang is out-dated and not compatible. With RPM-based disros, just Fedora has glslang. What we can do is to provide instructions on how to install the latest version of glslang for each distro.

[axsaucedo]

This is interesting, thank you for the update @aliPMPAINT - in regards to the points, I recently added in 0.7.0 the ability to build all dependencies as static_libraries by default, which means that we should be able to set up as submodule build for all configurations. Would this address all of the challenges outlined above?

When exploring integration and maintenance of package managers we'd ideally depend as little as possible on external dependencies, as otherwise we're at the mercy of all those packages being available - that or we have to maintain them ourselves which wouldn't really be something we can take on.

[axsaucedo]

Actually, the above fix would still not resolve the issues as it would still be necessary to ensure all the right headers are available in order for a dependency to be able to import and use the Kompute library, otherwise it will indeed complain about the headers not being present. Unfortunately it does seem like there's no simple solution for this...

An alternative is to adopt the headers as part of the install and install them as kompute/glslang/..., kompute/fmt/..., kompute/spdlog/..., but there's also disadvantages on this, mainly that there will be developers that will want to use their glslang headers / library instead - but it's not uncommon either, spdlog for example has their own in-repo version of fmt, so it could be something to potentially consider, although it certainly would add further complexity.

Another option would be to use the Google shaderc / glslangc dependency instead of glslang, which has a slower build time, but may be better supported across distros, perhaps would be worth having a look at what versions are available in the respective package manager - moving from glslang to shaderc wouldn't be too hard from the code perspective.

[unexploredtest]

When exploring integration and maintenance of package managers we'd ideally depend as little as possible on external dependencies, as otherwise we're at the mercy of all those packages being available - that or we have to maintain them ourselves which wouldn't really be something we can take on.

Yes I agree, but if we want to include them we have to be careful not to make a conflict with packages, as if someone has already installed glslang, spdlog or another external package, installing the package shouldn't overwrite the existing packages, otherwise it'll yield error.

An alternative is to adopt the headers as part of the install and install them as kompute/glslang/..., kompute/fmt/..., kompute/spdlog/..., but there's also disadvantages on this, mainly that there will be developers that will want to use their glslang headers / library instead - but it's not uncommon either, spdlog for example has their own in-repo version of fmt, so it could be something to potentially consider, although it certainly would add further complexity.

This sounds good.

Another option would be to use the Google shaderc / glslangc dependency instead of glslang, which has a slower build time, but may be better supported across distros, perhaps would be worth having a look at what versions are available in the respective package manager - moving from glslang to shaderc wouldn't be too hard from the code perspective.

I just checked and the problem is even worse with shaderc, it does not even exist in debian-based distros at all(though it does for Arch linux, Fedora and some other distros).

Another possible solution is, as our problem is just with glslang, we can package the binaries with glslang and declare it as a conflict with whatever glslang package that exists.

[StefanFabian]

I'm also interested in debian packages for Ubuntu 20.04+. Are there any updates on this? Regarding the issues mentioned here. Is there a PPA for glslang + shaderc with compatible versions? If so, the debian package could just specify the minimum or exact required version. If it's backwards compatible, packaging glslang sounds like a good option and wouldn't even need conflict, it could be set as replaces.

[sudokit]

Will windows precompiled binaries also be a possibility?