search

build2-packaging / SimpleBLE

build2 packages for SimpleBLE
MIT License
0 stars 0 forks source link

Bluetooth stack dependencies on CI #1

Closed Rookfighter closed 1 year ago

Rookfighter commented 1 year ago

Hi @boris-kolpackov,

This library depends on os-specific bluetooth stacks. How could we handle these dependencies for bdep ci?

Are you willing to install additional dependencies? If not can I then still publish this on cppget.org?

Edit: Here's a bdep cibuild: https://ci.cppget.org/@ee4fc56a-161b-47e1-ba5d-d7583377e7a1

boris-kolpackov commented 1 year ago

I could install these but this approach probably won't scale long-term. But I think we may have a potential solution for this in the works for 0.16.0. It has two parts:

  1. We are working as we speak on system package manager integration which will allow us to query and optionally automatically install system packages. Debian (and alike) support is already in master while Fedora (and alike) is in the works. With this I can for example do the following on my (Debian) machine:

    bpkg build --sys-install --sys-no-stub  sys:libbluetooth

    And it will install libbluetooth3 and libbluetooth-dev for me automatically.

  2. Another new feature that is already in 0.16.0 is support for custom CI configurations that can be specified in package manifest. Normally this will be used to test different package config.* combinations but can also be used to satisfy some dependencies as system packages.

Putting these two pieces together, it feels like we should be able to use this custom package configuration support to automatically install any missing system dependencies. At least for Debian/Fedora for starters (we have no immediate plans to look into Windows, Mac OS, or FreeBSD package managers but may do so in the future). There are still some bits missing (like a VM that would allow system package manager interactions) before we can try this on stage but hopefully soon.

So I would suggest that for now you go ahead and publish these packages as is but also leave this bug open so we can try/fix this for 0.16.0, once the above functionality is ready.

@Klaim FYI

Klaim commented 1 year ago

👍🏽 (I was aware of this work but not how to use it exactly)

(we have no immediate plans to look into Windows, Mac OS, or FreeBSD package managers but may do so in the future)

I dont know if it would be possible, but if there was ways to add/extend/configure such system package calling system to specify to build2 how to use whatever other system packager manager one have around, that would solve the issue for at least Windows and Mac (like for example support for conda/mamba/micromamba or brew or chocolatey, these are the main ones used on these platforms). I suspect maybe it's not possible if it's not just knowing which commands to run, but just stating in case it helps envision the future. In any way I note that it wouldn't come soon.

boris-kolpackov commented 1 year ago

I suspect maybe it's not possible if it's not just knowing which commands to run [...]

It is essentially which commands to run plus how to map things (package names, versions, etc) which can get full of heuristics. So I don't think just providing a bunch of command lines to query and install things is a realistic expectation. But it's also not terribly complicated and if someone wishes to contribute/maintain an implementation, we will definitely consider integrating it. The interface of the system package manager is documented here (as you can see it's more comments than code ;-)): system-package-manager.hxx. And the Debian implementation is here: system-package-manager-debian.hxx system-package-manager-debian.cxx. See also the *.test.cxx and *.test.testscript files for the way the whole thing is tested.

Klaim commented 1 year ago

Interesting, I'll note to make an attempt at the conda/mamba/micromamba one after 0.16.0 release (as my dayjob implies working on mamba/micromamba).

Rookfighter commented 1 year ago

@boris-kolpackov, I think I have reached a point where I am running out of ideas on how to install the missing dependencies on the windows runner on GitHub Actions. I have tried different solutions to install the missing dependencies, but none of them worked. I have successfully compiled the library on my Win10 and Win11 machines, where I only had to install the Desktop development with C++ and .NET desktop development workload package with default parameters. Ubuntu and MacOS also work on GitHub Actions. However, all builds on bdep ci fail (due to missing dependencies).

So how should I proceed? Can I simply publish the packages on cppget? And should I disable the builds for bdep ci? And since this package again compiles objective-c: Are there new objective-c features in build2 0.16.0 available which are worth to move this package to 0.16.0?

boris-kolpackov commented 1 year ago

I have successfully compiled the library on my Win10 and Win11 machines, where I only had to install the Desktop development with C++ and .NET desktop development workload package with default parameters.

Interesting, on CI we have Desktop development with C++ pretty much with default settings but not .NET. I wonder if this library needs .NET somehow.

So how should I proceed? Can I simply publish the packages on cppget? And should I disable the builds for bdep ci?

Yes, I think go ahead and publish it. I also wouldn't disable anything since we will hopefully address at least some of it once 0.16.0 is out.

And since this package again compiles objective-c: Are there new objective-c features in build2 0.16.0 available which are worth to move this package to 0.16.0?

Yes, there are some goodies: https://github.com/build2/build2/blob/master/NEWS#L3-L26

We use this in Qt builds: https://github.com/build2-packaging/Qt6/

Rookfighter commented 1 year ago

Interesting, on CI we have Desktop development with C++ pretty much with default settings but not .NET. I wonder if this library needs .NET somehow.

I am not too familiar with the Windows BLE API (after all I want to use a library which encapsulates these details), but as far as I understood it, the (here used) BLE API is part of UWP (see here). So I would have expected that I need to install the Universal Windows Platform development workload package. I had only installed Desktop development with C++ and.NET desktop development and gave the compilation a try and it just works. So I did not bother to dig any deeper. I tried to install all three workload packages on the GitHub runners using chocolatey, but it did not work out.

I also tried installing the Windows SDK as suggested by the simpleble docs (see here), but that also did not work out on GitHub actions. So I would consider the truly required dependency as unknown yet.

Yes, there are some goodies: https://github.com/build2/build2/blob/master/NEWS#L3-L26

Great! It makes the buildfiles so much cleaner and easier to read. Really appreciate this feature!

boris-kolpackov commented 1 year ago

Ok, we've now added the first configuration with enabled system package manager interaction (such machines are in the new sys class) and I gave this problem a try in the sys branch of this repository (it also contains a bunch of build fixes). Here is the result of bdep ci -d libsimpleble on this branch: https://ci.stage.build2.org/@42c9c6e7-8c45-4c82-8bd2-15f7308189de

You can also tweak/override all this from the bdep-ci command line (useful during development). For example:

bdep ci -d libsimpleble --builds sys --override 'default-build-config: sys:libbluetooth sys:libdbus-1'

For details, see:

https://build2.org/stage/bdep/doc/bdep-ci.xhtml https://build2.org/stage/bpkg/doc/build2-package-manager-manual.xhtml#manifest-package-build-config

When we add the similar Fedora configuration, I will try to update this branch to cover that as well.

boris-kolpackov commented 1 year ago

Ok, we now have a Fedora configuration with the system package manager enabled, and I've updated the sys branch to cover that: https://ci.stage.build2.org/@c28b936d-9767-425c-839b-964454edd383

Note that I had to fix build some more (see commits/comments for details).

Rookfighter commented 1 year ago

Hi @boris-kolpackov, thanks for all the work! Sounds great, I will have a look this weekend, didn't get the time during the week 👍

Rookfighter commented 1 year ago

@boris-kolpackov I had a look into your commits :+1: Do you mind if I merge this to master? Thanks again for the additions and fixes. I was expecting that the package still had its issues, as I was not able to run it on bdep ci before. One thing that I do not fully get: You are using a import statement for libdbus now, but there were no other additions in the manifest or anywhere else (maybe I just did not see it?). In the comment you write you go through pkg-config. How is the file pkg-config file picked up? Like, how is the mapping between the import statement in the buildfile and the actual pkg-config file established (search paths, file name etc.)? I was not able to find this in the build2 docs (but again maybe did not look thoroughly enough). Does this mechanism in general work for all system-installed libraries with valid pkg-config files? Finally, isn't this line for the dbus include path then obsolete? https://github.com/build2-packaging/SimpleBLE/commit/2373ff16e37604b0a0c8e65aa453f7639218abdd#diff-71ec215fdacb7d18874ead4caed7ec287d00232b5d59b390ed51ba8198ea7f07R36

boris-kolpackov commented 1 year ago

Do you mind if I merge this to master?

Sure, go ahead (generally, whatever I push to branches is good to use as you see fit).

One thing that I do not fully get: You are using a import statement for libdbus now, but there were no other additions in the manifest or anywhere else [...] How is the file pkg-config file picked up?

The pkg-config files are searched in the same directories as where the compiler would look for -lfoo libraries (i.e., we extract all those paths; for details see: https://github.com/build2/libpkg-config). And there is no requirement for a library with pkg-config file to be listed as a dependency in manifest.

Does this mechanism in general work for all system-installed libraries with valid pkg-config files?

Correct, you can import system libraries, it's just rarely needed.

Finally, isn't this line for the dbus include path then obsolete?

Yes, I missed that, it should no longer be needed. Looks like you were trying to hack around the same issue that the pkg-config file solves.

Rookfighter commented 1 year ago

Closed as resolved for Linux systems.