Add dev container with Celix dependencies

[rlenferink]

This allows building and developing of Celix by using this pre-built container

[codecov-commenter]

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (95bbcfc) 83.11% compared to head (3719128) 83.47%. Report is 9 commits behind head on master.

:exclamation: Current head 3719128 differs from pull request most recent head 21cf541. Consider uploading reports for the commit 21cf541 to get more accurate results

Files	Patch %	Lines
libs/dfi/src/dyn_avpr_type.c	50.00%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #656 +/- ## ========================================== + Coverage 83.11% 83.47% +0.35% ========================================== Files 254 254 Lines 32929 32979 +50 ========================================== + Hits 27369 27528 +159 + Misses 5560 5451 -109 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[rlenferink]

@pnoltes I tried the .gitpod.Dockerfile you put on your branch. That seems to start a container with the gitpod user, but I can't it to build Celix succesfully locally. The gitpod user has no access to write to the volume mounted files.

I also tried with my suggested Containerfile to execute as a non-root user within the container, but didn't succeed. This seems to be a known 'issue' with volume mounts. Normally for operational applications for data volume mounts there is an entrypoint which chowns the volume mount to the uid/gid within the container so that the user within the container can write to it.

When wanting to delete these files as a non-root user on the host you can do e.g. podman unshare rm -rf *. But for a development container that isn't something that is wanted IMO.

Were you able to build Celix using that gitpod container locally?

[pnoltes]

@pnoltes I tried the .gitpod.Dockerfile you put on your branch. That seems to start a container with the gitpod user, but I can't it to build Celix succesfully locally. The gitpod user has no access to write to the volume mounted files.

I also tried with my suggested Containerfile to execute as a non-root user within the container, but didn't succeed. This seems to be a known 'issue' with volume mounts. Normally for operational applications for data volume mounts there is an entrypoint which chowns the volume mount to the uid/gid within the container so that the user within the container can write to it.

When wanting to delete these files as a non-root user on the host you can do e.g. podman unshare rm -rf *. But for a development container that isn't something that is wanted IMO.

Were you able to build Celix using that gitpod container locally?

Yes, the main issue I had was the - at the time - I could not run (and thus also not debug) gtests from the IDE. (I think that should now be possible with C++ TestMate, but I am not a experienced vscode user.)

But do note that this was about 3 years ago, I am not sure how much gitpod has changed.

[rlenferink]

For me the build already fails with the gitpod user:

gitpod /home/rlenferink/workspace/asf/celix/celix-container/build (feature/add-dev-container) $ cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_TESTING=ON ..
CMake Error: Unable to (re)create the private pkgRedirects directory:
/home/rlenferink/workspace/asf/celix/celix-container/build/CMakeFiles/pkgRedirects
gitpod /home/rlenferink/workspace/asf/celix/celix-container/build (feature/add-dev-container) $ mkdir testDir
mkdir: cannot create directory ‘testDir’: Permission denied

I had the exact same issues when I tried a non-root user in the Containerfile of this pull request.

This seems intended behavior to lock down the users who are able to write to volumes.

but I am not a experienced vscode user

Neither am I. Every Apache committer is allowed a Jetbrains license for their OSS work, so that is what I am using and am experienced with.

My goal was to play around and see if I can extend the Containerfile for use with Clion and then as follow-up see how this can be integrated in e.g. GitHub codespaces.

[rlenferink]

I added the SSH package and configuration.

I tested this with CLion 2022.x and the latest CLion version (2023.2.x), and the IDE has an option to specify a remote development environment: https://www.jetbrains.com/help/clion/2023.2/remote-development-a.html

That means that instead of setting up a Remote Host and the IDE constantly copying files (even though the deployment mapping says otherwise), it downloads and starts an IDE backend within the container. The IDE on the host is connecting to that and there is no loss in speed between editing, building and testing.

Since this is up to the IDE, there are no Jetbrains specific items added, which means that the container can also be used standalone (volume mount the sources, open a shell and mkdir build && cmake .. && make -j && ctest). Next to that the possibility also still exists to use the Remote Host option (although that option being significantly slower).

Feedback / thoughts are welcome ;)

[pnoltes]

and the IDE has an option to specify a rem

Interesting, give me some time to try this out.

I am especially curious how much work it is to set this up and whether you can save your setup to quickly resume.

[PengZheng]

What's the advantage of using container over using a Vagrant box (virtual machine)?

In my personal experience, containers have far more restrictions than VM, and is generally more difficult to set up correctly. Most of the issues discussed here don't exist if VMs are used, and we won't have #658. Yes, containers are more efficient than VMs, but that does not mean much except for CI. Moreover, Vagrant can work with containers.

For individual developer, Vagrant can serve as a convenient CLI, whether using VMs or containers.

[pnoltes]

I cannot yet build the image, getting the following error:

Setting up python3.10-minimal (3.10.12-1~22.04.2) ... [Errno 13] Permission denied: '/usr/lib/python3.10/pycache/future.cpython-310.pyc.140478821390960'dpkg: error processing package python3.10-minimal (--configure): installed python3.10-minimal package post-installation script subprocess returned error exit status 1 Errors were encountered while processing: python3.10-minimal E: Sub-process /usr/bin/dpkg returned an error code (1) The command '/bin/sh -c DEBIAN_FRONTEND="noninteractive" apt-get update && DEBIAN_FRONTEND="noninteractive" apt-get install -y --no-install-recommends dos2unix git locales-all rsync tar ssh tzdata build-essential ccache cmake curl default-jdk gcc g++ gdb make python3 civetweb libavahi-compat-libdnssd-dev libcivetweb-dev libcpputest-dev libcurl4-openssl-dev libczmq-dev libffi-dev libjansson-dev libxml2-dev libzip-dev rapidjson-dev uuid-dev && apt-get clean' returned a non-zero code: 100

This is also why I am not sure if we should add a dev container or dev virtual machine configuration, this requires us to maintain and document the dev container image and/or virtual machine configuration.

The reason why I see more value in a gitpod config or Github codespaces is that this enable users / potential developers to try out Apache Celix without setting up a machine or even needing a machine (browser based).
So they could start coding with a minimal hassle. But also for gitpod / codespaces we should consider how much effort it is going to be to maintain the gitpod/codespaces configuration.

[rlenferink]

What's the advantage of using container over using a Vagrant box (virtual machine)?

In my personal experience, containers have far more restrictions than VM, and is generally more difficult to set up correctly. Most of the issues discussed here don't exist if VMs are used, and we won't have #658. Yes, containers are more efficient than VMs, but that does not mean much except for CI. Moreover, Vagrant can work with containers.

For individual developer, Vagrant can serve as a convenient CLI, whether using VMs or containers.

For me personally it is that I am familiar with containers and am using them extensively, and am not familiar / have never used vagrant. My experience with VMs is that it is quite often just a bit slower and file-sharing between host and VM is mostly a pain point (how else do we get to use an IDE ?).

I cannot yet build the image, getting the following error:

Setting up python3.10-minimal (3.10.12-1~22.04.2) ... [Errno 13] Permission denied: '/usr/lib/python3.10/pycache/future.cpython-310.pyc.140478821390960'dpkg: error processing package python3.10-minimal (--configure): installed python3.10-minimal package post-installation script subprocess returned error exit status 1 Errors were encountered while processing: python3.10-minimal E: Sub-process /usr/bin/dpkg returned an error code (1) The command '/bin/sh -c DEBIAN_FRONTEND="noninteractive" apt-get update && DEBIAN_FRONTEND="noninteractive" apt-get install -y --no-install-recommends dos2unix git locales-all rsync tar ssh tzdata build-essential ccache cmake curl default-jdk gcc g++ gdb make python3 civetweb libavahi-compat-libdnssd-dev libcivetweb-dev libcpputest-dev libcurl4-openssl-dev libczmq-dev libffi-dev libjansson-dev libxml2-dev libzip-dev rapidjson-dev uuid-dev && apt-get clean' returned a non-zero code: 100

Just for reference, the command I used was cd container && podman build -t apache/celix-dev:2.4.0-ubuntu -f Containerfile.ubuntu . (as non-root user) and the OS I am using is Fedora 38. I removed all my cached images and it still builds; so I am not sure what is going wrong here. What OS are you using?

This is also why I am not sure if we should add a dev container or dev virtual machine configuration, this requires us to maintain and document the dev container image and/or virtual machine configuration.

This indeed is true. Whether we offer a dev container, a VM template or something else, it needs to be maintained.

The reason why I see more value in a gitpod config or Github codespaces is that this enable users / potential developers to try out Apache Celix without setting up a machine or even needing a machine (browser based). So they could start coding with a minimal hassle. But also for gitpod / codespaces we should consider how much effort it is going to be to maintain the gitpod/codespaces configuration.

For me personally I am not extremely content with more-and-more usage of cloud-based technology for these kind of things. The main reason is that only the first 'X' amount of hours is for free and if a certain amount of hours is exceeded then there is a need to pay said providers (obviously). Since I am mostly using my personal environment for this and doing this outside of work I have no intend to pay for these services.

I am definitely not against having a codespaces/gitpod image, but at least there should be an alternative so we don't force people to use these providers if they don't want to (or there should be instructions on how to use it locally).

Maybe we should/could even be offering multiple developer solutions (vanilla container, codespaces container, VM template etc...) ?

[rlenferink]

@pnoltes I just got the Gitpod image working as well, however it is a bit slow. Feel free to give it a try via my own repo: https://gitpod.io/#https://github.com/rlenferink/celix

The speed of it makes me aim to also include the instructions to run it locally (which we maybe can even use the same gitpod image for)

[pnoltes]

@pnoltes I just got the Gitpod image working as well, however it is a bit slow. Feel free to give it a try via my own repo: https://gitpod.io/#https://github.com/rlenferink/celix

The speed of it makes me aim to also include the instructions to run it locally (which we maybe can even use the same gitpod image for)

Nice, I see that there is now more support than only browser vscode. I even could run individually test when installing the TestMate plugin and correctly configuring where the test executables can be found (in cmake-build-debug).

[pnoltes]

@pnoltes I just got the Gitpod image working as well, however it is a bit slow. Feel free to give it a try via my own repo: https://gitpod.io/#https://github.com/rlenferink/celix The speed of it makes me aim to also include the instructions to run it locally (which we maybe can even use the same gitpod image for)

Nice, I see that there is now more support than only browser vscode. I even could run individually test when installing the TestMate plugin and correctly configuring where the test executables can be found (in cmake-build-debug).

I tested the gitpod environment and from what I could see this works nicely. Few remarks:

• I could run individually unit tests from TestMate but not debug. Debug gives an error.
• What is the plans for support? Because I can see people trying this out - and that is a good thing -, but this also means that we a) get questions about the gitpod env and b) we should maintain the gitpod container file (for Apache Celix updates and for gitpod environment updates).

@rlenferink WDYT?

[rlenferink]

I tested the gitpod environment and from what I could see this works nicely. Few remarks:

* I could run individually unit tests from TestMate but not debug. Debug gives an error.

* What is the plans for support?  Because I can see people trying this out - and that is a good thing -, but this also means that we a) get questions about the gitpod env and b) we should maintain the gitpod container file (for Apache Celix updates and for gitpod environment updates).

@pnoltes in my opinion, what we definitely need to deal with is support for building and running the containers. E.g. if the build-ubuntu-container.sh command fails or the Gitpod environment fails to start (because of e.g. a missing dependency) that needs to be fixed.

However, there can be a lot of different IDE related questions and failures and I would say support for that is best effort. In case we have experienced the failure ourselves than we can potentially provide an answer, but otherwise we highly depend on the contributor. We can always ask the contributor if they can propose a fix themselves for the build problem they are facing.

[pnoltes]

we definitely need to deal with is support for building and running the containers

I agree, could you create a issue to update the github actions so that building in a container image is also tested?