Refactoring to fit dockerized architecture

[vdecaro]

Dear @chronis10, @tserpes and colleagues,

first of all, thank you for all your work on this project. The state we have reached with the dockerized architecture is certainly satisfactory. As I've already discussed with @tserpes in a recent WP1 update call, I would like to propose a further, hopefully last, step in refactoring the platform. I am convinced that the benefits of the few, simple operations that I will propose below can largely foster the flexibility of the architecture, open it up to potential CI/CD approaches, and allow individual WPs to work on their own tasks independently with minimal integration overhead.

Fragmentation to Self-Contained Repositories

Each Dockerfile should correspond to an application which is executable standalone (no matter whether the application can perform its own job without other participating nodes). As a result, in the current configuration, I identify the following standalone applications, to each of which I assign a dedicated repository:

• teaching-base: project for the base image(s), from which all the other projects will start. This means that the content of the directory basic_images must be a standalone repository;
• teaching-aitoolkit: project for the AI toolkit, and it is going to contain all the rationale concerning the deployment and execution of ML models;
• teaching-sensors: project for all the sensors that can be instantiated, from "file" sensors to cameras and wearables;
• teaching-data: project for persistent storage, e.g., the influxdb instance;
• teaching-logging: project for building logging interfaces.

Clearly, I may be missing something, and any advice about other standalone services is very welcome.

Advantage: this setup allows all WPs to work much more independently, internally and externally, as well as better identify codebases to refer to in case of changes and additions. Effort:

• [x] Creation of the repositories;
• [x] Identification of the code which needs to be moved to each of the repository;
• [x] Perform the corresponding refactoring.

Definition of a "quick start" repository

This repository (namely, teaching-app), includes all the other repositories listed above as linked submodules, and works as a wrapper for executing an application with docker-compose. Of course, git allows for recursive cloning of linked repositories.

Advantage: this repository includes all the scenarios independently of what happens into the linked submodules, and provides a single-purpose access point for starting a teaching application. Effort:

• [x] Creation of the repository;
• [x] Linking the other repositories via git submodule;
• [x] Changing the context for image building within the .yaml files;
• [x] Re-testing the platform.

Setup of Docker Hub

All the images, after the building phase (performed locally) are uploaded to Docker Hub. In this phase, all the images need to be kept up to date by manually committing the changes to Docker Hub.

Advantage: no need of cloning the code within different machines, docker is going to identify the images directly from docker hub. Effort:

• [x] Setup of Docker Hub for the TEACHING project;
• [x] Manual build and upload of all the required images.

Continuous Integration of images

With the use of git actions, there is no need of performing the manual build of the images. For each push on the designated branch, the action will be triggered to re-build the image and re-commit and push it to docker hub, automatizing what I mentioned in the previous point. Advantage: automatization of the image build, as well as opening the platform to CI/CD mechanisms. Effort:

• [x] Definition and setup of the git action for each repository;
• [x] Testing of the git action with "dummy" commits.

Final remarks and comments

As I mentioned in the beginning of the issue, I am strongly convinced that the effort (imho, small) required by this last, major refactoring, will be amply repaid by the benefits we will have in integrating any future developments, as well as by the overall quality of the platform. Again, any comments, advice, criticisms you have are welcome.

[patriziodazzi]

Probably it could make sense to add a new repo for the federated/cloud part concerning the aggregation of modules. An open question is if to introduce an even more repo for networking related issues.

[chronis10]

Hello to everyone, First of all, after the difficulties of building the basic images on imx8 and the discussion at the consortium meeting, I created a GitHub action that builds the basic images for arm64 and amd64 architectures. Also the same GitHub action pushes the images to a docker hub repository (chronis10/teaching_basic_image). The selection of the proper image for the architecture can be done with the proper tag. For example, for imx8 the proper tag is ":arm64". For now the action can be triggered manually but in the future we can find a better solution. The following days I will update the docker compose files and I will also create some bash scripts for easier deployment.

In the meanwhile, I think that the proposed solution of @vdecaro is in the correct direction, but the full decomposition of the whole platform can be time consuming and I think we should combine our efforts to optimize the performance of the platform as it is until the discussions about the refactoring are completed.

Also it's important to have in mind that the platform already has a very flexible module (process module) that offers a communication interface, has the capability to aggregate data for multiple sensors and can create batches of data using environment parameters. Finally the process module has the capability to execute custom code. For example, that module was used in the workshop in AVL for the integration of the communication module.

[vlomonaco]

I think we can close this issue, let me know if you want me to re-open it!