dts-scripts refactoring

[DaniilKl]

The problem you're addressing (if any)

The scalability problem of a project might be caused not only by tools being used, but by the way the tools are being used as well.

1) Bad scalability; a. No explicit coding rules and style -> the code is difficult to read and modify, the coding style of every contributor overlays with others, and the codebase shortly became messy; b. Not clearly separated systems -> the way the system works is not linear and, therefore, not clear, especially for novices; c. The language, which is not suitable for large codebases -> language issues and workarounds; d. Configuration and metadata directly stored in the codebase -> not convenient to scale and support; 2) Awful testing and verification routine; a. No test cases prepared -> no empirical evidence of working features; b. No hardware separation -> QEMU cannot be used for testing and verification; 3) High entry level; a. No documentation and comments; b. Highly non-linear structure;

Describe the solution you'd like

Nodes and paths description:

• Deployment - common provisioning logic (not tools or calls to the tools, but decisions what to deploy and why), the only module writing to hardware (via HAL), the common logic makes calls to a hardware specific logic (functions; 8) held by HAL. In case the testing is going on, the common logic instead of communicating with hardware via HAL, communicates with mocking HAL functions (8), that emulate the hardware.
• Configuration - logic responsible from getting and providing hardware information (2, 5). During the boot, it gets hardware inf. (7), and keeps it, then, according to the inf. gained, it chooses (6) the DPP configuration (links to artifacts, lists of hardware specific mocking functions, lists of variables that define deploying workflows, etc.) from Linux FS (or, theoretically, from cloud services (then, it will have to communicate with packaging node first)).
• User interface - is responsible for communication with user. It has some hardware specific stuff (like, whether it is possible to update or to install, or both) on which it decides based on inf. provided by configuration node (5). It provides user following functionalities: start deploying (1), start backup or restore (9) and manage packages (10).
• Backup and restore - logic, responsible for backing up and restoring firmware, takes full advantage of deploying logic (11) and its ability to communicate with hardware (8).
• Packaging - responsible for checking DPP subscription with cloud services, logging in, downloading artifacts, and managing packages. Could be referenced by deploying logic (3) or explicitly by user (10).
• Linux FS - holds some hardware-specific configs (that are currently held in board_config function) in .json files, as well as hardware-specific logic for HAL (some hardware-specific deploying functions and mock functions) and provides it on the start to configuration node (6).
• HAL - Separates all logic from calls to tools, which manage hardware (13); holds some hardware-specific functions between calling to the tools (8) and the deploying logic, so to separate highly non-linear deploying logic from step-by-step hardware-specific logic. The hardware specific log is being dynamically linked depending on the hardware being used (12), or being replaced by hardware-specific mock functions for tests on QEMU.

Documentation:

• Only the workflow of adding a new platform will be explicitly documented (these stuff on Linux FS on the diagram);
• All subsystems must have comments for every function, documenting what it is doing and why.
• Documentation should contain the diagram and a short description of the structure.

Tests:

• Every tool which is being added into meta-dts explicitly should have two short tests: 1) To verify that it is preset; 2) To verify that it is working;
• Every hardware-specific deploying workflow added to HAL should have a list of needed tests;
• Every subsystem should be tested separately or by testing use-cases where every system participates.

Where is the value to a user, and who might that user be?

Pros:

• Highly modular system:
  • Clear subsystems separation: no need to modify several subsystems in case only one subsystem functionality is being extended;
  • Better test coverage: every module can be tested separately enhancing unit-tests, the HAL will make testing on QEMU possible;
• Separation of hardware-dependent logic from common logic:
  • Easier platform support integration: step-by-step hardware-dependent workflows will be separated from highly non-linear common logic;
• Test coverage:
  • Easier changes verification
  • This will be a core argument on whether the changes made are working or not (we could ask for tests in every PR, and merge the PR basing on results).

Cons:

• Adding platform support will require not only extending configuration, but adding mocking functions and tests as well;

Describe alternatives you've considered

No response

Additional context

No response

[artur-rs]

@tym2k1 @m-iwanicki @PLangowski @macpijan marking for review, as agreed on the internal sync

[m-iwanicki]

@DaniilKl Am I right that https://github.com/Dasharo/dts-scripts/pull/26 partialy addresses this issue? It adds HAL and splits modules/functions based on functionality? About coding style/rules those are at least partially forced by pre-commit. Indentation will be when https://github.com/Dasharo/dts-scripts/pull/36 is merged.