search

exasol / integration-test-docker-environment

A docker-based environment for integration tests with the EXASOL DB.
https://exasol.github.io/integration-test-docker-environment/
MIT License
6 stars 2 forks source link

Update resource mechanism for templates #348

Open ckunki opened 1 year ago

ckunki commented 1 year ago

Current implementation

Currently ITDE uses some configuration files

Folder docker_db_config_template contains a subfolder for each relevant version of Exasol database, e.g. 7.1.0, 7.1.1, etc.. Some subfolders are symlinks as the respective version of Exasol database may share the templates with another version, e.g. subfolder 7.1.1 is a symlink to 7.1.0.

Git commit hook githooks/update_packaging.sh copies the templates to directory exasol_integration_test_docker_environment/docker_db_config/ as during runtime symlinks are not supported and we need actual files.

ITDE's python implementation uses pkg_resources to locate the template files, read them into a string and substitute placeholders by actual values.

Rating

In summary we have at least two problems and an architecture with significant dependencies and complex interactions:

Proposed change

The current ticket therefore proposes to replace the current implementation:

Acceptance criteria

  1. Folder docker_db_config_template is removed
  2. Folder exasol_integration_test_docker_environment/docker_db_config/ only contains subfolders 7.0, 7.1, and 8.17.0
  3. Deprecated pkg_resources is replaced by importlib.resources
  4. There is a python implementation
    • mapping Exasol database versions to template files
    • able to allocate and read a template file
  5. File noxfile.py is updated to no longer generate the list of supported versions of Exasol database based on the folders in docker_db_config_template but return a plain list

import importlib.resources from exasol_integration_test_docker_environment.lib import PACKAGE_NAME

# add to file eitde/lib/test_environment/spawn_test_database.py
def resource(version: str, filename: str) -> str:
    def directory(version: str):
        if version.startswith("7.0."):
            return "7.0.x"
        if version.startswith("7.1."):
            return "7.1.x"
        if version == "8.17.0":
            return version
        raise RuntimeError("Unsupported version " + version)
    res = (
        importlib.resources.files(PACKAGE_NAME)
        / "docker_db_config" / directory(version) / filename
    )
    with importlib.resources.as_file(res) as template:
        with open(template, "r") as file:
            return file.read()

# sample usage 
content = resource("7.1.1", "EXAConf")
print(f'{content}')
# noxfile.py
# or eitde/lib/test_environment/db_version.py
def get_db_versions() -> List[str]:
    """
    The ITDE only supports EXAConf templates for docker-db versions in the
    format major.minor.bugfix.

    If the specified version number contains additions, such as "d1", or
    "prerelease", ITDE filters these to abtain the base version number.
    Function get_db_versions however uses raw versio numbers to generate the
    test matrix for GitHub Actions requiring to adapt the list for images
    with special names.
    """
    result = [
        "default", "7.0.0", "7.0.1", "7.0.2", "7.0.3", "7.0.4", "7.0.6",
        "7.0.7", "7.0.8", "7.0.9", "7.0.10", "7.0.11", "7.0.12", "7.0.13",
        "7.0.14", "7.0.15", "7.0.16", "7.0.17", "7.0.18", "7.0.19", "7.0.20",
    ]
    result += [
        "7.1.0", "7.1.1", "7.1.2", "7.1.3", "7.1.4", "7.1.5", "7.1.6",
        "7.1.7", "7.1.8", "7.1.9", "7.1.10", "7.1.11", "7.1.12",
        "7.1.13", "7.1.14", "7.1.15", "7.1.16", "7.1.17",
    ]
    # Until an image file is available on DockerHub version 8.17.0 is replaced
    # by the corresponding pre-release.
    result += [
        "7.1.0-dl",
        "prerelease-8.17.0",
    ]
    return result
ckunki commented 1 year ago

Alternative approach: 

# file eitde/lib/test_environment/db_version.py
from typing import (List, Optional)

# eitde/lib/test_environment/db_version.py

class DbVersion:
    LATEST="7.1.17"
    DEFAULT="default"

    @classmethod
    def supports_custom_certificates(version: Optional[str]) -> bool:
        return DbVersion.from_string(version) > DbVersion(7, 0, 5)

    @classmethod
    def ci_build_versions(self) -> List[str]:
        """
        The ITDE only supports EXAConf templates for docker-db versions in the
        format major.minor.bugfix.

        If the specified version number contains additions, such as "d1", or
        "prerelease", ITDE filters these to abtain the base version number.
        Function get_db_versions however uses raw versio numbers to generate the
        test matrix for GitHub Actions requiring to adapt the list for images
        with special names.
        """
        return [
            DbVersion.DEFAULT,
            # 7.0.x
            "7.0.0", "7.0.1", "7.0.2", "7.0.3", "7.0.4", "7.0.6", "7.0.7",
            "7.0.8", "7.0.9", "7.0.10", "7.0.11", "7.0.12", "7.0.13", "7.0.14",
            "7.0.15", "7.0.16", "7.0.17", "7.0.18", "7.0.19", "7.0.20",
            # 7.1.x
            "7.1.0", "7.1.1", "7.1.2", "7.1.3", "7.1.4", "7.1.5", "7.1.6",
            "7.1.7", "7.1.8", "7.1.9", "7.1.10", "7.1.11", "7.1.12",
            "7.1.13", "7.1.14", "7.1.15", "7.1.16", "7.1.17",
            # special
            "7.1.0-dl",
            # Until an image file is available on DockerHub version 8.17.0 is replaced
            # by the corresponding pre-release.
            "prerelease-8.17.0",
        ]

    def __init__(self, major, minor, stable):
        self.major = major
        self.minor = minor
        self.stable = stable

    @classmethod
    def from_string(cls, db_version_str: Optional[str]):
        db_version = db_version_str
        if db_version_str in (None, DEFAULT_VERSION):
            db_version = LATEST_DB_VERSION
        if db_version.endswith("-d1"):
            db_version = "-".join(db_version.split("-")[0:-1])
        if db_version.startswith("prerelease-"):
            db_version = "-".join(db_version.split("-")[1:])
        version = tuple([int(v) for v in db_version.split(".")])
        if len(version) != 3:
            raise ValueError(f"Invalid db version given: {db_version}")
        return DbVersion(version[0], version[1], version[2])

    def __str__(self):
        return f"{self.major}.{self.minor}.{self.stable}"

    def to_tuple(self):
        return self.major, self.minor, self.stable

    def __eq__(self, other) -> bool:
        return self.to_tuple() == other.to_tuple()

    def __lt__(self, other) -> bool:
        return self.to_tuple() < other.to_tuple()

    def __le__(self, other) -> bool:
        return self.to_tuple() <= other.to_tuple()

    def __ge__(self, other) -> bool:
        return self.to_tuple() >= other.to_tuple()

    def __gt__(self, other) -> bool:
        return self.to_tuple() > other.to_tuple()
tkilias commented 1 year ago

We should use a mapping File which is stored as Ressource and don't define the mapping in python. Because this way CI and Code use the same source. Further we can add a fallback mechanism for new versions.

tkilias commented 1 year ago

@ckunki please check for overlapping tickets

ckunki commented 1 year ago

@ckunki please check for overlapping tickets

Hello @tkilias besides #344 neither by searching for "docker_db", "config", nor by "template" I could not find an overlapping ticket.

ckunki commented 1 year ago

How about a mapping file like the following? 

version-mapping:
  - version: latest
    - alias: 8.18.1
  - version: default
    - alias: 8.18.1
  - version: 7.0.0
    - exaconf: 7.0.0
  - version: 7.1.17
    - exaconf: 7.1.0
  - version: 7.1.0-dl
  - version: prerelease-8.17.0
ckunki commented 1 year ago

updated proposal for mapping file:

versions:
  - latest: 8.18.1
  - default: 8.18.1
  - mapping:
    # default for entries docker-image or exaconf is value of version
    - version: 7.0.0
    - version: 7.0.1
      exaconf: 7.0.0
    - version: 7.1.0
      docker-image: 7.1.0-dl
    - version: 8.17.0
      docker-image: prerelease-8.17.0