Unit tests in CI Pipeline

[onefloid]

To simplify the contributions and ensure high code quality, it would be beneficial to have a Github CI pipeline that runs the test suite. We need a TM1 server to run the tests, as mocking the whole rest api is not very practical. This could be a server that can be accessed from the pipeline or a TM1 database running in a container. I'm not sure if there are any license issues. Have you discussed this before?

[MariusWirtz]

I totally agree. @nicolasbisurgi is actually working on this as we speak :)

I think we can open a PR soon and continue the conversation there

[nicolasbisurgi]

Hi @onefloid I'm glad you opened this ticket. As @MariusWirtz mentioned I'm working on some GH Actions for our repos. For this specific case I was thinking on the below approach, I'm curious to hear your thoughts as well!

Workflow Steps:

1. Pull TM1 Objects from External Repository:

   • This external TM1 Repository will contain all the necessary objects that are required to run the testing scripts (i.e.: cubes, dimensions, csv files, json's, etc.). This should be done using Pull from TM1.Git integrations module.

2. Variables Configuration:

   • For each environment, we define two key variables:
     • TM1_CONNECTION_DETAILS: A stringified dictionary containing connection parameters, which will be converted into a Test/config.ini file during the workflow.
     • TM1_CONNECTION_PASSWORD: A secret variable to securely store the TM1 connection password.
   • OPTIONAL: We could make use of the GH repository environments. This would allow us to have the same variables but for different TM1 versions (e.g., v11_onprem, v11_cloud, v12_aws, etc).

3. Dependency Installation:

   • The action upgrades pip to the latest version.
   • Installs necessary packages:
     • Runs pip install .[pandas,dev] to install the repository with extra dependencies.
     • Executes pip install pytest pytest-xdist to install testing frameworks.

4. Execute Tests:

   • Utilizes pytest to run all tests located in the Test/ directory, ensuring that the TM1 objects integrate correctly without issues.

Optional Design Consideration:

-Multi-Environment Support:

• By leveraging the repository's environments feature, we can define and run the workflow across multiple TM1 versions.

-Forking Considerations:

• When a user forks the repository, the action retains the variable placeholders (without the actual secret values).
• This allows forked repositories to carry over the workflow structure, so contributors can set up their own environment variables and run the action within their own TM1 environments.

---

Deployment Diagram

flowchart LR
    subgraph GitHub_Action[GitHub Action Workflow]
        A[GitHub Action]
    end

    subgraph External_Repos[External TM1 GitHub Repositories]
        B1[TM1 Repo for v11_onprem]
        B2[TM1 Repo for v11_cloud]
        B3[TM1 Repo for v12_aws]
    end

    subgraph TM1_Environments[TM1 Models]
        C1[TM1 Model v11_onprem]
        C2[TM1 Model v11_cloud]
        C3[TM1 Model v12_aws]
    end

    A --> B1
    A --> B2
    A --> B3

    A --> C1
    A --> C2
    A --> C3

    style GitHub_Action fill:#f9f,stroke:#333,stroke-width:1px
    style External_Repos fill:#bbf,stroke:#333,stroke-width:1px
    style TM1_Environments fill:#bfb,stroke:#333,stroke-width:1px

Flow Diagram

flowchart TD
    Start([Start GitHub Action])

    Start --> LoopEnvs[Iterate Through Environments]

    subgraph Environment Loop
        LoopEnvs --> ReadVars[Pull TM1 Objects from Environment-Specific External Repo if required]
        ReadVars --> PullObjects[Convert Details to config.ini]
        PullObjects --> ConvertConfig[Read Environment Variables]
        ConvertConfig --> UpgradePip[Upgrade pip]
        UpgradePip --> InstallDeps[Install Dependencies]
        InstallDeps --> IntegrateTM1[Pull TM1 Objects from GitHub into TM1 Model]
        IntegrateTM1 --> ExecuteTests[Execute pytest Tests]
        ExecuteTests --> CollectResults[Collect Test Results]
    end

    CollectResults --> CheckNextEnv{More Environments?}
    CheckNextEnv -- Yes --> LoopEnvs
    CheckNextEnv -- No --> End([End Action])

I think this would allow us (Cubewise) to test it on our end but it will also allow any contributor to fork it and adjust the variable/environments to their specific needs. Please share your thoughts as this is not written in stone by any means!

Cheers,

nico

[onefloid]

Thank you, @nicolasbisurgi, for sharing your thoughts. The ability to test different TM1 versions and platforms is very beneficial and that it should be possible to use the pipeline in forks is very nice. I'm not sure if I would use it in a fork, as I have a local development environment with a running TM1 server which is very fast and handy.

A general question first: Have you considered running the TM1 server in a container? Then you could use this image for your testing job and it would simplify a lot.

To step 1) Is it planned to change the behavior that each test sets up its required TM1 objects? At the moment it is very easy to set up an empty tm1 database locally and start developing. This is very easy and you can run the tests again and again without worrying about the current state of the test database. This means more overhead and longer execution times (especially if you run tests single-threaded). If you run the tests in parallel, this is very useful because you can create separate TM1 objects for each test.

Step 2) I am not familiar with Github Actions. But is it possible to save the TM1_CONNECTION_DETAILS as an environment variable in your runner? So you can avoid storing credentials in Github variables?

Cheers,

onefloid

[nicolasbisurgi]

Hi @onefloid,

Thank you for your feedback.

Regarding your observation on Step 1: I apologize for the oversight on my part. I was under the impression that we needed a pre-loaded TM1 instance to run this. However, I ran into Marius yesterday, and he confirmed that an empty instance should suffice.

Regarding your observation on Step 2: As far as I know, GitHub Actions are non-persistent, so environment variables need to be set on every run.

Aside from that, I think adopting the following practice would be beneficial: the general workflow for TM1 contributors is to fork the repository, create a new branch with their changes and respective test cases, and then raise a PR. However, if they set up the values for the two variables (TM1_CONNECTION_DETAILS and TM1_CONNECTION_PASSWORD) for their respective environments, contributors could test locally before submitting the PR. My understanding is that when you clone a repository, the environments and variables are passed but without their original values, so you can fill them with your specific connection settings. This would be really beneficial as it would better distribute the load of testing, and I believe GitHub secrets provide sufficient security.

Regarding the use of TM1 Server as a container: I would love to explore that option, but I'm not sure how that's possible. If you have experience with this, please share it so we can see how we can incorporate it into the workflow.

Let me know your thoughts. I'll be available to work on this starting mid next week I think.

Cheers!

nico