[ITF] Enable config persistence in the Integrations Test Framework (ITF)

[ishleenk17]

As a part of ITF (Integrations Test Framework), we had a requirement of config persistence as stated below

While working with integrations, we need a way we can test with any type of customized config for any integration and a way to persist these configs for any future use. This ticket focuses on providing a way to persist these configs.

Eg: MSSQL service integration needs to be tested with 2 different configurations:

Config1: Bring up MSSQL DB and pre-populate user database udb1 Config2: Bring up MSSQL DB and pre-populate user database udb2

Relates: https://github.com/elastic/integrations/issues/5338

[ishleenk17]

Proposed Solution

ITF uses the dockerfiles present under _dev/deploy/docker for each integrations to deploy the services. The dockerfiles under this brings up the service with default config.

We can enable the handling of different configs for these integrations by making changes to these dockerfiles and pushing them to the oblt-cluster which would restart the service. But if we do changes to the existing dockerfiles, persistency will not be maintained.

We can add another service inside docker-compose with the relevant configs, parameters. Pass that service name in the oblt-cli config.

How to use these configs

oblt-cli package cluster service update <clustername> <servicename> -config=<service under docker-compose> Eg: config=service_config1, config=service_config2 If you want the default docker_compose (under deploy) to be picked up, config=default. If you don’t specify any config, default config will be considered while bringing up the container.

[lalit-satapathy]

Can we have a draft PR (as reference) where integration has multiple test configs?

[ishleenk17]

There is a better solution which we can have for this: We can add another service under the docker-compose file and bring up just a particular service container. The example can be seen in the dummy PR which will be linked to this issue.

I have taken example of salesforce. Currently we bring up 1 container with a particular config. This is used for system test as well. During an SDH we wanted to have it tested with another config (bulk response salesforce). We will add [profiles] to service containers which we don't want to be executed in system test

[ishleenk17]

Can we have a draft PR (as reference) where integration has multiple test configs?

@lalit-satapathy : I have shared a dummy PR of how we can manage the configs, while taking an actual example of salesforce which we handled currently.

[ishleenk17]

We will add [profiles] to service containers which we don't want to be executed in system test

We need not add profiles, for system tests even though all the containers (services) are brought up, the system tests runs only for the service which is mentioned in the config file. So, our system tests won't get impacted with this

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!