[META] [ITF] Scoping and development of test framework for integrations

[ishleenk17]

[ITF] Integrations Testing Framework Problem summary: The service integration team currently supports a wide number of integrations across platforms and standalone/cloud deployments. There is a time lapse when an integration goes GA and an issue gets reported from customers. The challenge is to quickly bring-up a specific service to test the corresponding integration and replicate the user issue when necessary.

Understand the scope of Development challenges faced while bringing up test environment framework. Initial document

• [x] https://github.com/elastic/integrations/issues/5488
• [ ] https://github.com/elastic/integrations/issues/5489
  • [x] https://github.com/elastic/integrations/issues/5490
  • [ ] https://github.com/elastic/integrations/issues/6354
  • [ ] User guide for ITF https://github.com/elastic/obs-infraobs-team/issues/1167

[ishleenk17]

Had an initial discussion with @cachedout and @kuisathaverat.

We would be contrasting the working of the elastic-package tool and oblt-cli tool for bringing up the stack, service and integration. This would help us to finalize if we can use either of them or if we have to use a combination of the 2 tools to setup the test environment.

The major difference between the two is that oblt-cli brings up the setup on cloud whereas elastic-package brings it up locally.

[ishleenk17]

There are 3 parts to be considered while bringing up the integration in an automated test environment:

1. Set up the stack: It includes bringing up Kibana, Elasticsearch, Fleet Server, Elastic Agent. This requires the creation of a new customized deployment for bringing up the stack through oblt-cli

2. Bringup the Service: oblt-cli should be able to bringup the service for which the integration is to be run. One of the ways to achieve this is to replicate bringup of the docker service by using the details from _dev folder (used for system tests) for all the integrations.

   • A disadvantage of this is how do we handle situations where there are no system tests (_dev folders).

3. Add integration :

   • What are the different types of inputs that need to be passed for integrations? Eg: hosts, metrics path etc.
   • Can these inputs be generalized for the integrations?
   • How frequently are these inputs changed?

[ishleenk17]

An initial effort in this direction is captured here. This has a few examples of how a deployment on integration looks like in K8s. cc: @kuisathaverat

[ishleenk17]

@kuisathaverat : Some open questions regarding these 2 parts:

3. Add integration :

   • What are the different types of inputs that need to be passed for integrations? Eg: hosts, metrics path etc.
   • Can these inputs be generalized for the integrations?
   • How frequently are these inputs changed?

While I try to get answers for the above questions. I have couple of queries in my mind wrt Elastic Cloud

1. How do we do testing in Elastic Cloud for changes that I have made locally in my system? In the elastic package tool, we have a package registry that picks up the integrations with my local changes when I update the registry. Is there something similar in Elastic cloud/oblt-cli as well?

2. Bringup the Service: oblt-cli should be able to bringup the service for which the integration is to be run. One of the ways to achieve this is to replicate the bringup of the docker service by using the details from _dev folder (used for system tests) for all the integrations.

1. Since we are planning to use the dockerfiles directly from _dev folders of integrations, is there a way instead of using the code we can somehow use the Dockerfile directly? That way it would avoid duplication and if there are any changes made in the _dev dockerfiles, it would automatically be taken into account.
2. The parts that would be changing for testing from the developers perspective would be the stack version or the parameters of the integration. The part that is going to remain constant would be the service. It would be good if we can have a service in a cluster and there is no need to relaunch the service when it is to be used by a different developer or for a different stack version. Is this design attainable?

[kuisathaverat]

How do we do testing in Elastic Cloud for changes that I have made locally in my system? In the elastic package tool, we have a package registry that picks up the integrations with my local changes when I update the registry. Is there something similar in Elastic cloud/oblt-cli as well?

Currently, there is nothing implemented, we implement a command to upload the package to a package-registry linked to the cluster. Something similar to the elastic-package build

Since we are planning to use the dockerfiles directly from _dev folders of integrations, is there a way instead of using the code we can somehow use the Dockerfile directly? That way it would avoid duplication and if there are any changes made in the _dev dockerfiles, it would automatically be taken into account.

Do you mean to use local _dev code for the deployment?

The parts that would be changing for testing from the developers perspective would be the stack version or the parameters of the integration. The part that is going to remain constant would be the service. It would be good if we can have a service in a cluster and there is no need to relaunch the service when it is to be used by a different developer or for a different stack version. Is this design attainable?

In general, all services are really cheap to deploy and configure, share service between developers makes things complicated, for example, several Elastic Agents with different configurations will take logs and metrics from the service, which could make it difficult to correlate errors in logs with calls to the API and stuff similar. I'd said that for most of the services, we should try to isolate developer's process. However, we will find some services that are expensive to deploy (in time and money e.g. VMWare vSphere) that we should share so we will have to see each particular case to find a solution.

[ishleenk17]

How do we do testing in Elastic Cloud for changes that I have made locally in my system? In the elastic package tool, we have a package registry that picks up the integrations with my local changes when I update the registry. Is there something similar in Elastic cloud/oblt-cli as well?

Currently, there is nothing implemented, we implement a command to upload the package to a package-registry linked to the cluster. Something similar to the elastic-package build

We do 2 steps to update the registry. We build the integration with elastic-package build and update the stack with elastic-package stack up --services package-registry. These 2 steps are needed.

Since we are planning to use the dockerfiles directly from _dev folders of integrations, is there a way instead of using the code we can somehow use the Dockerfile directly? That way it would avoid duplication and if there are any changes made in the _dev dockerfiles, it would automatically be taken into account.

Do you mean to use local _dev code for the deployment?

That's right, can we use the _dev code directly to have a single code base for Dockerfiles?

cc: @lalit-satapathy @ruflin

[ruflin]

For the upload package part, https://github.com/elastic/elastic-package/issues/1084 is currently in the works. It will skip the registry meaning a package can be pushed to any Kibana version. Hope this helps in this scenario.

[kuisathaverat]

That's right, can we use the _dev code directly to have a single code base for Dockerfiles?

I think so, we have to make it part of the configuration we deploy and that's it.

[ishleenk17]

For the upload package part, elastic/elastic-package#1084 is currently in the works. It will skip the registry meaning a package can be pushed to any Kibana version. Hope this helps in this scenario.

@ruflin Here we are not making use of elastic package, but want to do something similar through oblt-cli; uploading the package to Kibana for testing the local changes.

[lalit-satapathy]

In general, all services are really cheap to deploy and configure, share service between developers makes things complicated

Regarding service re-use, what we really need is service setup-reuse. The goal should be setup once and reuse multiple times. Service launch on fly is fine, so long as existing setup can be re-used.

Keen to see more details on the service setups, for example how the scenarios below works:

• (MS-SQL-setup1) Basic service which brings up MS-SQL DB
• (MS-SQL-setup2) Service which brings up MS-SQL DB and pre-populates few user databases

How MS-SQL-setup2 and MS-SQL-setup1 will be setup and is there is code reuse between these two?

[ruflin]

@ruflin Here we are not making use of elastic package, but want to do something similar through oblt-cli; uploading the package to Kibana for testing the local changes.

If you don't use elastic-package to upload the package to Kibana, what do you use then?

[kuisathaverat]

Regarding service re-use, what we really need is service setup-reuse. The goal should be setup once and reuse multiple times. Service launch on fly is fine, so long as existing setup can be re-used.

That's the goal, Elastic Stack (ES + Kibana + Integrations server) + Elastic Agent + service is deployed and configured only once, then you can make as many updates (or configuration changes) as you need on the packages and service.

Keen to see more details on the service setups, for example how the scenarios below works: (MS-SQL-setup1) Basic service which brings up MS-SQL DB (MS-SQL-setup2) Service which brings up MS-SQL DB and pre-populates few user databases How MS-SQL-setup2 and MS-SQL-setup1 will be setup and is there is code reuse between these two?

In this case the only changing part of the deployment is the two services of MS-SQL we are running. Here we have two options:

• Deploy isolate integrations per MS-SQL. we have two MS-SQL deploys and two Elastic Agents one per MS-SQL deployment.
• Deploy only one Elastic Agent and attach the two MS-SQL deployments to it.

I have a question here, we are evaluating using the _dev/docker folder as a system to run the service, in the case you expose there are two MS-SQL configurations but there is only one package _dev/docker folder. How MS-SQL's configurations are defined in this case?

[lalit-satapathy]

@kuisathaverat, Assuming, for the following task, we will need to do it outside the docker preferably in a script that can be run in the target service. > pre-populates few user databases

[kuisathaverat]

@kuisathaverat, Assuming, for the following task, we will need to do it outside the docker preferably in a script that can be run in the target service.

Would it be enough to allow to pass of a custom script as part of the CLI to execute when the service is ready?

In some BD containers, it is possible to configure init database script, so it is also possible to add the init sequence to the _dev/docker code

[lalit-satapathy]

Would it be enough to allow to pass of a custom script as part of the CLI to execute when the service is ready?

@kuisathaverat, @ishleenk17,

Let's cover some of these scenarios in our initial testing being done. There are multiple cases (DBs or other services), where we will require post-processing on the service to bring it to a desired state, before any testing can begin. Any Implementation approach should be ok, so long this is supported as part of the overall framework.

[ishleenk17]

@ruflin Here we are not making use of elastic package, but want to do something similar through oblt-cli; uploading the package to Kibana for testing the local changes.

If you don't use elastic-package to upload the package to Kibana, what do you use then?

We have to add support of pushing the package to cloud through oblt-cli

[ishleenk17]

We will be addressing the below use-cases

Usecase No	Usecase Type	Stack	Service	Elastic Agent	Integration	Comments
1	All on Cloud	Cloud	Cloud	Cloud	Cloud	Oblt-cli usage
2	Service on cloud	Local	Cloud	Local	Local	Elastic package and oblt-cli usage
3	Agent on local	Cloud	Cloud	Local	Cloud	Oblt-cli usage
4	All on local	Local	Local	Local	Local	Current Manual way using elastic package.

Phase 1 MVP:

Integration: MSSQL Usecases to be handled: Usecase 1: Through a single oblt-cli command, we should be able to bring up stack, service, add agent and integration in cloud. Usecase 2 : Stack, agent, integration will be brought up using elastic package. Oblt-cli should launch service in cloud and provide a way to connect to it through the stack.

While handling the above usecases, we should ensure that we are able to test the local changes. Currently, we are able to handle this in elastic package by pushing them to the registry. We should be able to do the same through oblt-cli.

Phase 2 MVP:

Integration: Oracle/Vsphere

Enhancements to Phase 1 usecases:

1. Support for logs.
2. Oblt-cli change for supporting different configurations
3. How will we handle if there are multiple containers being spun in a dockerfile by default
4. Being able to host multiple services in same cluster

Phase 3 MVP:

Integration: IIS

Enhancements to Phase 1 usecases:

1. Bringup of the services in OS other than Linux. For eg: IIS in windows
2. Bringup of cloud services

Usecases to be handled: 4 (Everything on local setup). Currently, we do this through elastic-package. We should be able to attain it through oblt-cli. Handling this use-case is not a must, but will be good to have in case it is not system heavy since the pre-requisite for this would be having K8s cluster on the local machine.

cc: @kuisathaverat @cachedout @lalit-satapathy @rameshelastic @ruflin

[kuisathaverat]

[WIP] this is the PoC we have implemented so far; it is not published yet. We are working on encapsulating the commands we have into oblt-cli

C4Component
title Integrations test environment

Boundary(cloud, "Public Cloud"){
  Boundary(oblt-cluster, "Oblt Clusters"){
    Boundary(gke, "Google Kuberneres Engine"){
      Boundary(pod, "Integration Pod"){
        System(dind,"Docker in Docker")
        System(integration,"Service")
        System(elastic-agent,"Elastic Agent")
        System(package-registry,"Package Registry")
        System(workspace,"Workspace")
        System(git-clone,"Git Clone")
      }
      System(shared,"Shared Volume")
    }

    Boundary(ess, "Elastic Cloud"){
      Boundary(stack, "Elastic Stack"){
        System(kibana, "Kibana")
        System(fleet, "Fleet Server")
        System(es, "Elasticsearch")
      }
    }

    Boundary(github, "GitHub"){
      System(source, "Source code repository")
      System(ci, "CI")
    }
  }
}

Boundary(laptop,"Local environment"){
  Person(user, "Developer")
  System(oblt-cli, "Oblt-cli")
  System(browser, "Browser")
}

Rel(user, oblt-cli, "Uses")
Rel(user, browser, "Uses")
Rel(browser, kibana, "Uses")
Rel(browser, workspace, "Uses")

Rel(oblt-cli, ci, "Commands")
Rel(ci, source, "Code")

Rel(oblt-cli, package-registry, "Publish")
Rel(oblt-cli, shared, "Sync")

BiRel(es, kibana, "")
BiRel(es, fleet, "")
BiRel(fleet, kibana, "")

Rel(integration, dind, "Run service")
Rel(elastic-agent, integration, "Collect data")
Rel(integration, dind, "Uses")
Rel(package-registry, dind, "Uses")
Rel(workspace, dind, "Uses")
Rel(git-clone,source, "Clone")
Rel(elastic-agent,fleet, "Send data")
Rel(kibana,package-registry, "Install packages")

Rel(integration,shared, "Shared storage")
Rel(elastic-agent,shared, "Shared storage")
Rel(git-clone,shared, "Shared storage")
Rel(dind,shared, "Shared storage")
Rel(package-registry,shared, "Shared storage")
Rel(workspace,shared, "Shared storage")

These are some sample commands

# create a new cluster
oblt-cli packages cluster create --stack-version 8.7.0 --integration microsoft_sqlserver

# publish a package
oblt-cli packages publish --cluster-name integrations-aaeeff-microsoft-sqlserver --package-folder /home/user/src/integrations/packages/microsoft_sqlserver

# Access logs
oblt-cli packages logs service --cluster-name integrations-aaeeff-microsoft-sqlserver 
oblt-cli packages logs elastic-agent --cluster-name integrations-aaeeff-microsoft-sqlserver 
oblt-cli packages logs package-registry--cluster-name integrations-aaeeff-microsoft-sqlserver 

# Access containers
oblt-cli packages shell service --cluster-name integrations-aaeeff-microsoft-sqlserver 
oblt-cli packages shell elastic-agent --cluster-name integrations-aaeeff-microsoft-sqlserver 
oblt-cli packages shell package-registry --cluster-name integrations-aaeeff-microsoft-sqlserver 

# VSCode on the browser (experimental)
oblt-cli packages vscode --cluster-name integrations-aaeeff-microsoft-sqlserver 

# Tools container (experimental)
oblt-cli packages shell workspace --cluster-name integrations-aaeeff-microsoft-sqlserver 

[kuisathaverat]

We did the first demo today with the implementation so far https://github.com/elastic/observability-test-environments/blob/main/docs/user-guide/integrations.md @ ishleenk17 give me some feedback about some stuff we have to change or implement:

• The cluster name is passed on every command. we have to find a way to pass it one time
• The current publish command runs some tests. We have to remove the test and put then on separate commands.
• The shell command will change to exec
• When changes are made on Docker compose files we have to push them to the pod, we have to implement a command to push those changes to the pod and restart the Docker compose service
• The oblt-cli packages logs service is not working
• The ports command that forwards ports from the pod to local needs an option to pass the ports
• We need a command to launch additional integrations in the same workspace
• Implement use case 3

[ishleenk17]

Some other feedbacks:

• The oblt-cli packages logs service command would also have to cater to scenario when we will have multiple services in a single stack version.
• Implementation of usecase 3, which should be a small change (codewise), where just the service is going to be exposed on cloud.
• Ability to create the cluster/service from a specific repo (other than the main repo).

[ishleenk17]

• The cluster name is passed on every command. we have to find a way to pass it one time

On second thought, we would need passing of cluster name in every command as there might be a scenario where we might be working on 2 clusters simultaneously. Cluster name will give the identification of the clusters while using the oblt-cli commands.

[ishleenk17]

@kuisathaverat: In the add Integration Service Deploy, I think for now we are able to serve only the metrics. For logs, the elastic agent needs access to the log file location on the local system. This scenario is yet to be addressed.

[kuisathaverat]

For logs, the elastic agent needs access to the log file location on the local system.

This command should tee the logs from the service to a local file, then you can configure your Elastic Agent to use that file as input

oblt-cli packages cluster logs service --cluster-name foo | tee /local/path/file

[ishleenk17]

For logs, the elastic agent needs access to the log file location on the local system.

This command should tee the logs from the service to a local file, then you can configure your Elastic Agent to use that file as input

oblt-cli packages cluster logs service --cluster-name foo | tee /local/path/file

We should add this logic to oblt-cli for handling the logs usecase.

[ishleenk17]

Demo of Phase 1 MVP : https://github.com/elastic/obs-infraobs-team/issues/1113

[ishleenk17]

Current Status:

Usecase 1: All on cloud and Usecase 2: Service on Cloud have been completed. These were tested with MYSQL. We would now like to publicize it and try it out within InfraObs-Team with other Integrations.

Known Blockers:

• Currently, there is a blocker in proceeding with All in Cloud Usecase. @kuisathaverat @cachedout : Could you please link the blocker issue here and let us know the ETA for this?
• Unable to get logs for Service Only Usecase https://github.com/elastic/integrations/issues/7141. Workaround is there to proceed with it.

Next Steps:

We would slightly like to reorder the MVPs. We can prioritize 2 things:

1. For supporting https://github.com/elastic/integrations/issues/6354, we need an option through the CLI to be able to trigger a specific service container from within the docker file.
2. Support for Cloud Integrations in ITF https://github.com/elastic/integrations/issues/7115

cc: @rameshelastic