Add libaio to docker-fleet-agent so that oracle client can be installed on need basis

[agithomas]

Oracle system test could not be created because oracle-client is not present in the docker-fleet-agent as part of elastic-package stack.

To install the oracle client, following library is a dependency.

• libaio

Since, there is no root permission on the image, the above library could not be installed.

Please help by installing libaio to docker-fleet-agent?

[agithomas]

Reference : https://docs.oracle.com/en/database/oracle/oracle-database/21/lacli/install-instant-client-using-zip.html#GUID-D3DCB4FB-D3CA-4C25-BE48-3A1FB5A22E84

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[cmacknz]

Can you provide more detail on what you are trying to do? I can see this might have to do with system testing, so I am not sure that adding this dependency to the Docker image we provide to users would make sense.

Depending on what exactly you are trying to do there may be several ways to solve this problem. The simplest from what you have written would be to build a custom Docker image for system testing that uses the existing agent Docker image as its base image.

[agithomas]

Hi @cmacknz , Sorry for the delayed response. I had been exploring alternatives but did not get any.

Please find the discussion i had with the eco-system team on the topic : Discussion with EcoSystem

Sharing a few thoughts

• If there exist a requirement from the customer to receive Oracle metrics from the Oracle database server, libaio is the minimum that is needed to be baked to the base image. Oracle specific files can be copied. The liblcntsh library, part of oracle client would need libaio
• I considered the option of using the base image and install libaio over it. But, base image doesn't have the root access or sudo to install the libaio and i could not proceed further.

Kindly advice

[cmacknz]

I considered the option of using the base image and install libaio over it. But, base image doesn't have the root access or sudo to install the libaio and i could not proceed further.

Which base image did you test? Does it fail building a new image, or executing in the existing image? What commands did you run?

[agithomas]

I think, i found the issue i was doing which solved the first problem

FROM docker.elastic.co/elastic-agent/elastic-agent-complete:8.4.0-SNAPSHOT
USER root
RUN apt-get update && apt-get -y install \
    libaio1

with this i could create the base image.

root@802829fbe682:/usr/share/elastic-agent# apt search libaio
Sorting... Done
Full Text Search... Done
libaio-dev/focal 0.3.112-5 amd64
  Linux kernel AIO access library - development files

libaio1/focal,now 0.3.112-5 amd64 [installed]
  Linux kernel AIO access library - shared library

This would help me resolve the issue in my local system to have a docker-fleet-agent having the dependency installed.

But, when the CI/CD pipeline, how can i ensure this custom image (after i push this image to docker repo) is the one that will be used for running the system test? This had been the concern raised by the ecosystem team

[cmacknz]

There are a few ways to get make this image available.

1. The simplest if you are using docker-compose for testing is to define it as a service with a build command referencing a Dockerfile. Here is an example docker-compose file and the Dockerfile it references. If you are not using docker-compose you could just add a command to build the image before running the tests (magefile, Makefile, Python or Go shell command, etc).

2. You can publish a custom image to docker.elastic.co. Here is a PR doing this for another test utility. You would likely want to contact the observability robots team for help if you chose this approach.

[agithomas]

Reopening the discussion thread as a solution to this problem must be found as part of 8.6 deliverable.

@jsoriano, I noticed a readme by you . Does this solve the above problem?

[jsoriano]

@jsoriano, I noticed a readme by you . Does this solve the above problem?

Not sure of understanding. This links is to a docker compose file used as base for running elastic agents with custom options, as described here. But in principle the base image used is still the same. I guess this could solve the problem if what is exposed is enough to run the tests with the library, if not, maybe it could be extended.

But is the library only needed on build time, or also on runtime? Would this oracle integration work with current published elastic agent images?

[agithomas]

libaio is needed during the runtime.

The docker fleet agent container, created by the CI/CD should have the layers ( fleet-agent base image + libaio + oracle client files) to do the system testing.

In a production environment, If a customer needs to fetch Oracle metrics, the customer need to first install libaio and then install oracle client . The oracle client library can be installed by copying files or doing apt-get or yum install.

If ap-get or yum are chosen as the option for installation of oracle client, libaio is a docker build dependency. If copying the files is chosen as the mode of installation, libaio becomes runtime dependency

[agithomas]

Oracle integration works from 8.4 version. But, the user must install libaio + oracle client library . This is mentioned in the oracle integration dependency section.

When a system testing is performed by the CI/CD pipeline, the docker fleet-agent must need to have libaio + oracle client libary to fetch the oracle metrics.

[jsoriano]

Hey @agithomas,

I guess that for this case you need both things:

• A custom elastic-agent deployment, extending the elastic-agent image with the missing libraries. This may be supported with https://github.com/elastic/elastic-package/blob/ba7e6801f6944617e5ca33ebd60402385bf079fc/docs/howto/system_testing.md#agent-service-deployer
• A Oracle database running. I guess this can be done with the docker compose service deployer, using a image like the one used to test the Metricbeat module: https://github.com/elastic/elastic-package/blob/ba7e6801f6944617e5ca33ebd60402385bf079fc/docs/howto/system_testing.md#docker-compose-service-deployer

It'd be nice if you could give a try to this.

There are two possible limitations you may find:

• We have never customized the build steps of the custom elastic-agent deployment, this would be done with something like a custom-agent.yml file with this, and providing a Dockerfile. This may not work.

  version: '2.3'
  services:
  docker-custom-agent:
    image: custom-agent-oracle
    build: .

• I think elastic-package doesn't support the use multiple service deployers. You would need to use two here.

Give it a try, if you can confirm these limitations we can create issues in elastic-package to address them.

[agithomas]

@jsoriano thanks for the pointers. I will check and confirm.

@jsoriano , @ruflin ,

libaio + oracle client library is pre-requisite for oracle integration to work. Please refer integration instruction

If the user choose to use docker fleet agent to collect the oracle metrics, integration not work because the above mentioned dependencies are met.

Can there be a usecase such as this in any environment (, including elastic cloud / customer managed k8s , etc), where the user choose docker-fleet agent to collect metrics ?

If yes,, shouldn't libaio and oracle-client library be added to fleet-agent base image?

[ruflin]

If yes,, shouldn't libaio and oracle-client library be added to fleet-agent base image?

This is one option but it became a slippery slope. If other integrations also require a lib, do we add all of them to the container? @cmacknz It is a more general discussion we should have.

If we don't include the lib, what would be our recommended way to add it to container? Mount volume? Composed from a second container which contains the lib? @chrsmark and team are doing something similar at the moment around getting input templates into the standalone agent.

[jsoriano]

If yes,, shouldn't libaio and oracle-client library be added to fleet-agent base image?

This is one option but it became a slippery slope. If other integrations also require a lib, do we add all of them to the container?

So far we have been distributing Metricbeat without the libraries needed for the Oracle module, and I haven't ever seen any request for having these libraries in the docker image. Is there any request now? Is it frequent to deploy Oracle on dockerized environments?

Also take into account that we may not even be able to distribute the oracle-client library, depending on the EULA or license it has.

On non-dockerized environments the user can install the libraries on their system, as it is documented and I guess they are doing now.

Maybe in the future we can manage this from packages, with some kind of "pre-install" instructions to install these dependencies wherever the agent runs. But this would need to be defined with care, and we may enter in licensing territories there.

[cmacknz]

Maybe in the future we can manage this from packages, with some kind of "pre-install" instructions to install these dependencies wherever the agent runs. But this would need to be defined with care, and we may enter in licensing territories there.

Isn't this as simple as building a new Docker image that includes the Oracle dependencies using the existing agent image as a base? Could we simply document that and publish the Dockerfile used to build it? Especially if there are licensing constraints with the Oracle client this seems like a reasonable approach.

The other option would be distributing an image with every dependency of every integration, which we have some precedent for with the elastic-agent-complete that includes synthetics dependencies. This image would eventually become quite large over time as we add dependencies to it, but the only other option is more integration specific images.

[agithomas]

Updating the thread with the various tests and test outcome.

The above mentioned requirement can be met using custom-agent deployer

version: '2.3'
services:
  docker-custom-agent:
    image: docker.elastic.co/employees/agithomas/elastic-agent-oracle-client:0.1
    user: root
  oracle:
    image: docker.elastic.co/observability-ci/database-enterprise:12.2.0.1
    ports:
      - 1521:1521
      - 5500:5500

as mentioned in the link

Dockerfile

FROM docker.elastic.co/elastic-agent/elastic-agent-complete:8.4.0
USER root
RUN apt-get update && apt-get -y install \
    libaio1 \
    wget \
    unzip
WORKDIR /opt/oracle
RUN wget https://download.oracle.com/otn_software/linux/instantclient/214000/instantclient-basic-linux.x64-21.4.0.0.0dbru.zip && unzip -o instantclient-basic-linux.x64-21.4.0.0.0dbru.zip
RUN wget https://download.oracle.com/otn_software/linux/instantclient/217000/instantclient-sqlplus-linux.x64-21.7.0.0.0dbru.zip && unzip -o instantclient-sqlplus-linux.x64-21.7.0.0.0dbru.zip
RUN echo /opt/oracle/instantclient_21_4 > /etc/ld.so.conf.d/oracle-instantclient.conf && ldconfig
RUN export LD_LIBRARY_PATH=/opt/oracle/instantclient_21_4:$LD_LIBRARY_PATH  && export PATH=/opt/oracle/instantclient_21_7:$PATH
ENV LD_LIBRARY_PATH "${LD_LIBRARY_PATH}:/opt/oracle/instantclient_21_4"
ENV PATH "${PATH}:/opt/oracle/instantclient_21_7"

[agithomas]

To make this work, the issue must be resolved.

[jsoriano]

image: docker.elastic.co/employees/agithomas/elastic-agent-oracle-client:0.1

@agithomas regarding the custom image, did you try to include in this docker compose file the build instructtions? This would be preferred so the dockerfile is included in the package.

It would be something like this:

services:
  docker-custom-agent:
    build:
      context: .
    user: root

To make this work, the issue must be resolved.

Would you like to open a PR to discuss about this change?

[agithomas]

@jsoriano , i get the below error when i do elastic-package lint

item [Dockerfile] is not allowed in folder [/home/agikthomas/go/src/github.com/elastic/integrations/packages/oracle/data_stream/performance/_dev/deploy/agent]

[agithomas]

To avoid the above mentioned error, i went with the image: docker.elastic.co/employees/agithomas/elastic-agent-oracle-client:0.1

[agithomas]

@jsoriano , i believe the spec file has the validation rule.

Can this be modified to additionalContents: true ??

[ChrsMark]

If yes,, shouldn't libaio and oracle-client library be added to fleet-agent base image?

This is one option but it became a slippery slope. If other integrations also require a lib, do we add all of them to the container? @cmacknz It is a more general discussion we should have.

If we don't include the lib, what would be our recommended way to add it to container? Mount volume? Composed from a second container which contains the lib? @ChrsMark and team are doing something similar at the moment around getting input templates into the standalone agent.

In general you can mount extra files ad-hoc using a shared volume either in k8s or docker-compose.

In k8s we enable an init container to download the templates on startup and then the main Agent container of the Pod will find the downloaded files available. This help us avoid having the extra files being baked into the base Agent image.

Happy to provide more info/consulting here if needed.

cc: @gizas

[jsoriano]

@jsoriano , i believe the spec file has the validation rule.

Can this be modified to additionalContents: true ??

@agithomas Please open a PR with this change and we can continue the discussion there. I would like to try to make this less permissive and we would need some test files. We could include a test case to allow also the customization of the build step.

[agithomas]

@ChrsMark Thanks for sharing the information.

These files are needed only when Oracle integration is enabled.

Will the current pod(s) gets deleted and new template (with oracle requirements) will be downloaded when Oracle Integration is enabled?

If yes, shouldn't some changes be made in the template to support Oracle?

[agithomas]

Can this discussion be closed considering that no action is needed to support k8s or agent running as a container ?

All the other points, the solutions are identified and it is currently being tested

[jsoriano]

Ok, closing this, lets continue on the other open issues and PR. Thanks @agithomas!