Integrate with IKS

[KeranYang]

Problem Statement

Currently, Perfman is built for running on a local cluster. We should enable it to run on any production environment.

Acceptance Criteria

Using Intuit developer system as an example, build the application in a way that it can run on IKS as a devportal asset.

Considered Solutions

In general we have two approaches:

1. Using the existing Prometheus service in the addon-xxx-ns namespace:

• Deploying a service that does not require cluster level permissions, and can scrape metrics, i.e. does not require creation of CRDs, cluster role bindings, etc., e.g. Telegraf
• Portforward pods to port 2469 and scrape directly from the metrics endpoint
• Deploy perfman in a system namespace, or a managed namespace that is whitelisted through a network policy so that it can communicate with the existing Prometheus operator on the cluster
• Create own network policy to enable egress, or remove configured network policy that is preventing access to the Prometheus service
• Deploy perfman as a pod in desired namespace, and use PVCs to mount volumes

2. Creating a new Prometheus service in our namespace as we do locally

[ayildirim21]

Once we figure out how to integrate with IKS/connect to the prometheus service running there, the updated Dockerfile should look like this:

####################################################################################################
# base
####################################################################################################
FROM --platform=${BUILDPLATFORM:-linux/amd64} golang:1.22.4-alpine AS builder

RUN apk update && apk upgrade && \
   apk add --no-cache tzdata curl unzip

ARG TARGETOS=linux
ARG TARGETARCH=amd64
ARG COMMIT_SHA
ENV KUBECONFIG=""
ENV USER=perfmanuser
ENV UID=10001

# Create non-privileged user as processes running
# on containers run with root by default
RUN adduser \
   --disabled-password \
   --home "/home/${USER}" \
   --uid "${UID}" \
   "${USER}"

WORKDIR /app

# Cache deps before building and copying source
COPY go.mod .
COPY go.sum .
RUN go mod download
RUN go mod verify

COPY . .

RUN CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} go build  \
   -ldflags "-s -w -X github.com/numaproj-labs/numaflow-perfman/util.CommitSHA=$COMMIT_SHA"  \
   -v -o dist/perfman main.go

# https://github.intuit.com/EIAM/eiamCli-golang#for-linux-users
RUN curl -L -o eiamcli.zip  \
   https://artifactory.a.intuit.com/nexus/content/repositories/IBP.Intuit-Releases/com/intuit/ebs/eiam/eiamCli-linux/3.0.6/eiamCli-linux-3.0.6.zip
RUN unzip eiamcli.zip

RUN chmod 750 /app/eiamCli-linux/install.sh

####################################################################################################
# perfman
####################################################################################################
FROM --platform=${BUILDPLATFORM:-linux/amd64} alpine:3.15 AS perfman

RUN apk update && apk upgrade

RUN apk --no-cache add ca-certificates wget && \
   wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub && \
   wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.28-r0/glibc-2.28-r0.apk && \
   apk add glibc-2.28-r0.apk

COPY --from=builder /usr/share/zoneinfo /usr/share/zoneinfo
COPY --from=builder /etc/passwd /etc/passwd
COPY --from=builder /etc/group /etc/group
COPY --from=builder /app/config /home/perfman/config
COPY --from=builder /app/dist/perfman /bin/perfman
COPY --from=builder /app/eiamCli-linux /eiamCli-linux

WORKDIR /eiamCli-linux
RUN ./install.sh

#USER perfmanuser:perfmanuser

WORKDIR /home/perfman

ENTRYPOINT ["/bin/ash"]

And add the following to the Makefile:

# You should have your KUBECONFIG environment variable already exported locally
# Run eiamCli login once in the shell
.PHONY: run-iks
run-iks:
    mkdir -p output
    docker run -it --network host \
    -e KUBECONFIG=${KUBECONFIG} \
    -v ${KUBECONFIG}:${KUBECONFIG} \
    -v ./output:/home/perfman/output ${IMAGE_REGISTRY}