Open networkop opened 3 years ago
Hi @networkop, apologies for the delay. This is great! Kafka and Clickhouse are big deployments and require operators: it might be interesting to have a Deployment/Helm for GoFlow. Do you have a repository with the files?
so far I have a bunch of interactive steps. with a bit of effort they can be turned into some flux manifests to deploy all this stuff in one go. Here's how it looks like
Install kafka
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install sflow bitnami/kafka \
--set persistence.enabled=false \
--set zookeeper.persistence.enabled=false
# The resulting kafka API endpoint will be sflow-kafka.sflow.svc.cluster.local:9092
Install clickhouse
curl -LO https://raw.githubusercontent.com/netsampler/goflow2/main/compose/kcg/clickhouse/create.sh
curl -LO https://raw.githubusercontent.com/netsampler/goflow2/main/compose/kcg/clickhouse/protocols.csv
curl -LO https://raw.githubusercontent.com/netsampler/goflow2/main/pb/flow.proto
sed -i 's/kafka:9092/sflow-kafka.sflow.svc.cluster.local:9092/' create.sh
# this is needed because configmaps are mounted as read-only and clickhouse expects
# to chown files in these directories and will crash if it can't do it
sed -i '1 a cp /tmp/flow.proto /var/lib/clickhouse/format_schemas/flow.proto' create.sh
sed -i '1 a cp /tmp/protocols.csv /var/lib/clickhouse/user_files/protocols.csv' create.sh
kubectl -n sflow create cm initdb --from-file=create.sh
kubectl -n sflow create cm proto --from-file=flow.proto
kubectl -n sflow create cm csv --from-file=protocols.csv
helm repo add slamdev https://slamdev.github.io/helm-charts/
helm install sflow-db slamdev/altinity-clickhouse-operator --version 0.0.5
cat << EOF > db.yml
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
name: "sflow-db"
spec:
configuration:
users:
grafana/password: grafana
grafana/profile: default
grafana/quota: default
grafana/networks/ip: "::/0"
clusters:
- name: "clickhouse"
templates:
podTemplate: db-with-goflow
serviceTemplates: chi-service-template
layout:
shardsCount: 1
replicasCount: 1
templates:
serviceTemplates:
- name: chi-service-template
spec:
type: ClusterIP
podTemplates:
- name: db-with-goflow
spec:
containers:
- name: clickhouse
image: yandex/clickhouse-server:21.5.6.6
ports:
- name: http
containerPort: 8123
volumeMounts:
- name: volume-initdb
mountPath: /docker-entrypoint-initdb.d/
- name: volume-proto
mountPath: /tmp/flow.proto
subPath: flow.proto
readOnly: true
- name: volume-csv
mountPath: /tmp/protocols.csv
subPath: protocols.csv
readOnly: false
volumes:
- name: volume-initdb
configMap:
name: initdb
- name: volume-proto
configMap:
name: proto
- name: volume-csv
configMap:
name: csv
EOF
Install goflow2
cat << EOF > deploy.yml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: goflow
name: goflow
namespace: sflow
spec:
selector:
matchLabels:
app: goflow
template:
metadata:
labels:
app: goflow
spec:
containers:
- image: netsampler/goflow2:latest
name: goflow
args:
- -transport.kafka.brokers=sflow-kafka.sflow.svc.cluster.local:9092
- -transport=kafka
- -transport.kafka.topic=flows
- -format=pb
- -format.protobuf.fixedlen=true
EOF
k expose -n sflow deploy goflow --type NodePort --port 6343 --protocol UDP
k -n sflow get svc goflow
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
goflow NodePort 10.107.183.222 <none> 6343:30323/UDP 30s
Install Grafana
curl -LO https://raw.githubusercontent.com/netsampler/goflow2/main/compose/kcg/grafana/dashboards/viz-ch.json
curl -LO https://raw.githubusercontent.com/netsampler/goflow2/main/compose/kcg/grafana/datasources-ch.yml
curl -LO https://raw.githubusercontent.com/netsampler/goflow2/main/compose/kcg/grafana/dashboards.yml
sed -i 's/db:8123/clickhouse-sflow-db:8123/' datasources-ch.yml
sed -i "s/user: ''/user: 'grafana'/" datasources-ch.yml
sed -i "s/password: ''/password: 'grafana'/" datasources-ch.yml
kubectl -n sflow create cm grafana-ch --from-file=viz-ch.json
kubectl -n sflow create cm grafana-source --from-file=datasources-ch.yml
kubectl -n sflow create cm grafana-dashboards --from-file=dashboards.yml
kubectl label cm grafana-source grafana_datasource=include
kubectl label cm grafana-ch grafana_dashboard=include
helm repo add grafana https://grafana.github.io/helm-charts
helm install grafana grafana/grafana \
--set sidecar.datasources.enabled=true \
--set sidecar.datasources.labelValue=include \
--set sidecar.dashboards.enabled=true \
--set sidecar.dashboards.labelValue=include \
--set service.type=NodePort \
--set adminPassword=grafana \
--set plugins=vertamedia-clickhouse-datasource \
--set env.GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=vertamedia-clickhouse-datasource
k get svc grafana
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
grafana NodePort 10.102.189.108 <none> 80:31174/TCP 37s
Hey @lspgn 👋 I've done a POC recently of one of your example pipelines (Flows + Kafka + Clicklhouse + Grafana) deployed on top of k8s. Would you be interested in that? I still have all of the commands and files available.