Closed JSchulte01 closed 2 years ago
Version Details
Kowl CI fa13b2d (built March 23, 2022)
Hey @JSchulte01 , this is unfortunately not so easy debug, could you provide some proto files + Kowl config so that I can reproduce the issue? Then I can debug this locally.
Thanks, Martin
I've not been able to create a reproducible example in docker locally - this example is hosted in a kubernetes cluster - that said I can reliably reproduce the error in k8s with the following configuration:
Deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: kafka-dashboard
labels:
name: kafka-dashboard
app: kafka-dashboard
spec:
selector:
matchLabels:
name: kafka-dashboard
revisionHistoryLimit: 1
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
name: kafka-dashboard
app: kafka-dashboard
spec:
terminationGracePeriodSeconds: 30
volumes:
- name: config
configMap:
name: kafka-dashboard-cfg
- name: schemas
configMap:
name: protobuf-schemas-cfg
containers:
- name: kowl-kafka-dashboard
image: 'quay.io/cloudhut/kowl:master'
imagePullPolicy: Always
ports:
- name: http
containerPort: 8080
protocol: TCP
volumeMounts:
- name: config
mountPath: /mnt/config
- name: schemas
mountPath: /mnt/schemas
env:
- name: CONFIG_FILEPATH
value: /mnt/config/kowl-config.yaml
ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: protobuf-schemas-cfg
data:
sparkplug.proto: |
syntax = "proto3";
package maxedge;
message Payload {
uint64 timestamp = 1;
string test = 2;
}
kowl.yaml
protobuf:
enabled: true
mappings:
- topicName: test
valueProtoType: maxedge.Payload
# # Map the proto type names for each of your topics. These proto types will be used for deserialization
# # - topicName: xy
# # valueProtoType: fake_model.Order # You can specify the proto type for the record key and/or value (just one will work too)
# # keyProtoType: package.Type
# # SchemaRegistry does not require any mappings to be specified. The schema registry client configured on the kafka level will be reused.
# schemaRegistry:
# enabled: true
# refreshInterval: 1m
# # FileSystem can be configured if you want Kowl to search the local file system for the .proto files
fileSystem:
enabled: true
paths:
- /mnt/schemas
refreshInterval: 60m
It would seem that this is related to K8s versioning of configMap mounts using symlinks
There is a solution - though not ideal which involves mapping each file individually: https://stackoverflow.com/questions/50685385/kubernetes-config-map-symlinks-data-is-there-a-way-to-avoid-them
Thanks for the follow up and detailed information - highly appreciated!
Hey, @weeco! The proto-decoding feature is great! The issue is still there. The files were imported twice in Kubernetes, and it was painful to map config map with many photos.
The problem and root cause are shown in the picture: The root cause is scanning hidden folders.
Steps to reproduce on any Linux environment:
/tmp/protos/..data
/tmp/protos/..data/tbmsg.proto
/tmp/protos/tbmsg.proto
to /tmp/protos/..data/tbmsg.proto
kafka:
protobuf:
enabled: true
fileSystem:
enabled: true
paths:
- /tmp/protos
The fix is to avoid scanning hidden files and folders (same as ls
command do).
Otherwise, the Kubernetes deployment became painful. Some users might give up using the proto-decoding feature because of such an issue.
I am preparing the Redpanda console deployment for the ThingsBoard IoT framework and I don't really want to submit a non-perfect deployment for a wide community. Unfortunately, I can not help you with a good PR because it is not my language at this moment. Let me know if you need to create a new issue. Will be appreciated if you can implement the fix required for Kubernetes users 🙏 .
@pon0marev FYI
Hey @smatvienko-tb , thanks for your detailed comment. This helps me understanding the issue.
I think we could add a config option that skips hidden dirs (requires an OS specific implementation). Would that work for you? Possibly I'll make that an opt-in to not break existing deployments.
Hey @weeco, Thank you for your quick response!
For me, any solution will be just fine!
But I do not believe someone intentionally injected a hidden file or nested directory as a proto. That has no clear purpose.
The hidden files are implemented in any OS. And I believe that will be not too much headache to implement this. Thank you!
Kowl is unable to read protobufs from the file system appropriately - It appears to find 2 protobuf files, despite there being only 1 in the directory. I have confirmed there is exactly one protobuf file available and yet it still tries to read two and raises an error about duplicate symbols.
Logs
Config