search

redpanda-data / console

Redpanda Console is a developer-friendly UI for managing your Kafka/Redpanda workloads. Console gives you a simple, interactive approach for gaining visibility into your topics, masking data, managing consumer groups, and exploring real-time data with time-travel debugging.
https://redpanda.com
3.79k stars 347 forks source link

Support regex in topic protobuf mappings #405

Closed upils closed 4 months ago

upils commented 2 years ago

For the protobuf topic mapping, we currently need to set the exact topic name.

We have a use case where our topic names are generated dynamically so we cannot give the full list in the configuration. But they all contains the same kind of message, so we would like to deserialize them all the same way.

We would like a way to give a regex in the topicName so all of our topics would match.

What do you think about this feature?

The current implementation is filling a map with the topicName as the key. https://github.com/redpanda-data/console/blob/222b498ec0295acf0ea33888f20aa3a0440757f6/backend/pkg/proto/service.go#L96

Then this map is used to check if a mapping exists for a given topic: https://github.com/redpanda-data/console/blob/222b498ec0295acf0ea33888f20aa3a0440757f6/backend/pkg/proto/service.go#L238

We could consider the topicName as a regex, try to compile it when the config is loaded (and return an error if this is invalid) and keep using the given string as the key in the map.

Then after testing if the topic given in getMessageDescriptor() exists in the map, we would loop on the map to check if the topic match any regex. This would be a bit slower than the current implementation.

weeco commented 2 years ago

Hey @upils , thanks for sharing your usecase. I think what you are suggesting makes sense to me. Do you want to work on this and submit a PR for this?

We should make sure that imports / nested types are supported. - regardless whether we use regexed strings or not. If I recall correctly we use protoreflect's proto registry to register all proto types in advance, but if your topics share the some proto schemas (with the same name) this should not be a problem I think.

upils commented 2 years ago

Thanks for the reply.

Yes I might give it a try.

I am not sure to understand what you mean about the proto registry. It might be clearer if I dive in the code.

weeco commented 2 years ago

I was referring to this part https://github.com/redpanda-data/console/blob/222b498ec0295acf0ea33888f20aa3a0440757f6/backend/pkg/proto/service.go#L370-L374 , but I think it's not relevant to your problem, because you are sharing the same protos across several topics and you only want to tell Console for what topics it should use the proto schemas.

upils commented 2 years ago

OK. Yes, I understand it should not be an issue.

smatvienko-tb commented 1 year ago

Hey, guys! I need some regex in topic protobuf mappings!

I have a lot of protos for ThingsBoard and many topics created in flight with the service_id suffix.

A piece of my config looks weirdly long and does not cover deployments for any scale. And makes some pain in my eyes 😭.

I would be happy if you could put the final effort into finishing such a great feature! image

Here is an example of a small part of ThingsBoard's mappings :

kafka:
  protobuf:
    enabled: true
    mappings:
      ####################     CORE     ######################
      - topicName: tb_core.0
        valueProtoType: transport.ToCoreMsg
      - topicName: tb_core.1
        valueProtoType: transport.ToCoreMsg
      - topicName: tb_core.2
        valueProtoType: transport.ToCoreMsg
      - topicName: tb_core.3
        valueProtoType: transport.ToCoreMsg
      - topicName: tb_core.4
        valueProtoType: transport.ToCoreMsg
      - topicName: tb_core.5
        valueProtoType: transport.ToCoreMsg
      - topicName: tb_core.6
        valueProtoType: transport.ToCoreMsg
      - topicName: tb_core.7
        valueProtoType: transport.ToCoreMsg
      - topicName: tb_core.8
        valueProtoType: transport.ToCoreMsg
      - topicName: tb_core.9
        valueProtoType: transport.ToCoreMsg
      - topicName: tb_core.10
        valueProtoType: transport.ToCoreMsg
      - topicName: tb_core.11
        valueProtoType: transport.ToCoreMsg
      ##########################################
      - topicName: tb_core.notifications.tb-core-0
        valueProtoType: transport.ToCoreNotificationMsg
      - topicName: tb_core.notifications.tb-core-1
        valueProtoType: transport.ToCoreNotificationMsg
      - topicName: tb_core.notifications.tb-core-2
        valueProtoType: transport.ToCoreNotificationMsg      
      - topicName: tb_core.notifications.tb-core-3
        valueProtoType: transport.ToCoreNotificationMsg
      - topicName: tb_core.notifications.tb-core-4
        valueProtoType: transport.ToCoreNotificationMsg
      - topicName: tb_core.notifications.tb-core-5
        valueProtoType: transport.ToCoreNotificationMsg
      - topicName: tb_core.notifications.tb-core-6
        valueProtoType: transport.ToCoreNotificationMsg
      - topicName: tb_core.notifications.tb-core-7
        valueProtoType: transport.ToCoreNotificationMsg
      - topicName: tb_core.notifications.tb-core-8
        valueProtoType: transport.ToCoreNotificationMsg
      - topicName: tb_core.notifications.tb-core-9
        valueProtoType: transport.ToCoreNotificationMsg
      - topicName: tb_core.notifications.tb-core-10
        valueProtoType: transport.ToCoreNotificationMsg
      - topicName: tb_core.notifications.tb-core-11
        valueProtoType: transport.ToCoreNotificationMsg
      #################### INTEGRATIONS ######################
      - topicName: tb_ie.api.requests
        valueProtoType: integration.IntegrationApiRequestMsg
      ##########################################
      - topicName: tb_ie.api.responses.tb-ie-main-0
        valueProtoType: integration.IntegrationApiResponseMsg
      - topicName: tb_ie.api.responses.tb-ie-main-1
        valueProtoType: integration.IntegrationApiResponseMsg
      - topicName: tb_ie.api.responses.tb-ie-main-2
        valueProtoType: integration.IntegrationApiResponseMsg
      - topicName: tb_ie.api.responses.tb-ie-other-0
        valueProtoType: integration.IntegrationApiResponseMsg
      - topicName: tb_ie.api.responses.tb-ie-other-1
        valueProtoType: integration.IntegrationApiResponseMsg
      - topicName: tb_ie.api.responses.tb-ie-other-2
        valueProtoType: integration.IntegrationApiResponseMsg
      ##########################################
brycefisher commented 8 months ago

I'd love to get this feature merged into master and shipped in a release. Are you still planning to work on this @upils or did you want another contributor to take over from here? No worries either way!

bojand commented 4 months ago

This should be addressed now via #1291.