infinyon / fluvio

Lean and mean distributed stream processing system written in rust and web assembly. Alternative to Kafka + Flink in one.
https://www.fluvio.io/
Apache License 2.0
3.88k stars 491 forks source link

[Bug]: Non alphanumeric characters in topic name hangs CLI #535

Closed nacardin closed 3 years ago

nacardin commented 3 years ago

Describe the setup Fluvio Cloud

$ fluvio version
Fluvio version : 0.6.0-alpha.5
Git Commit     : 4144c3aa9f74d1a7808e476e85c1369e7addd413
OS Details     : Linux 5.8.0-1013-aws x86_64 GNU/Linux
Rustc Version  : 1.48.0 (7eac88a 2020-11-16)

Steps to reproduce the behavior:

  1. Run the command 'fluvio topic create t2#r'

Log output

Dec 01 22:08:11.817 ERROR outer_loop{self=Topic K8StateDispatcher namespace="acct-a1ab43b5-f4e8-4147-a7fa-f16fe0a4771e"}:inner_loop:process_ws_action: fluvio_stream_dispatcher::dispatcher::k8_dispatcher: error: Topic, update spec client error: 422 Unprocessable Entity
Dec 01 22:08:11.817 DEBUG outer_loop{self=Topic K8StateDispatcher namespace="acct-a1ab43b5-f4e8-4147-a7fa-f16fe0a4771e"}:inner_loop: fluvio_stream_dispatcher::dispatcher::k8_dispatcher: dispatcher waiting
Dec 01 22:08:12.772 DEBUG fluvio_stream_dispatcher::store::context: Topic store, didn't receive wait,exiting,continue waiting
Dec 01 22:08:12.772 DEBUG fluvio_stream_dispatcher::store::context: Topic store, waiting for store event
Dec 01 22:08:13.773 DEBUG fluvio_stream_dispatcher::store::context: Topic store, didn't receive wait,exiting,continue waiting
Dec 01 22:08:13.773 DEBUG fluvio_stream_dispatcher::store::context: Topic store, waiting for store event
Dec 01 22:08:14.773 DEBUG fluvio_stream_dispatcher::store::context: Topic store, didn't receive wait,exiting,continue waiting
Dec 01 22:08:14.773 DEBUG fluvio_stream_dispatcher::store::context: Topic store, waiting for store event
Dec 01 22:08:15.140 DEBUG flv_tls_proxy::proxy: total 58 bytes copied from source to target: 192.168.49.1:43850->0.0.0.0:9005    
Dec 01 22:08:15.773 DEBUG fluvio_stream_dispatcher::store::context: Topic store, didn't receive wait,exiting,continue waiting
Dec 01 22:08:15.773 DEBUG fluvio_stream_dispatcher::store::context: Topic store, waiting for store event
Dec 01 22:08:16.773 DEBUG fluvio_stream_dispatcher::store::context: Topic store, didn't receive wait,exiting,continue waiting
Dec 01 22:08:16.773 DEBUG fluvio_stream_dispatcher::store::context: Topic store, waiting for store event
Dec 01 22:08:17.773 DEBUG fluvio_stream_dispatcher::store::context: Topic store, didn't receive wait,exiting,continue waiting
Dec 01 22:08:17.773 DEBUG fluvio_stream_dispatcher::store::context: Topic store, waiting for store event
Dec 01 22:08:18.773 DEBUG fluvio_stream_dispatcher::store::context: Topic store, didn't receive wait,exiting,continue waiting
Dec 01 22:08:18.773 DEBUG fluvio_stream_dispatcher::store::context: Topic store, waiting for store event
Dec 01 22:08:19.773 DEBUG fluvio_stream_dispatcher::store::context: Topic store, didn't receive wait,exiting,continue waiting
Dec 01 22:08:19.773 DEBUG fluvio_stream_dispatcher::store::context: Topic store, waiting for store event
Dec 01 22:08:20.774 DEBUG fluvio_stream_dispatcher::store::context: Topic store, didn't receive wait,exiting,continue waiting
Dec 01 22:08:20.774 DEBUG fluvio_stream_dispatcher::store::context: Topic store, waiting for store event
Dec 01 22:08:21.774 DEBUG fluvio_stream_dispatcher::store::context: Topic store, didn't receive wait,exiting,continue waiting
Dec 01 22:08:21.774 DEBUG fluvio_stream_dispatcher::store::context: Topic store, waiting for store event
Dec 01 22:08:21.774 DEBUG fluvio_sc::services::public_api::public_server: waiting for next api request: PublicAPI
sehz commented 3 years ago

Fluvio topic name is limited to character set supported by Kubernetes Id. We should reject any topic name that doesn't conform to it.

nicholastmosher commented 3 years ago

In that case we should probably create a specialized type such as struct KubeId(String) with the invariant that it always is a valid Kubernetes Id character set. We can make it so every possible way to construct it will reject any string with characters outside of the valid charset, and we can give it the same serialization rules as string.

nicholastmosher commented 3 years ago

For a short-term fix, I'm going to make the CLI reject any non-alphanumeric topic names

nicholastmosher commented 3 years ago

We should make a followup issue that allows topic names to be any utf-8 name. My guess for why this doesn't work today is that the topic names may be used as file names somewhere, and not all utf-8 is a valid filename. If that's the case, we may be able to get around this by base64 encoding topic names before they are used as filenames