tikv / grpc-rs

The gRPC library for Rust built on C Core library and futures
Apache License 2.0
1.81k stars 253 forks source link

Two rpc servers configure the same address but one successfully start, the other failed #598

Closed JasonThon closed 2 years ago

JasonThon commented 2 years ago

Problem Description

My project has two different services but with the same grpc-rs dependency. However, one can start normally and the other failed; The exception message is simple: [1] 11810 segmentation fault cargo run --manifest-path src/worker/Cargo.toml

When I debug deeply, I found a strange phenomenon:

This screenshot is for the normally started service, you can see on the top left, the parsed_addr variable is equal to addr, seemingly.

image

This screenshot is for the failed service, you can see on the top left, the parsed_addr variable is an unexpected value.

image

Code Difference

Normally start service grpc binding code

    let mut grpc_server = grpcio::ServerBuilder::new(sync::Arc::new(grpcio::Environment::new(10)))
        .register_service(service)
        .bind("0.0.0.0", config.port as u16)
        .build()
        .expect("grpc server create failed");
    grpc_server.start();

Failed start service grpc binding code

    let grpc_server = grpcio::ServerBuilder::new(sync::Arc::new(grpcio::Environment::new(10)))
        .register_service(service)
        .bind("0.0.0.0", config.port as u16)
        .build();

    if grpc_server.is_err() {
        panic!("{:?}", grpc_server.unwrap_err())
    }

    let mut unwrap_server = grpc_server.expect("grpc server start failed");
    unwrap_server.start();

You can see their binding configurations are the same;

Dependencies

The two projects dependencies are the followings

Normally start service dependencies

[dependencies]
tokio = { version = "1", features = ["rt-multi-thread", "sync", "macros", "signal"] }
serde = { version = "1.0", features = ["derive"] }
log = "0.4"
common = { path = "../common" }
proto = { path = "../proto", features = ["coordinator", "worker"] }
serde_json = "1.0.59"

grpcio = "0.10.3"
protobuf = "2.27.1"
rocksdb = { version = "0.19.0", features=["lz4"] }

Failed start service dependencies

[dependencies]
common = { path = "../common" }
stream = { path = "../stream" }
proto = { path = "../proto", features = ["worker"] }
protobuf = "2.27.1"
grpcio = "0.10.3"
serde = { version = "1.0", features = ["derive"] }
tokio = { version = "1", features = ["rt-multi-thread", "sync", "signal", "macros"] }
log = "0.4"
serde_json = "1.0.59"
env_logger = "0.9.0"

The common module dependencies

[dependencies]
serde = { version = "1.0", features = ["derive"] }
tokio = { version = "1", features = ["sync"] }
regex = "1"
bytes = "1.1.0"
chrono = "0.4"
proto = { path = "../proto", features = ["proto-common", "worker"] }
serde_json = "1.0.59"
grpcio = "0.10.3"
log = "0.4"
protobuf = "2.27.1"
rdkafka = "0.29.0"
futures-executor = "0.3"

[dev-dependencies]
common = { path = "../common" }
tokio = { version = "1", features = ["test-util"] }

The proto module dependencies

[dependencies]
grpcio = "0.10.3"
protobuf = "2.27.1"

The stream module dependencies

[dependencies]
common = { path = "../common" }
chrono = "0.4"
tokio = { version = "1", features = ["rt"] }
proto = { path = "../proto", features = ["worker"] }
log = "0.4"
serde = { version = "1.0", features = ["derive"] }
crossbeam-channel = "0.5"
v8 = "0.54.0"
protobuf = "2.27.1"

[dev-dependencies]
tokio = { version = "1", features = ["test-util"] }
stream = { path = "../stream" }

Expected behavior

I think for the same binding configuration, the parsed_addr should be expected to be equal to addr.

System information

Another context

I use Visual Studio Code for debug, with latest CodeLLDB and rust-analysor extensions

BusyJay commented 2 years ago

I tried your failed service and didn't reproduced. I have a few questions:

Is the bug reproduced constantly? If the port is changed to the same as the successful one, does the bug still occur? Are these two services running in one process?

BusyJay commented 2 years ago

Note, 0.12.0 is released, you can also give it a try.

JasonThon commented 2 years ago

I tried your failed service and didn't reproduced. I have a few questions:

Is the bug reproduced constantly? If the port is changed to the same as the successful one, does the bug still occur? Are these two services running in one process?

  1. This bug always happens;
  2. I tried to set a same port, but it still fails to start;
  3. They are running in two isolated processes;
  4. I think it's somehow related to my computer's environment, let me check it;
BusyJay commented 2 years ago

The error should occur when the string_view is converted to string in URI::PercentDecode. It somehow starts from an invalid offset and with an invalid length.

JasonThon commented 2 years ago

The error should occur when the string_view is converted to string in URI::PercentDecode. It somehow starts from an invalid offset and with an invalid length.

There two things I think you want to know:

  1. On my Intel Chip Mac, it's always wrong;
  2. But on M1 Max Chip Mac, everything is fine;

So seems like this bug is somewhat related to different Mac's environment settings or chip?

JasonThon commented 2 years ago

Looks like it's caused by different hardware envs. I will close this issue for now. thank you so much~