Azure / azure-sdk-for-rust

This repository is for active development of the *unofficial* Azure SDK for Rust. This repository is *not* supported by the Azure SDK team.
MIT License
680 stars 232 forks source link

pub_blob_block hangs when running under actix server #1639

Closed cswinter closed 2 months ago

cswinter commented 2 months ago

I'm running into a very strange issue where put_blob_block invocations seem to hang indefinitely when running inside an actix request. In the main function of the same process, an identical put_blob_block invocation work fine. Still working on narrowing this down, my most minimal repro is currently https://github.com/cswinter/LocustDB/commit/bf52508377b9c727e8bd0e0d0d7dba24885933e5 (I run the db_bench binary, note the storeblob function which is called immediately in the #[tokio::main] async main function. calling an identical storeblob function in server/mod.rs when serving requests inside actix runtime spawned on different thread then blocks):

(base) clemens@LAPTOP-NV53OUEJ:~/src/LocustDB$ RUST_BACKTRACE=1 RUST_LOG=info,locustdb::disk_store::azure_writer=debug cargo run --bin db_bench -- --load-factor=4 --large-only --db-path=az://locustdbstoragetesting/dev/xtest01
   Compiling locustdb v0.4.7 (/home/clemens/src/LocustDB)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 24.01s
     Running `target/debug/db_bench --load-factor=4 --large-only '--db-path=az://locustdbstoragetesting/dev/xtest01'`
obtained default creds
obtained storage creds
created blob client BlobClient { container_client: ContainerClient { service_client: BlobServiceClient { pipeline: Pipeline { pipeline: [TelemetryPolicy { header: "azsdk-rust-storage/0.19.0 (1.78.0-nightly; linux; x86_64)" }, CustomHeadersPolicy, ExponentialRetryPolicy { initial_delay: 200ms, max_retries: 8, max_elapsed: 60s, max_delay: 30s }, TimeoutPolicy { default_timeout: None }, AuthorizationPolicy { credentials: StorageCredentials { credential: "TokenCredential" } }, TransportPolicy { transport_options: TransportOptions { inner: Http { http_client: Client { accepts: Accepts, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*"} } } } }] }, cloud_location: Public { account: "locustdbstoragetesting" }, credentials: StorageCredentials { credential: "TokenCredential" } }, container_name: "dev" }, blob_name: "xtest01/wal/0.wal" }
putting blob...
success! time: 2.216060083s
getting blob...
result: [1, 70, 70, 1]
 INFO 2024-04-13T23:50:32Z: db_bench: Store blob successful
 INFO 2024-04-13T23:50:32Z: db_bench: Creating LocustDB at "az://locustdbstoragetesting/dev/xtest01"
DEBUG 2024-04-13T23:50:32Z: locustdb::disk_store::azure_writer: Checking if blob exists in Azure: "xtest01/meta"
 INFO 2024-04-13T23:50:34Z: locustdb::disk_store::storage: Recovering from wal checkpoint 0
DEBUG 2024-04-13T23:50:34Z: locustdb::disk_store::azure_writer: Listing blobs from Azure: "xtest01/wal"
 INFO 2024-04-13T23:50:35Z: actix_server::builder: starting 16 workers
 INFO 2024-04-13T23:50:35Z: actix_server::server: Actix runtime found; starting in Actix runtime
 INFO 2024-04-13T23:50:35Z: db_bench: Starting large table logging
 INFO 2024-04-13T23:50:35Z: locustdb::logging_client: Creating request data for 128 events
 INFO 2024-04-13T23:50:35Z: locustdb::logging_client: Sending data (37942 B)
 INFO 2024-04-13T23:50:35Z: locustdb::server: LCREaTING BLOB IN INSERT BIN
obtained default creds
obtained storage creds
created blob client BlobClient { container_client: ContainerClient { service_client: BlobServiceClient { pipeline: Pipeline { pipeline: [TelemetryPolicy { header: "azsdk-rust-storage/0.19.0 (1.78.0-nightly; linux; x86_64)" }, CustomHeadersPolicy, ExponentialRetryPolicy { initial_delay: 200ms, max_retries: 8, max_elapsed: 60s, max_delay: 30s }, TimeoutPolicy { default_timeout: None }, AuthorizationPolicy { credentials: StorageCredentials { credential: "TokenCredential" } }, TransportPolicy { transport_options: TransportOptions { inner: Http { http_client: Client { accepts: Accepts, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*"} } } } }] }, cloud_location: Public { account: "locustdbstoragetesting" }, credentials: StorageCredentials { credential: "TokenCredential" } }, container_name: "dev" }, blob_name: "xtest01/wal/0.wal" }
putting blob...
cswinter commented 2 months ago

ok i think it's related to use of block_on, presumably the actix runtime doesn't have enough spare threads