search

kamu-data / kamu-cli

Next-generation decentralized data lakehouse and a multi-party stream processing network
https://kamu.dev
Other
293 stars 12 forks source link

E2E: Flaky test errors for MySQL implementation #718

Open s373r opened 1 month ago

s373r commented 1 month ago

Describe the bug

We have periodic errors when working with E2E tests for MySQL. On CI it is enough (at the moment) to restart the test when errors occur, but in local development it is sometimes not enough.

The purpose of this ticket is to investigate the causes of deadlocks and how to fix them.

Steps To Reproduce

make test-e2e

Current Behavior

Sometimes, our team members get an error even on re-runs

Expected Behavior

No issues

kamu system info

latest

Logs

left: Err(InternalError { source: CommandError { cmd: ["kamu", "-q", "--system-time", "2024-07-17T00:00:00.000Z", "--e2e-output-data-path", "/tmp/.tmpamARRx/e2e-output-data.txt", "system", "api-server", "--address", "127.0.0.1"], error: CriticalFailure { source: InternalError { source: Internal(InternalError { source: Database(MySqlDatabaseError { code: Some("40001"), number: 1213, message: "Deadlock found when trying to get lock; try restarting transaction" }), backtrace: Some(Backtrace [{ fn: "internal_error::InternalError::new", file: "/home/sergeiz/kamu/kamu-cli/src/utils/internal-error/src/lib.rs", line: 38 }, { fn: "<E as internal_error::ErrorIntoInternal>::int_err", file: "/home/sergeiz/kamu/kamu-cli/src/utils/internal-error/src/lib.rs", line: 76 }, { fn: "<core::result::Result<OK,E> as internal_error::ResultIntoInternal<OK>>::int_err", file: "/home/sergeiz/kamu/kamu-cli/src/utils/internal-error/src/lib.rs", line: 91 }, { fn: "<kamu_accounts_mysql::repos::mysql_account_repository::MySqlAccountRepository as kamu_accounts::repos::account_repository::AccountRepository>::find_account_id_by_provider_identity_key::{{closure}}", file: "/home/sergeiz/kamu/kamu-cli/src/infra/accounts/mysql/src/repos/mysql_account_repository.rs", line: 262 }, { fn: "<core::pin::Pin<P> as core::future::future::Future>::poll", file: "/rustc/1a648b397dedc98ada3dd3360f6d661ec2436c56/library/core/src/future/future.rs", line: 124 }, { fn: "<kamu_accounts_services::authentication_service_impl::AuthenticationServiceImpl as kamu_accounts::services::authentication_service::AuthenticationService>::login::{{closure}}", file: "/home/sergeiz/kamu/kamu-cli/src/domain/accounts/services/src/authentication_service_impl.rs", line: 196 }, { fn: "<core::pin::Pin<P> as core::future::future::Future>::poll", file: "/rustc/1a648b397dedc98ada3dd3360f6d661ec2436c56/library/core/src/future/future.rs", line: 124 }, { fn: "kamu_cli::commands::system_api_server_run_command::APIServerRunCommand::get_access_token::{{closure}}::{{closure}}::{{closure}}", file: "/home/sergeiz/kamu/kamu-cli/src/app/cli/src/commands/system_api_server_run_command.rs", line: 117 }, { fn: "database_common::transactions::db_transaction_manager::DatabaseTransactionRunner::transactional_with::{{closure}}::{{closure}}::{{closure}}", file: "/home/sergeiz/kamu/kamu-cli/src/utils/database-common/src/transactions/db_transaction_manager.rs", line: 109 }, { fn: "database_common::transactions::db_transaction_manager::DatabaseTransactionRunner::transactional::{{closure}}", file: "/home/sergeiz/kamu/kamu-cli/src/utils/database-common/src/transactions/db_transaction_manager.rs", line: 73 }, { fn: "database_common::transactions::db_transaction_manager::DatabaseTransactionRunner::transactional_with::{{closure}}", file: "/home/sergeiz/kamu/kamu-cli/src/utils/database-common/src/transactions/db_transaction_manager.rs", line: 111 }, { fn: "kamu_cli::commands::system_api_server_run_command::APIServerRunCommand::get_access_token::{{closure}}", file: "/home/sergeiz/kamu/kamu-cli/src/app/cli/src/commands/system_api_server_run_command.rs", line: 121 }, { fn: "<kamu_cli::commands::system_api_server_run_command::APIServerRunCommand as kamu_cli::commands::Command>::run::{{closure}}", file: "/home/sergeiz/kamu/kamu-cli/src/app/cli/src/commands/system_api_server_run_command.rs", line: 132 }, { fn: "<core::pin::Pin<P> as core::future::future::Future>::poll", file: "/rustc/1a648b397dedc98ada3dd3360f6d661ec2436c56/library/core/src/future/future.rs", line: 124 }, { fn: "kamu_cli::app::run::{{closure}}::{{closure}}::{{closure}}", file: "/home/sergeiz/kamu/kamu-cli/src/app/cli/src/app.rs", line:

Anything else?

There is an incompatibility between nextest and sqlx:

Collaborative work will be supported in https://github.com/launchbadge/sqlx/pull/3334

s373r commented 2 weeks ago

Waiting for the next PR to be merged