bmrlab / gendam

A privacy-first generative DAM
6 stars 1 forks source link

prisma 并发写入 sqlite 到达一定量都会有数据丢失。而且没报错。有点像是数据被“覆盖”了。 #7

Closed web3nomad closed 3 months ago

web3nomad commented 3 months ago

不管是每次重新创建 prisma client 还是 share 一个 prisma client 的引用,并发写入到达一定量都会有数据丢失。而且没报错。有点像是数据被“覆盖”了。

测试用例见 https://github.com/bmrlab/tauri-dam-test-playground/blob/8a5c9c2a09bf9925f29ed839edfde801a1fdf50e/crates/prisma/src/lib.rs#L40

web3nomad commented 3 months ago

找到原因了,并发太高导致写入超时以后 rollback 了。

2024-03-11T17:15:46.273550Z DEBUG quaint:query{db.statement=BEGIN}: quaint::connector::metrics: query=BEGIN params=[] result="success" item_type="query" is_query=true duration_ms=0
2024-03-11T17:15:51.475881Z DEBUG prisma:engine:interpret:prisma:engine:write-execute:quaint:query{db.statement=INSERT INTO `main`.`AssetObject` (`note`, `createdAt`, `updatedAt`) VALUES (?,?,?) RETURNING `id` AS `id`}: quaint::connector::metrics: query=INSERT INTO `main`.`AssetObject` (`note`, `createdAt`, `updatedAt`) VALUES (?,?,?) RETURNING `id` AS `id` params=["530","2024-03-11 17:15:45.909 UTC","2024-03-11 17:15:45.909 UTC"] result="error" item_type="query" is_query=true duration_ms=5202
2024-03-11T17:15:51.476667Z DEBUG quaint:query{db.statement=ROLLBACK}: quaint::connector::metrics: query=ROLLBACK params=[] result="success" item_type="query" is_query=true duration_ms=0

错误是超时 Execute(Error { is_panic: false, inner: Unknown(UnknownError { message: "Error occurred during query execution:\nConnectorError(ConnectorError { user_facing_error: None, kind: ConnectionError(Timed out during query execution.) })", backtrace: None }), batch_request_idx: None })

之前忘记处理 Result 一下判断是否执行成功了。这里的超时看起来不是 connection 抢不到超时,而是写入 sqlite 文件超时。

所以,单一 prisma client 同时 connection_limit > 1 并发开大,是可以的。 但是如果遇到执行超时,其实也没有办法,只能控制好频率,然后接口要处理好 Result 并返回错误,任务要有重试机制。