search

facebookincubator / velox

A composable and fully extensible C++ execution engine library for data management systems.
https://velox-lib.io/
Apache License 2.0
3.53k stars 1.16k forks source link

Flaky test AsyncDataCacheTest.ttl/0 #11551

Open zhli1142015 opened 1 week ago

zhli1142015 commented 1 week ago

Problem description

[ RUN      ] AsyncDataCacheTest/AsyncDataCacheTest.ttl/0
I20241114 08:55:17.632177 21144 SsdCache.cpp:45] [SSDCA] SSD cache config: 4 shards, capacity 128.00MB, checkpoint size 6.40MB, file cow ENABLED, checksum DISABLED, read verification DISABLED
W20241114 08:55:17.632278 21144 SsdFile.cpp:878] [SSDCA] Starting shard 0 without checkpoint, with checksum write disabled, read verification disabled, checkpoint file /tmp/velox_test_k2IJrO/cache0.cpt
W20241114 08:55:17.632366 21144 SsdFile.cpp:878] [SSDCA] Starting shard 1 without checkpoint, with checksum write disabled, read verification disabled, checkpoint file /tmp/velox_test_k2IJrO/cache1.cpt
W20241114 08:55:17.632467 21144 SsdFile.cpp:878] [SSDCA] Starting shard 2 without checkpoint, with checksum write disabled, read verification disabled, checkpoint file /tmp/velox_test_k2IJrO/cache2.cpt
W20241114 08:55:17.632550 21144 SsdFile.cpp:878] [SSDCA] Starting shard 3 without checkpoint, with checksum write disabled, read verification disabled, checkpoint file /tmp/velox_test_k2IJrO/cache3.cpt
W20241114 08:55:17.632655 21144 MemoryArbitrator.cpp:84] Query memory capacity[32.00MB] is set for NOOP arbitrator which has no capacity enforcement
I20241114 08:55:17.758888 43268 AsyncDataCache.cpp:573] [SSDCA] Limiting SSD save batch to 280 entries
I20241114 08:55:17.758948 43268 AsyncDataCache.cpp:573] [SSDCA] Limiting SSD save batch to 295 entries
I20241114 08:55:17.758965 43268 AsyncDataCache.cpp:573] [SSDCA] Limiting SSD save batch to 277 entries
I20241114 08:55:17.758972 43268 AsyncDataCache.cpp:573] [SSDCA] Limiting SSD save batch to 282 entries
I20241114 08:55:17.762882 43315 SsdFile.cpp:345] [SSDCA] Grow cache file /tmp/velox_test_k2IJrO/cache0 to 1 regions (max: 1)
I20241114 08:55:17.764694 43317 SsdFile.cpp:345] [SSDCA] Grow cache file /tmp/velox_test_k2IJrO/cache1 to 1 regions (max: 1)
I20241114 08:55:17.769876 43318 SsdFile.cpp:345] [SSDCA] Grow cache file /tmp/velox_test_k2IJrO/cache2 to 1 regions (max: 1)
I20241114 08:55:17.775424 43319 SsdFile.cpp:345] [SSDCA] Grow cache file /tmp/velox_test_k2IJrO/cache3 to 1 regions (max: 1)
I20241114 08:55:17.779536 43319 SsdCache.cpp:138] [SSDCA] Wrote 2.97MB, inf bytes/s
I20241114 08:55:17.878214 43256 AsyncDataCache.cpp:573] [SSDCA] Limiting SSD save batch to 358 entries
I20241114 08:55:17.878252 43256 AsyncDataCache.cpp:573] [SSDCA] Limiting SSD save batch to 366 entries
I20241114 08:55:17.878265 43256 AsyncDataCache.cpp:573] [SSDCA] Limiting SSD save batch to 370 entries
I20241114 08:55:17.895213 43372 SsdFile.cpp:747] [SSDCA] Checkpointing shard 1, force: 0 bytesAfterCheckpoint: 1.95MB checkpointIntervalBytes: 1.60MB
I20241114 08:55:17.898605 43374 SsdFile.cpp:747] [SSDCA] Checkpointing shard 2, force: 0 bytesAfterCheckpoint: 1.63MB checkpointIntervalBytes: 1.60MB
I20241114 08:55:17.901418 43374 SsdFile.cpp:857] [SSDCA] Checkpoint persisted with 163 cache entries
I20241114 08:55:17.905488 43377 SsdFile.cpp:747] [SSDCA] Checkpointing shard 3, force: 0 bytesAfterCheckpoint: 1.68MB checkpointIntervalBytes: 1.60MB
I20241114 08:55:17.911123 43377 SsdFile.cpp:857] [SSDCA] Checkpoint persisted with 168 cache entries
I20241114 08:55:17.919091 43372 SsdFile.cpp:857] [SSDCA] Checkpoint persisted with 195 cache entries
I20241114 08:55:17.919142 43372 SsdCache.cpp:138] [SSDCA] Wrote 3.70MB, inf bytes/s
I20241114 08:55:17.927124 43385 AsyncDataCache.cpp:618] [CACHE] Removed 0 AsyncDataCache entries.
I20241114 08:55:17.929157 43385 AsyncDataCache.cpp:618] [CACHE] Removed 510 AsyncDataCache entries.
I20241114 08:55:17.932183 43385 AsyncDataCache.cpp:618] [CACHE] Removed 517 AsyncDataCache entries.
I20241114 08:55:17.935236 43385 AsyncDataCache.cpp:618] [CACHE] Removed 522 AsyncDataCache entries.
I20241114 08:55:17.936296 43385 SsdFile.cpp:671] [SSDCA] Removed 141 entries from /tmp/velox_test_k2IJrO/cache0. And erased 1 regions with 50% entries removed, and 0 left.
I20241114 08:55:17.936650 43385 SsdFile.cpp:671] [SSDCA] Removed 195 entries from /tmp/velox_test_k2IJrO/cache1. And erased 1 regions with 50% entries removed, and 0 left.
I20241114 08:55:17.936947 43385 SsdFile.cpp:671] [SSDCA] Removed 163 entries from /tmp/velox_test_k2IJrO/cache2. And erased 1 regions with 50% entries removed, and 0 left.
I20241114 08:55:17.937261 43385 SsdFile.cpp:671] [SSDCA] Removed 168 entries from /tmp/velox_test_k2IJrO/cache3. And erased 1 regions with 50% entries removed, and 0 left.
I20241114 08:55:17.937273 43385 CacheTTLController.cpp:68] Succeeded applying cache TTL of 98 seconds. Entries from 66 files are to be removed, while 0 files are retained
I20241114 08:55:17.937394 43384 AsyncDataCache.cpp:618] [CACHE] Removed 497 AsyncDataCache entries.
I20241114 08:55:17.938243 43384 AsyncDataCache.cpp:618] [CACHE] Removed 0 AsyncDataCache entries.
I20241114 08:55:17.938344 43384 AsyncDataCache.cpp:618] [CACHE] Removed 0 AsyncDataCache entries.
I20241114 08:55:17.938447 43384 AsyncDataCache.cpp:618] [CACHE] Removed 0 AsyncDataCache entries.
I20241114 08:55:17.938455 43384 SsdFile.cpp:671] [SSDCA] Removed 0 entries from /tmp/velox_test_k2IJrO/cache0. And erased 0 regions with 50% entries removed, and 0 left.
I20241114 08:55:17.938464 43384 SsdFile.cpp:671] [SSDCA] Removed 0 entries from /tmp/velox_test_k2IJrO/cache1. And erased 0 regions with 50% entries removed, and 0 left.
I20241114 08:55:17.938472 43384 SsdFile.cpp:671] [SSDCA] Removed 0 entries from /tmp/velox_test_k2IJrO/cache2. And erased 0 regions with 50% entries removed, and 0 left.
I20241114 08:55:17.938480 43384 SsdFile.cpp:671] [SSDCA] Removed 0 entries from /tmp/velox_test_k2IJrO/cache3. And erased 0 regions with 50% entries removed, and 0 left.
I20241114 08:55:17.938488 43384 CacheTTLController.cpp:68] Succeeded applying cache TTL of 98 seconds. Entries from 66 files are to be removed, while 0 files are retained
/home/runner/work/velox/velox/velox/velox/common/caching/tests/AsyncDataCacheTest.cpp:1354: Failure
Expected equality of these values:
  statsTtl.ssdStats->entriesAgedOut
    Which is: 667
  statsT1.ssdStats->entriesCached
    Which is: 297
I20241114 08:55:17.938970 21144 SsdCache.cpp:209] [SSDCA] SSD cache is shutting down
I20241114 08:55:17.938992 21144 SsdFile.cpp:747] [SSDCA] Checkpointing shard 0, force: 1 bytesAfterCheckpoint: 1.41MB checkpointIntervalBytes: 1.60MB
I20241114 08:55:17.941179 21144 SsdFile.cpp:857] [SSDCA] Checkpoint persisted with 0 cache entries
I20241114 08:55:17.941211 21144 SsdFile.cpp:747] [SSDCA] Checkpointing shard 1, force: 1 bytesAfterCheckpoint: 0B checkpointIntervalBytes: 1.60MB
I20241114 08:55:17.941892 21144 SsdFile.cpp:857] [SSDCA] Checkpoint persisted with 0 cache entries
I20241114 08:55:17.941926 21144 SsdFile.cpp:747] [SSDCA] Checkpointing shard 2, force: 1 bytesAfterCheckpoint: 0B checkpointIntervalBytes: 1.60MB
I20241114 08:55:17.942683 21144 SsdFile.cpp:857] [SSDCA] Checkpoint persisted with 0 cache entries
I20241114 08:55:17.942718 21144 SsdFile.cpp:747] [SSDCA] Checkpointing shard 3, force: 1 bytesAfterCheckpoint: 0B checkpointIntervalBytes: 1.60MB
I20241114 08:55:17.943461 21144 SsdFile.cpp:857] [SSDCA] Checkpoint persisted with 0 cache entries
I20241114 08:55:17.943491 21144 SsdCache.cpp:216] [SSDCA] SSD cache has been shutdown
I20241114 08:55:17.953382 21144 TempDirectoryPath.cpp:29] TempDirectoryPath:: removing all files from /tmp/velox_test_k2IJrO
[  FAILED  ] AsyncDataCacheTest/AsyncDataCacheTest.ttl/0, where GetParam() = 2-byte object <00-00> (321 ms)

System information

https://github.com/facebookincubator/velox/actions/runs/11833246203/job/32971531602

CMake log

No response