search

AbsaOSS / enceladus

Dynamic Conformance Engine
Apache License 2.0
29 stars 14 forks source link

Counting runs count takes too much MongoDB CPU #2199

Open benedeki opened 8 months ago

benedeki commented 8 months ago

Describe the bug

When application is counting Runs count for the landing page info, it consumer too much CPU.

To Reproduce

See server logs

Expected behavior

It might not be necessary to run full scan of the table.

Log

2023-10-27T06:31:47.590+0000 I COMMAND  [conn7104853] command menas_galactic.run_v1 command: aggregate { aggregate: "run_v1", pipeline: [ { $match: {} }, { $group: { _id: 1, n: { $sum: 1 } } } ], cursor: {}, $db: "menas_galactic", $clusterTime: { clusterTime: Timestamp(1698388263, 159), signature: { hash: BinData(0, E86133DC3872EAE27881D51B179B0E150F1E874A), keyId: 7256402329547571201 } }, lsid: { id: UUID("5a5b1fa6-3fc7-4f36-9a38-e9ae6c73e7cd") } } planSummary: COLLSCAN keysExamined:0 docsExamined:6077042 cursorExhausted:1 numYields:47554 nreturned:1 reslen:259 locks:{ Global: { acquireCount: { r: 47556 } }, Database: { acquireCount: { r: 47556 } }, Collection: { acquireCount: { r: 47556 } } } storage:{ data: { bytesRead: 15558954293, timeReadingMicros: 17675795 }, timeWaitingMicros: { cache: 2796143 } } protocol:op_msg 42437ms
benedeki commented 7 months ago

Release notes: Replaced DB-intensive documentCount for estimatedDocumentCount. That should improve the load on MongoDB