AbsaOSS / enceladus

Dynamic Conformance Engine
Apache License 2.0
31 stars 14 forks source link

Counting runs count takes too much MongoDB CPU #2199

Open benedeki opened 1 year ago

benedeki commented 1 year ago

Describe the bug

When application is counting Runs count for the landing page info, it consumer too much CPU.

To Reproduce

See server logs

Expected behavior

It might not be necessary to run full scan of the table.

Log

2023-10-27T06:31:47.590+0000 I COMMAND  [conn7104853] command menas_galactic.run_v1 command: aggregate { aggregate: "run_v1", pipeline: [ { $match: {} }, { $group: { _id: 1, n: { $sum: 1 } } } ], cursor: {}, $db: "menas_galactic", $clusterTime: { clusterTime: Timestamp(1698388263, 159), signature: { hash: BinData(0, E86133DC3872EAE27881D51B179B0E150F1E874A), keyId: 7256402329547571201 } }, lsid: { id: UUID("5a5b1fa6-3fc7-4f36-9a38-e9ae6c73e7cd") } } planSummary: COLLSCAN keysExamined:0 docsExamined:6077042 cursorExhausted:1 numYields:47554 nreturned:1 reslen:259 locks:{ Global: { acquireCount: { r: 47556 } }, Database: { acquireCount: { r: 47556 } }, Collection: { acquireCount: { r: 47556 } } } storage:{ data: { bytesRead: 15558954293, timeReadingMicros: 17675795 }, timeWaitingMicros: { cache: 2796143 } } protocol:op_msg 42437ms
benedeki commented 11 months ago

Release notes: Replaced DB-intensive documentCount for estimatedDocumentCount. That should improve the load on MongoDB