GolosChain / golos

Socio-economic mediablockchain
https://developers.golos.io
Other
59 stars 36 forks source link

Mongo Add account name indexes to query data faster #843

Open maslenitsa93 opened 6 years ago

maslenitsa93 commented 6 years ago

Analysts are often using Mongo aggregation queries with $lookup, like this one:

db.account_object.aggregate([{
$match: {
}}, {
    $lookup: {
        "from": "comment_object",
        let: {
            authorr: "$name"
        },
        pipeline: [{$match: {$expr: {$and: [{"$eq": ["$author", "$$authorr"]}]}}}],
        "as": "his_comments"
    }
}])

Such queries are very slow. Let's run such query when we have ~590.000 comments. Result: 15:32 - Started 15:38 - Ended

It is so slow because we using comment_object.author field which is just string, and not indexed ObjectId field.

Let's add comment_object.author_id field with index on it, and run slightly rewrited query after full re-sync:

db.account_object.aggregate([{
$match: {
}}, {
    $lookup: {
        "from": "comment_object",
        let: {
            authorr: "$_id"
        },
        pipeline: [{$match: {$expr: {$and: [{"$eq": ["$author_id", "$$authorr"]}]}}}],
        "as": "his_comments"
    }
}])

Result: 15:43:02 - Started 15:43:03 - Ended

Indexes could strongly improve performance of using & development of Mongo aggregation queries.

In this issue:

afalaleev commented 6 years ago

Looks good. But decision is not required, now.