Update server to allow extended Mongo query syntax

[nescohen]

Use mongodb parser and EJSON stringify to manipulate mongoDB query. This prevents some valid mongoDB queries from throwing JSON parsing errors.

Also in the same vain, fix bug whereby parenthesis in the JSON query cause parsing error.

[Mijobar]

Having trouble with the below query @nescohen . Using your forked code. Any suggestions? Thanks!

db.database.aggregate([{"$match": {"FileName": { "$in": [ /.paf/i]}}}, { "$group" : { "_id": "test", "avg_time": { "$avg": {"$divide": [{"$subtract": ["$running_endTime","$running_startTime"]},1000]} } } }])

[nescohen]

Hey @Mijobar, Can you go into a bit more detail about what issues you are running into? I tried running this query on my instance and I did not see any parsing errors. Although the query returned no results presumably because my DB schema does not match yours. One thing to make sure to do is to restart the node server if you have not done that. My patch made changes mostly to that.

[Mijobar]

Thanks for reaching out @nescohen ! A picture/s worth a thousand words! =P

Query from Robo3T: Works as expected

Query from Grafana Using your Mongo Server:

If I put quotes around what is in the array, (Ex: [" /.paf/i"] ) I don't get an error, but the query returns zero matches.

Thanks!

[Mijobar]

Hi

Thanks for reaching out @nescohen ! A picture/s worth a thousand words! =P

Query from Robo3T: Works as expected

Query from Grafana Using your Mongo Server:

If I put quotes around what is in the array, (Ex: [" /.paf/i"] ) I don't get an error, but the query returns zero matches.

Thanks!

HI @nescohen Any thoughts on what the issue could be? Thanks!

[Mijobar]

Hi @nescohen bumping this issue. Should I continue pursuing this issue or look to other avenues for a solution?

Thanks for all your help thus far! Much appreciated!

[nescohen]

@Mijobar, sorry man I missed this. One thing you could try here is quoting the /.paf/i in your query. This looks like a json parsing issue because the plugin must parse the json of your query in order to send it to grafana. Hopefully that would not mess up your query.

I don't know if you are familiar with javascript as well, but if you look into the source for the plugin, you can see the parsing steps taken. Some mongo queries may not be parsable into correct JSON which can make this process difficult.

[Mijobar]

Hi @nescohen ! Thanks for getting back to me. Unfortunately, "/.paf/i" doesn't work.

It's definitely, a JSON parsing issue. I am not familiar with JavaScript, do you mind just pointing me to the file/lines in the source that handles the JSON parsing? I'll then try to make a fix.

Thanks!

[nescohen]

@Mijobar, Alright here's what I would recommend: I see two possible locations this error is occurring in. Both of them in server/mongodb-proxy.js.

One on line 13:

app.use(bodyParser.json());

This line sets the http request parser to JSON mode, so I believe all incoming requests must be valid json. It may be possible to work around this by only decoding the request as a string.

One on line 251:

docs = mongoParser(args)

This line actually parses the json into a query to send to mongo. It comes from this library: https://www.npmjs.com/package/mongodb-query-parser

I think some of the awkwardness of this comes from the fact that these libraries or really expecting you to build a query inside of javascript, I think they were not really designed to turn a string sent by graphana into a query. I hope this helps!

[sy6sy2]

Hi @nescohen, thank you for this PR. I tried your fork because I was getting the Unexpected end of JSON input error with original plugin. But even with your fork I get this error 😕

This is my query, maybe can you find the bug? BTW, this query works great on the MongoDB shell.

Thank you!

Edit: I found the issue, but I don't know how to fix it. The issue is caused by the ( and ) symbols in the project state in the words Filename(s) and Station(s)

db.manual.aggregate([
    {
        "$match": {
            "$or" : [
                {"md5": "foo"},
                {"md5": "fii"}
            ]
        }
    },
    {
        "$lookup": {
            "from": "stationfiles",
            "localField": "md5",
            "foreignField": "md5",
            "as": "files"
        }
    },
    {"$unwind" : "$files"},
    {
        "$lookup": {
            "from": "stations",
            "localField": "id",
            "foreignField": "station_id",
            "as": "stations"
        }
    },
    {"$unwind" : "$stations"},
    {
        "$group": {
            "_id": "$md5",
            "unique_filepaths": {
                "$addToSet": "$files.filepath"  
            },
            "unique_stations": {
                "$addToSet": "$stations.hostname"  
            },
            "category":  {"$first": "$category"},
            "md5": {"$first": "$md5"},
            "comment": {"$first": "$comment"}
        }
    },
    {
        "$project": {
            "_id": 0,
            "MD5": "$md5",
            "Filename(s)": {
                "$reduce": {
                    "input": {
                            "$map" : {
                                "input": "$unique_filepaths",
                                "as": "filepath",
                                "in": {"$last": {"$split" : ["$$filepath", "\\"]}}
                            }
                        },
                    "initialValue": "",
                    "in": {
                        "$concat": [
                            "$$value",
                            {"$cond": [{"$eq": ["$$value", ""]}, "", ", "]}, 
                            "$$this"]
                    }
                }
            },
            "Station(s)": {
                "$reduce": {
                    "input": {
                            "$map" : {
                                "input": "$unique_stations",
                                "as": "station",
                                "in": {"$last": {"$split" : ["$$station", "\\"]}}
                            }
                        },
                    "initialValue": "",
                    "in": {
                        "$concat": [
                            "$$value",
                            {"$cond": [{"$eq": ["$$value", ""]}, "", ", "]}, 
                            "$$this"]
                    }
                }
            },
            "Category": "$category",
            "Comment": "$comment"
        }
    }  
])

[nescohen]

@sy6sy2 Huh, that is interesting. If parenthesis are causing you problems, it is probably due to this line in server/mongodb-proxy.js:

    var closeBracketIndex = query.lastIndexOf(')')

I am guessing that removing the parenthesis from these keys is not an option? Are you familiar with javascript? Unfortunately the solution probably involves going in and improving the logic for detecting the beginning and end of the query. It's not immediately obvious to me what is going wrong.

[sy6sy2]

Thanks for your answer. It's not a problem for me to remove parenthesis from the key strings so everything works fine for me 😉. And no, I am not a JS expert at all 😕.