Cannot create dynamic Document for arrays and nested jsons

[sujaykakkad]

var username = "mongo"
var array = [1,2,3,4]
var dict = ["name": "kitten", "type": "cat"]

let userCollection = database["users"]

let userDocument: Document = [
            "username":  .string(username),
            "array": .array(arr),
            "dictionary": .array(dict),
        ]

try userCollection.insert(userDocument)

error: cannot convert value of type '[Int]' to expected argument type 'Document'
              array: .array(array),
                            ^~~~~

Why do you need Document type and why can't we create it dynamically?

Please provide JSON or Dictionary datatype support for queries in mongodb or atleast provide raw query support so that we can use raw strings to query mongo database

[Joannis]

I'll need to discuss with @Obbut if we can support JSON queries, and if so, how we'll implement it.

But about creating Document types dynamically, you can.

Our MongoKitten/BSON libraries support three methods of building documents.

• literals
• Value enum cases
• variables that are prefixes with the ~ operator (which converts them to a Value enum case

So your example code should be able to look like this too:

let userCollection = database["users"]

let userDocument: Document = [
            "username":  "mongo",
            "array": [1, 2, 3, 4],
            "dictionary": ["name": "kitten", "type": "cat"],
        ]

try userCollection.insert(userDocument)

or

var username = "mongo"
var array = [1,2,3,4]
var dict = ["name": "kitten", "type": "cat"]

let userCollection = database["users"]

let userDocument: Document = [
            "username":  ~username,
            "array": ~arr,
            "dictionary": ~dict,
        ]

try userCollection.insert(userDocument)

If you replace the lines:

var array = [1,2,3,4]
var dict = ["name": "kitten", "type": "cat"]

with this alternative version

var array = [1,2,3,4] as Document
var dict = ["name": "kitten", "type": "cat"] as Document

Your problems should be fixed, too. The problem here is that you're trying to assign a regular Swift Array/Dictionary to a Value. We can't make Swift support multiple input values for an enum case and convert it. So we'll need you to explicitly tell Swift that you're working with a Document variable from the start so that it can infer the rest of the Values automatically.

[sujaykakkad]

So what I am trying to achieve is that I get a json request and insert it to mongodb database.

Tell me a way to deserialize the data and add it to document object.

I also tried raw data but document is getting empty data

I am using Kitura framework for request and response

    var data: Foundation.Data = Foundation.Data()
    let result = try request.read(into: &data)
    let document: Document = Document(data: data)
    Log.info("Document: \(document)")

    INFO: MyWeb main.swift line 59 - Document: []

[Joannis]

Ah.. I understand the problem here. So when you initialize a Document using Data or [UInt8] it's expecting BSON data, not JSON. BSON is the format behind MongoDB. It's really fast and type safe, but not compatible with JSON. However, your JSON data can be parsed by our JSON library. We're using it ourselves without a problem.

Document(extendedJSON: myJSONString) will tell our BSON library to parse your JSON into a Document object which can then be inserted into MongoDB.

[sujaykakkad]

Thanks it worked as expected. Is there any method for array of jsons? eg:

var myJSONString ="[{'name': 'mongo', 'value': },{'name': 'kitten', 'value': 30}]" 

[Joannis]

Works the same way. extendedJSON also parses arrays as the top level element. It'll output a Document that's formatted as an Array. This looks in swift like this:

[
   "0": ["name": "mongo", "value": ...],
  "1": ["name": "kitten", "value": 30]
]

This can be converted to a real array, if you want, like this:

let documentArray = Document(extendedJSON: "[{'name': 'mongo', 'value': },{'name': 'kitten', 'value': 30}]")
let documents = documentArray.arrayValue

These documents you can bulk-insert or insert one-by-one.

[sujaykakkad]

Actually I want it work like multiple documents. If I get an array of jsons then it should work like insert[document, document] Also is there any library that can convert swifty json object to document directly?

[Joannis]

Do you mean a [String]? because you can map those.

let jsonStrings: [String] = ...

let documents = jsonStrings.map { Document(extendedJSON: $0) }

As for SwiftyJSON, we don't support it out of the box. Writing a converter shouldn't be too difficult, but it might not be something we'd like to support for various reasons.

[sujaykakkad]

I am creating 2 APIs One which gets only one json and insert it to mongo Another which gets an array of jsons and inserts it in separate documents in mongo So I am getting a request which is entirely string eg

1) "{"name": "mongo"}"
2) "[{"name": "mongo"}, {"name": "kitten"}]"

So the first one should insert the json document to mongo The next one should insert 2 separate documents to mongo

[Joannis]

let json = "[{\"name\": \"mongo\"}, {\"name\": \"kitten\"}]"

        let inputDocument = try Document(extendedJSON: json)

        // If this is an array
        if inputDocument.validatesAsArray() {
            let values = inputDocument.arrayValue
            let documents = values.flatMap { $0.documentValue }

            // TODO: Process these Documents, for example, store them in MongoDB

        } else {
            // single document

            // TODO: Process the single Document, for example, store it in MongoDB
        }

Before you process this information by inserting it directly into the database I'd highly recommend validating the input BSON. We've wrote some helpers for API Pagination, input validation etc in this repository. Due to the amount of work the two of us are facing (the two of us, being the only active contributors to OpenKitten) we can't actively maintain all libraries as quickly as we would like to. So I'd recommend copying any functionality you'd like from the files in the repository. Note that most of them are Vapor oriented. If you don't use Vapor, don't copy those features.

[sujaykakkad]

Thanks for the solution it is working great. I am doing input validation before converting to Document. Also can you tell me how can I handle the error if the connection is closed?

do{
   let insertUser = try database["user"].insert(insertDocument)
}catch{
   Log.info("\(error)")
}

The mongo connection is established then the mongod service is stopped so this will throw an error. However I am not able to catch the error in my catch block.

The error is somehow stuck in some loop

The MongoDB background loop encountered an error: Socket failed with code 54 
("Exchange full") 
[readFailed] "Unknown error"

I want to catch the error and send error response in my API.

[Joannis]

This is a bug we've fixed in 2.0.0. Until we're able to get our new API for Documents (which I hope to have fixed this weekend) there's no guarantee when we'll release the final version. Until then you'll have to keep up with our changes.

I'm unable to backport this to MongoKitten 1.x, so I'm afraid I can't help much with that beyond 2.0.0

[sujaykakkad]

Ok that won't be any problem for now. There is also another bug when I turn off my system and then turn on the mongod service is still running but the monogokitten never connects in first try. After sometime it gets connected I don't know how. It always shows this error when trying to connect for the first time when the system is turned on.

Socket failed with code 61 ("No data available") [connectFailed] "Unknown error"

Try to fix this issue as well in version 2

[Joannis]

@sujaykakkad Are you using MongoKitten 2.x? This should be fixed now. I'd like to hear if you're still experiencing issues

[sujaykakkad]

So here is what I did. I created a singleton static object of server connection.

public final class MongoDatabase{
    private static var server: Server!

    private init() {}

    public static func connect() throws {
        if server != nil{
            return
        }else{
            do{
                server = try Server(mongoURL: "mongodb://localhost:27017", automatically: true)
            }catch{
                throw CustomConnectionError
            }
        }
    }

    public static func getConnection() -> Server{
        return server
    }
}

So then I connect using connect method get server connection

Then I use this object to query database in other function

let server = MongoDatabase.getConnection()
try server["dbname"]["collection"]?.find()

After I use connect method to connect to database I stop mongod service So it tries to reconnect to mongo and if I start again it connects and gives result If I start and stop mongd service multiple times it throws Broken Pipe Error

So my question is I want to check if mongo is connected and if not try to connect it and if still the connection fails it should throw an error and I can send connection failed response. As soon as the connection problem is solved it should reconnect when the query is called and not give broken pipe error

Something like this

if !server.isConnected { try server.connect() }
try server["dbname"]["collection"]?.find()

But the if condtion is always false even if I stop mongod service Can you suggest me a better to solve this problem

[Joannis]

Sorry for the late reply! I didn't see this in my feed and just checked back on this ticket.

isConnected needs to be reimplemented I think. We've moved to another connection system underneath. I'll give this a try right now.

[sujaykakkad]

Ok. Also tell me a better way to handle this and can mongodb interaction be asynchronous like in node.js mongodb driver?

[Joannis]

It sure can! But we kept it synchronous within the driver to make most easier. I recommend using Dispatch queues for making calls asynchronous.

[sujaykakkad]

Ok I implemented dispatch queue and it is working asynchronously. So now I have a json string and I want to perform some validation. I want to create a json schema and validate input against it.

{
 "name": "abc" ,    //Required Not null String
 "email": "abc@gmail.com" ,  //Can be null String
 "age": 25    //Required Not null Integer
}

Is there any way I can perform validation with document also how can I convert document to dictionary?

[Joannis]

I personally use this for schemas and validation. And Document(extendedJSON: myJsonString) works for parsing MongoDB extended JSON as described here.

document.dictionaryValue is used for converting a Document to a Dictionary. But we use BSON's Value type as the Dictionary Value.

[sujaykakkad]

Ok So majority problem is solved with MongoKitten. Only isConnected has to be reimplemented so how long will it take to reimplement it?

[Joannis]

It's already done :)

[Joannis]

MongoKitten 2's latest patch

[sujaykakkad]

So should I delete Packages folder and then rebuild the swift project?

[Joannis]

swift build --clean dist to clean the folder swift package generate-xcodeproj to create a new project

That should do the trick!

[sujaykakkad]

How can I call a function in mongodb?

[Joannis]

What kind of a function?

[sujaykakkad]

A stored function in mongodb

[sujaykakkad]

https://docs.mongodb.com/manual/tutorial/store-javascript-function-on-server/

[Joannis]

I'm unsure if you can call a function from MongoKitten since I can't find anything in the MongoDB documentation about calling javascript functions.

[Joannis]

This apparently is a MongoDB client only feature. The database and connectors like MongoKitten can't do this.

[sujaykakkad]

Yes I thought so. Also use of functions is not recommended.

[sujaykakkad]

How can I partially update document

db.products.update(
   { _id: 100 },
   { $set:
      {
        quantity: 500,
        details: { model: "14Q3", make: "xyz" },
        tags: [ "coats", "outerwear", "clothing" ]
      }
   }
)

[Joannis]

Same way.

try productsCollection.update(matching: "_id" == 100, to: [
  "$set": [
    "quantity": 500,
    "details": ...
  ]
])

[sujaykakkad]

cannot insert document with key as "balance"

let doc: Document = ["name": "mongo"]
doc["balance"] = 0.0
print(doc["balance"]) //This prints balance as 0.0
try collection.insert(doc) //This inserts document but my balance field is not present

In mongo database only object_id and name field are there

{
_id: ObjectId("3f934658aff5802af4ad7690"),
name: "mongo"
}

Is there some bug with field name as "balance"

[sujaykakkad]

"Balance" is working but there is something wrong with "balance"

[Joannis]

I'm unable to reproduce the problem. But since you're using Mainecoon I'll look into any problems there might be with the schema backing this. Mainecoon's schema is also uploaded to the MongoDB collection which then also validates the Document.

[Joannis]

Which MongoKitten and MongoDB version do you run? This might help me investigate a bit better.

[sujaykakkad]

MonoKitten - 2.0.8 MongoDB - 3.2.10

[sujaykakkad]

I don't think it is the Mainecoon issue because I am printing the balance value after validation.

[sujaykakkad]

Also I have a json

{"name": "mongo"}

I want to check if the value is string or not

let json = Document(extendedJSON: "{\"name\": \"mongo\"}")
if (json["name"] == .string){
    //do something
}

[lgaches]

if (json["name"] is String) {
   print("is a string")
}

[Joannis]

@sujaykakkad

if case .string(_) = json["name"] {
  print("is a string")
}

This will change for the better in MongoKitten 3 😬

[sujaykakkad]

It is not working. It is giving a warning and the result is always false.

if (json["name"] is String) //Cast from Value to unrelated type String always fails

[Joannis]

I've updated my answer since GitHub didn't show all of it.

[sujaykakkad]

Why can't I insert document with "balance" as key?

[sujaykakkad]

Did you find the solution?

[Joannis]

I don't know any reason why it wouldn't be able to insert "balance". It even functions as expected on all servers I tested.

[sujaykakkad]

Ok I found the bug. It is not related with the key "balance". There is some name conflict and it also changes the value inside another key. eg

var mydoc: Document = ["balance_status": "Pending"] 
mydoc["balance"] = 50 //If I use the starting part of any key of mydoc it will happen. This will also happen if I use mydoc["balance_"] 
print("\(mydoc)") //This prints {"balance_status":{"$numberLong":"50"}}
print("\(mydoc['balance'])") //This prints int64(50)
try collection.insert(mydoc) 

When I do this "balance" key is not added in document but "balance_status" is changed to 50. So it is altering my document and also not adding my key. Please try to fix this.

[Joannis]

I wonder how we never stumbled upon this bug... I'm making your provided code a test and I'll push the fix

[sujaykakkad]

I guess I should close this issue and reopen another one with this bug.