Loopback Issue with Update Operation - Data Type not stored correctly in Mongo DB

SandeepVanama commented 5 years ago

I am not able to update the document with correct field data type in Mongo DB using loopback mechanism. I tried with the model JSON file but it is only work for create operation but not for the update operation.

Loopback update operation defining the data types in a model file, is only working for the fields present at root level but not the fields present at Object, Arrary or nested object, nested array level. Fields present at root level alone are updated and saved correctly with the data type defined in model JSON file.

I have attached the sample model JSON file (movies.txt) along with the test output data (movies_sample_output.txt). I have tested the behavior with the loopback-connector-mongodb@3.9.2 version.

I am new to loopback coding, so can someone help check the issue and implement the fix? I can perform the testing and confirm the results.

Please let me know for any more details needed. Thank you in advance..!!

movies.txt Movies_sample_output.txt

SandeepVanama commented 5 years ago

@dhmlau, could you help with the fix for this issue? We have a project deadline and got stuck with this issue. I am happy to test and confirm the changes upon implementing the fix. Please let me know.

dhmlau commented 5 years ago

@SandeepVanama, I've created a sample app based on the information you've provided. https://github.com/dhmlau/loopback-movie-test.

I've tried the following and it seems to be working fine, so if you could let me know how you reproduce the issue, i'd appreciate it.

after starting the app, in the API Explorer (http://localhost:3000/explorer), i called /POST using the json in Movies_sample_forCreate.txt which is a modified version of your txt so that I can successfully make the call.
then I call /PATCH with the only instance I've put, and made changes in the following attributes and they seem to get updated successfully as well:
- title (at the root level)
- states (root level but an array)
- tickets theatre (nested properties)

cc @b-admike

SandeepVanama commented 5 years ago

@dhmlau, Thank you foe checking this.

The main issue here is with the field data type which is not retained as per the model JSON file after the update operation execution. I see the fields that you have tested are string types, not the Decimal, Date and Integer types. Please correct me if I am wrong.

Upon preforming the update operation, the Decimal, Date and Integer type fields are not getting updated with its relevant data types defined in MongoDB (such as NumberDecimal, ISODate and NumberInt etc). Instead they are represented as String data after the update operation.

I have attached the actual and expected results in "MongoDB_Update_Operation _Loopback-connector-mongodb_3_9_2.docx". Can you please test the same scenario with Decimal, Date and Integer type fields? Please let me know for any more details needed.

Please check and let me know for any further details needed.

SandeepVanama commented 5 years ago

Actual results vs Expected results Document uploaded.

MongoDB_Update_Operation _Loopback-connector-mongodb_3_9_2.docx

b-admike commented 5 years ago

Thank you for the updates @SandeepVanama. Where are you checking the values from to compare actual vs expected? Are you querying what is stored in the mongo database directly or are you checking it in LoopBack's API Explorer?

SandeepVanama commented 5 years ago

@b-admike I am checking the results by directly logging into MongoDB database through Studio 3T & Compass tool. Please let me know for anymore details needed.

b-admike commented 5 years ago

@SandeepVanama I compared two requests (POST and then PATCH by id) using @dhmlau's sample repo, but I don't see any change in the type of the values returned with an update in Compass from the file provided to show actual vs expected results:

create_movies_instance

I changed some int/decimal values to 1111 and date values to end in 0000: update_movies_instance Similarly from the mongo shell:

> db.movies.find({})
{ "_id" : ObjectId("5c631f211dc8f557f6d80fec"), "title" : "Once Upon a Time in the West-5", "releaseDate" : ISODate("2019-01-29T04:00:39.828Z"), "rating" : NumberDecimal("4.25"), "runtime" : 135, "states" : [ "IL", "MI", "IN" ], "randomReview" : [ "3.5", "4.5", "4.0" ], "randomReleaseDates" : [ "Tue Feb 12 2019 14:31:45 GMT-0500 (Eastern Standard Time)", "Tue Feb 12 2019 14:31:45 GMT-0500 (Eastern Standard Time)", "Tue Feb 12 2019 14:31:45 GMT-0500 (Eastern Standard Time)" ], "tickets" : [ { "theatre" : "AMC", "unitprice" : "19.5", "capacity" : 205 }, { "theatre" : "IMAX", "unitprice" : "39.5", "capacity" : 300 } ], "awards" : { "wins" : 4, "prizeMoney" : "100000.00", "currency" : "USD" }, "imdb" : { "duration" : 135, "reviewDate" : ISODate("2019-01-29T04:00:39.828Z"), "rating" : "4.5", "innerArray" : [ { "testNumber" : 1234, "testDate" : ISODate("2019-01-30T23:26:38.551Z"), "testDecimal" : "99.55", "ObjInsideAnArray" : { "testNumber" : 1234, "testDate" : ISODate("2019-01-30T23:26:38.551Z"), "testDecimal" : "99.55" }, "nestedArray" : [ { "testNumber" : 1234, "testDate" : ISODate("2019-01-30T23:26:38.551Z"), "testDecimal" : "99.55" }, { "testNumber" : 1234, "testDate" : ISODate("2019-01-30T23:26:38.551Z"), "testDecimal" : "99.55" } ] }, { "testNumber" : 5678, "testDate" : ISODate("2019-01-30T23:26:38.551Z"), "testDecimal" : "99.55", "ObjInsideAnArray" : null, "nestedArray" : null } ], "innerObj" : { "innerObj" : { "innerObj" : { "innerObj" : { "testNumber" : 6666, "testDate" : ISODate("2019-01-30T23:26:38.551Z"), "testDecimal" : "55.55" } } } } } }

> db.movies.find({})
{ "_id" : ObjectId("5c631f211dc8f557f6d80fec"), "title" : "Once Upon a Time in the West-5", "releaseDate" : ISODate("2019-01-29T04:00:39.828Z"), "rating" : NumberDecimal("4.25"), "runtime" : 135, "states" : [ "IL", "MI", "IN" ], "randomReview" : [ "3.5", "4.5", "4.0" ], "randomReleaseDates" : [ "Tue Feb 12 2019 16:38:46 GMT-0500 (Eastern Standard Time)", "Tue Feb 12 2019 16:38:46 GMT-0500 (Eastern Standard Time)", "Tue Feb 12 2019 16:38:46 GMT-0500 (Eastern Standard Time)" ], "tickets" : [ { "theatre" : "AMC", "unitprice" : "19.5", "capacity" : 111 }, { "theatre" : "IMAX", "unitprice" : "39.5", "capacity" : 300 } ], "awards" : { "wins" : 111, "prizeMoney" : "100000.00", "currency" : "USD" }, "imdb" : { "duration" : 111, "reviewDate" : ISODate("2019-01-29T04:00:00Z"), "rating" : "4.5", "innerArray" : [ { "testNumber" : 1234, "testDate" : ISODate("2019-01-30T23:26:38.551Z"), "testDecimal" : "99.55", "ObjInsideAnArray" : { "testNumber" : 1111, "testDate" : ISODate("2019-01-30T23:26:38Z"), "testDecimal" : "11.11" }, "nestedArray" : [ { "testNumber" : 1111, "testDate" : ISODate("2019-01-30T23:26:38.551Z"), "testDecimal" : "99.55" }, { "testNumber" : 1234, "testDate" : ISODate("2019-01-30T23:26:38.551Z"), "testDecimal" : "99.55" } ] }, { "testNumber" : 5678, "testDate" : ISODate("2019-01-30T23:26:38.551Z"), "testDecimal" : "99.55", "ObjInsideAnArray" : null, "nestedArray" : null } ], "innerObj" : { "innerObj" : { "innerObj" : { "innerObj" : { "testNumber" : 1111, "testDate" : ISODate("2019-01-30T23:26:38.551Z"), "testDecimal" : "55.55" } } } } } }

Were you using a different endpoint? Also, can you please use the latest loopback-connector-mongodb version (4.0.0) to run your tests? I am using latest version of mongo as well (4.0.6):

# mongo --version
MongoDB shell version v4.0.6
git version: caa42a1f75a56c7643d0b68d3880444375ec42e3
OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
allocator: tcmalloc
modules: none
build environment:
    distmod: ubuntu1604
    distarch: x86_64
    target_arch: x86_64

SandeepVanama commented 5 years ago

@b-admike, I am using IBM App Connect Enterprise (ACE) v11.0.0.2 middleware product to integrate with MongoDB. This product has Loopback Request Node to perform loopback operations. The source code is invisible and embedded with in the product.

The Loopback Request Node takes care of executing the requests by using the connector modules pre-loaded onto ACE. I will upgrade the loopback-connector-mongodb module to version 4.0.0 and test again.

I am using Studio 3T / Compass software only to verify the results in MongoDB.

Please let me for any clarifications required.

b-admike commented 5 years ago

@SandeepVanama While I reproduce this issue in a test in mongodb connector, we have landed a recent fix on loopback-connector to get nested property definitions properly. Can you please try again using loopback-connector@4.6.0? A simple npm install if you have the latest (4.0.0) version of mongodb connector should be sufficient. You can check with npm ls loopback-connector after.

SandeepVanama commented 5 years ago

@b-admike, I have tested the scenario with the loopback-connector-mongodb@4.0.0 and loopback-connector@4.6.0 and no change in the behavior. The issue still persists with these connectors modules as well.

Please find below the npm ls loopback-connector and npm ls mongodb command output.

C:\Projects\ace\TESTNODE_ace\node_modules>npm ls loopback-connector C:\Projects\ace\TESTNODE_ace -- loopback-connector-mongodb@4.0.0 -- loopback-connector@4.6.0

C:\Projects\ace\TESTNODE_ace\node_modules>npm ls mongodb C:\Projects\ace\TESTNODE_ace -- loopback-connector-mongodb@4.0.0 -- mongodb@3.1.13

Could you check and let me know if anything else can be tried? Thank you for your support.

SandeepVanama commented 5 years ago

@b-admike, Please find attached the trace files generated during our testing today with the standalone loopback application shared by @dhmlau.

Sample JSON File used for testing: SampleJSON.txt

Debug Trace Files: CreateOperation_DebugTrace.txt UpdateOperation_DebugTrace.txt

Issue Discovered:

Create Operation:

The Decimal type fields not stored correctly as NumberDecimal when present in nested array level

Update Operation:

The Decimal type fields not stored correctly as NumberDecimal when present in any level except at root level. Issue with Date fields not working as expected. Date fields stored as string value instead of actual Date type value. Eg: Actual format stored: "testDate" : "TIMESTAMP '2019-02-27 16:31:07.813371'", Expected format: "testDate" : ISODate("2019-02-27T16:31:07.813+0000"),

Please let me know for any more details needed.

b-admike commented 5 years ago

@SandeepVanama Are you able to test with the code in #501? Please note that it won't fix all of the issues you've observed because it has some dependence on https://github.com/strongloop/loopback-datasource-juggler/pull/1702.

b-admike commented 5 years ago

This closed automatically because #501 has landed, but will need the fix from https://github.com/strongloop/loopback-datasource-juggler/pull/1702 as well.

dhmlau commented 5 years ago

Closing as resolved.

loopbackio / loopback-connector-mongodb

Loopback Issue with Update Operation - Data Type not stored correctly in Mongo DB #493

Create Operation:

Update Operation: