NS 15.0.2 on Azure( with CosmosDB ) craches when using API V3

[jlinden79]

Describe the bug

Nightscout docker container craches and restarts when using nsclient v3 in AAPS fullrelease or AAPS Client. AAPS 3.2 and NS 15.0.2 with nsclient v1 work fine, but changing to nsclient v3 aaps connects and can upload treatments. but at a specific request (getLastModified) makes NS to crash.

AAPs client cant get any data at all, since it seems that it starts by making getLastModified request. and the NS instance is restarted

Expected behavior Nightscout handles this exception in some way.

Your setup information

• Nightscout 15.0.2
• AAPs 3.2 collecting and uploading all data to NS.
• Nightscout hosted in Azure web app
• CosmosDB in Azure ( free tier)

Additional Info Attach logs that I found in Azure Portal file called docker file called default_docker and AAPS 3.2 logs logs.txt

aaps logs is in CET time and Docker logs in UTC logs shows exception at 20:26:03 (19:26:03 UTC)

[jlinden79]

Im new the source code of NS and AAPS. and to MongoDB as well I found the file where the exception is thrown. And logged in to my CosmosDB instance, to try get the collection as in the getLastModified function. lib/api3/storage/mongoCollection/index.js:81:37

I guess project(....) return an colletion and not throw a typeError. My question here is if my query is wrong. Or if I found why CosmosDB is not working well with nsclient V3? apperently the the find ...limit(1) returns data

[jlinden79]

After some digging the project function seems to be a nodejs function and not available in mongo shell 🤔 Question though Package MongoDB 3.6.0 compability list only is compatible up to nodejs 14.x and not nodejs 16.x https://www.mongodb.com/docs/drivers/node/current/compatibility/

Could that be the problem? The recommended nodejs version for azure is 16 .16 according to readme.md

[jlinden79]

Update: Adding index 'created_at' on collection food and profile fixed the Nightscout crasch issue, when calling GET ...../api/v3/lastModified

those "fallbackDateField" fields is not ensured. so they are not created

The API for MongoDB behaves differently from the Azure Cosmos DB for NoSQL, which indexes all fields by default. https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/indexing

Now to the Next CosmosDB incompability issue when calling GET ...../api/v3/food

the following exception is thrown this is handle by a try catch, so NS is still running

2023-11-16T18:19:57.405956408Z MongoError: Error=2, Details='Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: 1999b68e-e672-4dfc-9438-af91b5fd2895; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: 1999b68e-e672-4dfc-9438-af91b5fd2895; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: 1999b68e-e672-4dfc-9438-af91b5fd2895; Reason: (Message: {"Errors":["The order by query does not have a corresponding composite index that it can be served from."]}
2023-11-16T18:19:57.406011310Z ActivityId: 1999b68e-e672-4dfc-9438-af91b5fd2895, Request URI: /apps/d26640e5-33e4-43af-b997-dfd51c386334/services/949509aa-3868-4194-be5a-24b4995a2485/partitions/f9d88ed3-3232-4322-b2ec-eaa3eeba9b9b/replicas/133287153381072735s/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.17763 cosmos-netstandard-sdk/3.18.0);););
2023-11-16T18:19:57.406022210Z     at MessageStream.messageHandler (/opt/app/node_modules/mongodb/lib/cmap/connection.js:299:20)
2023-11-16T18:19:57.406506727Z     at MessageStream.emit (node:events:527:28)
2023-11-16T18:19:57.406525928Z     at processIncomingData (/opt/app/node_modules/mongodb/lib/cmap/message_stream.js:144:12)
2023-11-16T18:19:57.406533028Z     at MessageStream._write (/opt/app/node_modules/mongodb/lib/cmap/message_stream.js:42:5)
2023-11-16T18:19:57.406539428Z     at writeOrBuffer (node:internal/streams/writable:389:12)
2023-11-16T18:19:57.406545728Z     at _write (node:internal/streams/writable:330:10)
2023-11-16T18:19:57.406551928Z     at MessageStream.Writable.write (node:internal/streams/writable:334:10)
2023-11-16T18:19:57.406557429Z     at TLSSocket.ondata (node:internal/streams/readable:754:22)
2023-11-16T18:19:57.406563629Z     at TLSSocket.emit (node:events:527:28)
2023-11-16T18:19:57.406569129Z     at addChunk (node:internal/streams/readable:315:12) {
2023-11-16T18:19:57.406667132Z   ok: 0,
2023-11-16T18:19:57.406765036Z   code: 2,
2023-11-16T18:19:57.406772836Z   codeName: 'BadValue'
2023-11-16T18:19:57.406778936Z }

Is the search function https://github.com/nightscout/cgm-remote-monitor/blob/master/lib/api3/generic/search/operation.js using multiple fields in the sort method? perhaps in that case cosmosdb need compound indexes https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/indexing#compound-indexes-mongodb-server-version-36

Its to bad CosmosDB dont work as well with Nightscout any longer, when using api/v3 setting it all up in azure where quite easy.

I know it is very hard to test all querys against cosmosDB. and find deviations from mongodb.

Any Comments from DEV, regarding the compability with CosmosDB?

[jlinden79]

Update. Adding manual index in cosmosDB fixed the issue with .GET ...../api/v3/food/ Executed this to create indexes, using the mongoshell (mongosh)

db.food.createIndex( {
   identifier: 1,
   created_at:  1,
   date:  1
} )

db.food.createIndex( {
   identifier: -1,
   created_at:  -1,
   date:  -1
} )

indexes created from the lib/api3/generic/search/input.js sort function since this is a generic function for all collections I must create the same indexes for all collections that are searched for using api/v3

[jlinden79]

Added the previous mentioned index helped for using GET httpss://........../api/v3/collection

but i didnt fix the issue requesting data using parameters,

Example that aapsClient is using GET /api/v3/entries?sort=date resturns: Invoke-WebRequest: {"status":500,"message":"Database error"} and the exception in NS 2023-12-02T18:24:09.737298838Z: [ERROR] MongoError: Error=2, Details='Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: 4beb52b7-53a3-437d-bcde-d6f5725c991b; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: 4beb52b7-53a3-437d-bcde-d6f5725c991b; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: 4beb52b7-53a3-437d-bcde-d6f5725c991b; Reason: (Message: {"Errors":["The order by query does not have a corresponding composite index that it can be served from."]}

Any suggestions? Creating even more indexes, or migrate away from CosmosDB?

As I think it because of this https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/indexing#compound-indexes-mongodb-server-version-36