Can't connect to Azure CosmosDB (MongoDB)

[devrnt]

Can't connect to Azure CosmosDB instance with MongoDB api with the fiware/orion-ld image. However, this works perfectly fine with the fiware/orion:latest image (latest being 3.1.0-PRE-43).

This orion-ld image works perfectly fine when MongoDB runs on another docker container (test it by replacing the orion-ld image with orion in the command below).

Docker

docker run -d --name orion-ld -p 1026:1026 fiware/orion-ld:latest -dbhost <azure-cosmos-db-host> -dbuser <azure-cosmos-db-user> -dbpwd <azure-cosmos-db-password> -rplSet globaldb -dbTimeout 120000 -logLevel DEBUG -dbSSL -writeConcern 0

Orion logs

time=Monday 26 Apr 11:40:03 2021.139Z | lvl=INFO | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[1049]:main | msg=Orion Context Broker is running

time=Monday 26 Apr 11:40:03 2021.139Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[863]:versionInfo | msg=Version Info:

time=Monday 26 Apr 11:40:03 2021.139Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[864]:versionInfo | msg=-----------------------------------------

time=Monday 26 Apr 11:40:03 2021.140Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[865]:versionInfo | msg=orionld version: post-v0.7

time=Monday 26 Apr 11:40:03 2021.140Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[866]:versionInfo | msg=based on orion: 1.15.0-next

time=Monday 26 Apr 11:40:03 2021.140Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[867]:versionInfo | msg=git hash: nogitversion

time=Monday 26 Apr 11:40:03 2021.140Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[868]:versionInfo | msg=build branch:

time=Monday 26 Apr 11:40:03 2021.140Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[869]:versionInfo | msg=compiled by: root

time=Monday 26 Apr 11:40:03 2021.140Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[870]:versionInfo | msg=compiled in: buildkitsandbox

time=Monday 26 Apr 11:40:03 2021.140Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[871]:versionInfo | msg=-----------------------------------------

time=Monday 26 Apr 11:40:08 2021.321Z | lvl=ERROR | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=mongoConnectionPool.cpp[195]:mongoConnect | msg=Database Startup Error (cannot connect to mongo - doing 100 retries with a 1000 microsecond interval)

Related?

• https://github.com/telefonicaid/fiware-orion/issues/3797: I saw something about the use of a new MongoDB driver, I'm not sure if this orion-ld image is already updated

[kzangeli]

Sorry to hear you're having problems ... First doubt, does "Azure CosmosDB" open the port 27017, just like mongo?

Personally I have never tried out Orion-LD with anything but mongo. I know some people have successfully tried it with Amazon's DocumentDB.

About Orion-LD / Orion: Orion-LD forked out from Orion back in Sep 2019, and hasn't been refreshed since then (too much overhead). Some day, the idea is that the Orion-LD code is to be merged into the Orion repo, but I don't even have an estimated date for that. The merge is a huge job and it will take quite some time for it to happen.

If you instruct me on how to start a azure cosmosdb, I could try to connect Orion-LD to it. But, officially, only mongo is supported I'm afraid (at least for now).

[ AFAIK, DocumentDB doesn't support geofencing, so it's not really an option to replace mongo. Hopefully CosmosDB does ]

[devrnt]

Sorry to hear you're having problems ... First doubt, does "Azure CosmosDB" open the port 27017, just like mongo?

It's exposed on another port, but this is taken into account when passing the <azure-cosmos-db-host> as mentioned in https://fiware-orion.readthedocs.io/en/master/admin/database_admin/index.html#database-authorization

Personally I have never tried out Orion-LD with anything but mongo. I know some people have successfully tried it with Amazon's DocumentDB.

About Orion-LD / Orion: Orion-LD forked out from Orion back in Sep 2019, and hasn't been refreshed since then (too much overhead). Some day, the idea is that the Orion-LD code is to be merged into the Orion repo, but I don't even have an estimated date for that. The merge is a huge job and it will take quite some time for it to happen.

If you instruct me on how to start a azure cosmosdb, I could try to connect Orion-LD to it.

Sure, it's kind of straightforward if you are familiar with Azure DevOps, this document show how to add a CosmosDB (with MongoDB API) in Azure DevOps: https://docs.microsoft.com/en-us/azure/cosmos-db/create-mongodb-dotnet#create-a-database-account

But, officially, only mongo is supported I'm afraid (at least for now).

I understand, but it seems to be specific for the orion-ld image (the orion image manages to connect to the Azure CosmosDB just fine)

[ AFAIK, DocumentDB doesn't support geofencing, so it's not really an option to replace mongo. Hopefully CosmosDB does ]

[anupamme]

Hello,

I'm also trying to use Azure Cosmos DB Mongo API. I am following the docker installation steps outlined at [1].

In order to use Cosmos DB I'm replacing the mongo db steps and my docker-compose.yml looks like this:

` version: "3.5" services: orion: image: fiware/orion-ld hostname: orion container_name: fiware-orion expose:

• "1026" ports:
• "1026:1026" command: -dbhost "mongo-perf-testing.mongo.cosmos.azure.com:10255" -db "fiver_db" -dbuser "mongo-perf-testing" -dbpwd "" -dbSSL -dbDisableRetryWrites -logLevel DEBUG -t 100 healthcheck: test: curl --fail -s http://orion:1026/version || exit 1

volumes: mongo-db: ~ `

However, when I run docker-compose up I get the following error:

Attaching to fiware-orion fiware-orion | multiple use of option '-db' fiware-orion | fiware-orion | multiple use of option '-db' fiware-orion |

Can you please suggest what is going wrong here?

1. https://github.com/FIWARE/context.Orion-LD/blob/develop/doc/manuals-ld/installation-guide-docker.md

[kzangeli]

multiple use of option '-db' fiware-orion ...

You can use either normal CLI parameters OR environment variables when starting Orion/Orion-LD. They're one and the same, handled by the same library.

E.g. the -db CLI parameter corresponds to the env var ORIONLD_MONGO_DB. It seems like you have both of them set. It's either that or you have -db duplicated.

With command you add CLI params for the start of the broker, but, are there any default CLI params? Also, make sure the env var ORIONLD_MONGO_DB isn't set.

[anupamme]

I could get around this multiple use of option -db by removing -dbSSL -dbDisableRetryWrites from the command so the new command is:

-dbhost "mongo-perf-testing.mongo.cosmos.azure.com:10255" -db "fiver_db" -dbuser "mongo-perf-testing" -dbpwd "<cosmosdb_password>" -logLevel DEBUG -t 100

However, now I run into the following (new) issue:

fiware-orion | time=Friday 06 May 07:32:49 2022.347Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=mongocInit.cpp[49]:mongocInit | msg=Connecting to mongo for the C driver, with URI 'mongodb://<primary_key>@mongo-perf-testing.mongo.cosmos.azure.com:10255'
fiware-orion | time=Friday 06 May 07:33:50 2022.623Z | lvl=ERROR | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=**mongocContextCachePersist.cpp[65]:mongocContextCachePersist | msg=Database Error** (persisting context: **No suitable servers found: `serverSelectionTimeoutMS` expired: [Failed to read 4 bytes: socket error or timeout])**
fiware-orion | time=Friday 06 May 07:33:50 2022.643Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=mongoConnectionPool.cpp[324]:mongoConnectionPoolInit | **msg=Connecting to mongo for the C++ legacy driver**
fiware-orion | time=Friday 06 May 07:33:50 2022.647Z | lvl=DEBUG | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | **op=mongoConnectionPool.cpp[128]:mongoConnect | msg=Connection info: dbName='fiver_db', rplSet='', timeout=4666723172467343.000000**

It connect to the Cosmos DB Mongo API and it seems that C++ legacy drivers may be the issue (as you had pointed out in the call). Can you suggest what are the possible ways forward?

I'd be happy to share the Cosmos DB credentials privately if it helps.

[kzangeli]

So, let me explain. Yes, Orion-LD relies heavily on the old Mongo Legacy driver.

BUT, I am working on it, and I now have 4 (well, very soon - PR on its way) operations that use the new mongoc driver:

• PATCH /entities/{entityId}
• PUT /entities/{entityId}
• POST /entities (only if the broker is started with the CLI option -experimental)
• POST /subscriptions (PR pending + CLI option -experimental)

Apart from that, the entire "Persist Contexts to mongo" is using the new mongoc driver.

In the error you see "mongocContextCachePersist" - mongoc is the prefix for functions using the new driver "mongoc". Functions using the Legacy driver have the prefix mongoCppLegacy (or older functions inherited from Orion - any name).

So, the C++ legacy driver is not the problem. Something else is.

Perhaps we should look at this together. My expertise in the broker and yours in Azure Cosmos DB. I'll send you my Skype handle in an email.

[anupamme]

Hi @kzangeli

Quick question on the mongodb connection url getting created: I'm seeing some issue in the url. The url created is this:

mongodb://:==@mongo-perf-testing.mongo.cosmos.azure.com:10255

where it should be

mongodb://:==@mongo-perf-testing.mongo.cosmos.azure.com:10255

Essentially dbuser is not getting used in creating the connection url. This looks incorrect.

I can share the credentials privately for you to verify what I am saying.

[kzangeli]

ok, I did touch that part recently. Perhaps I broke something. Let's talk!

[oerodriguezn]

Hi, have you fixed this? We are having problems connecting to Azure CosmoDB Mongo

[kzangeli]

I believe so, but, if not, just use the CLI -dbURI and give the complete URI to the broker. It is used "as is".

[rafagm]

Hello, I'm trying to connect to Azure CosmosDB MongoDB using the CLI -dbURI but I can't make it work.

I'm running the following service in a docker-compose:

orion-ld: container_name: orion-ld image: fiware/orion-ld:latest ports:

• "8088:8088" command: -port 8088 -dbURI "mongodb://db_user>:<db_pwd>==@<db_host:10255/?ssl=true&replicaSet=globaldb&retrywrites=false&maxIdleTimeMS=120000&appName=@@"

I get this error

2023-11-14 11:13:16 time=Tuesday 14 Nov 10:13:16 2023.358Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[796]:versionInfo | msg=Version Info: 2023-11-14 11:13:16 time=Tuesday 14 Nov 10:13:16 2023.358Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[797]:versionInfo | msg=----------------------------------------- 2023-11-14 11:13:16 time=Tuesday 14 Nov 10:13:16 2023.358Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[798]:versionInfo | msg=orionld version: post-v1.4.0 2023-11-14 11:13:16 time=Tuesday 14 Nov 10:13:16 2023.358Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[799]:versionInfo | msg=based on orion: 1.15.0-next 2023-11-14 11:13:16 time=Tuesday 14 Nov 10:13:16 2023.358Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[800]:versionInfo | msg=core @context: https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.6.jsonld 2023-11-14 11:13:16 time=Tuesday 14 Nov 10:13:16 2023.359Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[801]:versionInfo | msg=git hash: nogitversion 2023-11-14 11:13:16 time=Tuesday 14 Nov 10:13:16 2023.359Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[802]:versionInfo | msg=build branch:
2023-11-14 11:13:16 time=Tuesday 14 Nov 10:13:16 2023.359Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[803]:versionInfo | msg=compiled by: root 2023-11-14 11:13:16 time=Tuesday 14 Nov 10:13:16 2023.359Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[804]:versionInfo | msg=compiled in:
2023-11-14 11:13:16 time=Tuesday 14 Nov 10:13:16 2023.359Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[805]:versionInfo | msg=----------------------------------------- 2023-11-14 11:13:16 time=Tuesday 14 Nov 10:13:16 2023.363Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=mongocInit.cpp[298]:mongocInit | msg=Connecting to mongo for the C driver (URI: mongodb://db_user>:<db_pwd>@<db_host:10255/?ssl=true&replicaSet=globaldb&retrywrites=false&maxIdleTimeMS=120000&appName=@@&) 2023-11-14 11:13:17 time=Tuesday 14 Nov 10:13:17 2023.794Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=mongoConnectionPool.cpp[313]:mongoConnectionPoolInit | msg=Connecting to mongo for the C++ legacy driver 2023-11-14 11:13:17 time=Tuesday 14 Nov 10:13:17 2023.795Z | lvl=ERROR | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=mongoConnectionPool.cpp[152]:mongoConnect | msg=Database Startup Error (cannot connect to mongo - doing 20 retries with a 500 millisecond interval) 2023-11-14 11:13:27 time=Tuesday 14 Nov 10:13:27 2023.811Z | lvl=FATAL | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=mongoConnectionPool.cpp[320]:mongoConnectionPoolInit | msg=Database Error (unable connect to mongo after a number of retries)

However, if I try the same URI with the option -mongocOnly

It doesn't throw any errors. But, I need to use the legacy driver to have available the /ngsi-ld/v1/ api

2023-11-14 11:06:40 time=Tuesday 14 Nov 10:06:40 2023.320Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[796]:versionInfo | msg=Version Info: 2023-11-14 11:06:40 time=Tuesday 14 Nov 10:06:40 2023.320Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[797]:versionInfo | msg=----------------------------------------- 2023-11-14 11:06:40 time=Tuesday 14 Nov 10:06:40 2023.320Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[798]:versionInfo | msg=orionld version: post-v1.4.0 2023-11-14 11:06:40 time=Tuesday 14 Nov 10:06:40 2023.320Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[799]:versionInfo | msg=based on orion: 1.15.0-next 2023-11-14 11:06:40 time=Tuesday 14 Nov 10:06:40 2023.320Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[800]:versionInfo | msg=core @context: https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.6.jsonld 2023-11-14 11:06:40 time=Tuesday 14 Nov 10:06:40 2023.320Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[801]:versionInfo | msg=git hash: nogitversion 2023-11-14 11:06:40 time=Tuesday 14 Nov 10:06:40 2023.320Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[802]:versionInfo | msg=build branch:
2023-11-14 11:06:40 time=Tuesday 14 Nov 10:06:40 2023.320Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[803]:versionInfo | msg=compiled by: root 2023-11-14 11:06:40 time=Tuesday 14 Nov 10:06:40 2023.320Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[804]:versionInfo | msg=compiled in:
2023-11-14 11:06:40 time=Tuesday 14 Nov 10:06:40 2023.320Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[805]:versionInfo | msg=----------------------------------------- 2023-11-14 11:06:40 time=Tuesday 14 Nov 10:06:40 2023.324Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=mongocInit.cpp[298]:mongocInit | msg=Connecting to mongo for the C driver (URI: mongodb://db_user>:<db_pwd>==@<db_host:10255/?ssl=true&replicaSet=globaldb&retrywrites=false&maxIdleTimeMS=120000&appName=@@&) 2023-11-14 11:06:42 time=Tuesday 14 Nov 10:06:42 2023.091Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[1335]:main | msg=Initialization is Done 2023-11-14 11:06:42 time=Tuesday 14 Nov 10:06:42 2023.091Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[1336]:main | msg= Accepting REST requests on port 8088 (experimental API endpoints are enabled) 2023-11-14 11:06:42 time=Tuesday 14 Nov 10:06:42 2023.091Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[1337]:main | msg= TRoE: Disabled 2023-11-14 11:06:42 time=Tuesday 14 Nov 10:06:42 2023.091Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[1338]:main | msg= Distributed Operation: Disabled 2023-11-14 11:06:42 time=Tuesday 14 Nov 10:06:42 2023.091Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[1339]:main | msg= Health Check: Disabled 2023-11-14 11:06:42 time=Tuesday 14 Nov 10:06:42 2023.091Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[1344]:main | msg= Mongo Server Version: 4.2.0 2023-11-14 11:06:42 time=Tuesday 14 Nov 10:06:42 2023.091Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[1348]:main | msg= Mongo Driver: mongoc driver- ONLY (MongoDB C++ Legacy Driver is DISABLED) 2023-11-14 11:06:42 time=Tuesday 14 Nov 10:06:42 2023.091Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[1349]:main | msg= MongoC Driver Version: 1.22.0

Am I doing anything wrong? Any help is welcome.

[kzangeli]

ok ... the -dbURI looks good. It works with mongoc but not with the C++ legacy driver ... That would happen in a mongodb 6.x (the support of the legacy driver has been removed in v6.x) What's your mongodb version? Wait, is this against CosmosDB ?

I really don't think CosmosDB supports the old legacy driver. You'll have to use "-mongoOnly" with CosmosDB.

[kzangeli]

Sorry, my bad. The name of the option is "-mongocOnly" !!!

FYI, with that option set, Orion-LD uses only the new mongo driver (mongoc) and as a "side effect", all NGSIv2 services are no longer supported. Only NGSI-LD.