search

FIWARE / context.Orion-LD

Context Broker and CEF building block for context data management which supports both the NGSI-LD and the NGSI-v2 APIs
https://www.etsi.org/deliver/etsi_gs/CIM/001_099/009/01.06.01_60/gs_CIM009v010601p.pdf
GNU Affero General Public License v3.0
51 stars 43 forks source link

Are there plans to get rid of MongoDB Legacy C++ Driver? #1062

Open rschwebel opened 2 years ago

rschwebel commented 2 years ago

The latest release of the legacy drivers is from 2018 (https://github.com/mongodb/mongo-cxx-driver/releases/tag/legacy-1.1.3), and the git repo says it should not be used any more, as it is out of support, since 6 years. Do you have plans to refactor the orion-ld codebase to get rid of it?

kzangeli commented 2 years ago

Yes, definitely. I'm working on it, It's gonna take quite some time though. A year easily, to get rid of the old Legacy driver completely. Probably more, depends on the priorities of the FIWARE Foundation. All new services are implemented with the new mongoc driver.

kzangeli commented 2 years ago

Update on the ongoing work to replace the old Legacy Driver

Two new services have been implemented lately (using mongoc, of course):

Broker in-built features that now use mongoc:

Existing services that now optionally can use mongoc instead of the Legacy driver:

These three "mongoc optional use" you turn on by starting the broker with a new hidden CLI option - -experimental.

I keep working on this and hope to have it all replaced by ... Christmas? It's hard to predict the future ...

rschwebel commented 2 years ago

Thanks for the update!

rschwebel commented 2 years ago

Any news regarding this?

kzangeli commented 2 years ago

Oh yes. Everything except Registrations (that still aren't really supported anyway) now uses mongoc and not the legacy driver, if the broker is started with -experimental (or with the env var ORIONLD_EXPERIMENTAL=TRUE).

If you want to go one step further and completely disable the old legacy driver, use -mongocOnly (env var ORIONLD_MONGOCONLY=TRUE). If you do that, then all old NGSIv2 will give you a 501 error response, and the NGSI-LD registrations as well. I'm working on Registrations and Forwarding right now - gonna take a while ... (minimum 6 months)

Drbvr commented 2 years ago

Definitely looking forward to this! Any chance the mongoc-functionality will become available in a stable release sooner than you have been able to fix it all? The already available services promise to be a great help, but I'd rather not use them in experimental mode.

kzangeli commented 2 years ago

Yeah, well, I have a few users "brave enough" to use the "-experimental" mode. It sounds more drastic than it actually is.

The more users I have, the faster I will feel ok with making the mongoc stuff the default mode. I'd say, turn it on, and if you run into troubles, turn it back off.

FYI, In a few weeks the regs will be implemented using mongoc as well.

Drbvr commented 2 years ago

Ok, let's give it a try. Should we use 'latest' release for this? I don't think this is in 1.1.1 yet, right?

kzangeli commented 2 years ago

Don't ever use "latest", as a tag. Pick the latest tag you can find, that's fine. And 1.1.1 comes from 1.1.0 which is quite old, so, no, use the newest tag you can find, and stick to it until you need to update it, depending on new functionality or bug fixes.

rschwebel commented 2 years ago

Update on the ongoing work to replace the old Legacy Driver

Two new services have been implemented lately (using mongoc, of course):

  • PATCH /entities/{entityId}
  • PATCH /entities/{entityId}

Broker in-built features that now use mongoc:

  • Tenants (the broker maintains a list of the tenants of the system)
  • Geo Indexes
  • Work has started on the subscription cache, will take a few weeks more before it's finished

Existing services that now optionally can use mongoc instead of the Legacy driver:

  • POST /entities
  • GET /subscriptions/{subId} (it gets the subscription from the cache instead of from the DB, so, no DB driver needed)
  • GET /entities/{entityId}

These three "mongoc optional use" you turn on by starting the broker with a new hidden CLI option - -experimental.

I keep working on this and hope to have it all replaced by ... Christmas? It's hard to predict the future ...

Unfortunately, as far as I see, this "-experimental" method doesn't remove the compile time dependency on the 2016 deprecated mongo-cxx-driver-1.1, right?

kzangeli commented 2 years ago

Unfortunately, as far as I see, this "-experimental" method doesn't remove the compile time dependency on the 2016 deprecated mongo-cxx-driver-1.1, right?

Yes, that's right. And, that part may not be removed in a long time. Orion-LD still supports NGSIv2 but it needs the legacy driver for that. To overcome that obstacle isn't easy. You'd need to "mock" the legacy driver

rschwebel commented 2 years ago

I'm not sure if I understand you correctly.

mongo-cxx-driver-1.1 has been deprecated by the maintainers back in 2016, six years ago. That means that the code won't get any support any more; it doesn't compile on modern distributions (it contains an old Python 2 based build system which is deprecated as well), and of course it won't get any security updates any more as well. The advise of the mongo-cxx-driver maintainers is to migrate any user towards version 3.x, which is alive and supported.

Could you elaborate the the long term maintenance strategy of the fiware project? Basing an infrastructure core component on unsupported, deprecated software doesn't sound like something to build your smart city on. Shouldn't there be a strategy to refactor the codebase and migrate to v3 as fast as possible?

kzangeli commented 2 years ago

Yeah, I know all that, perfectly. The thing is, Orion-LD is about NGSI-LD, the NGSIv2 part was inherited from Orion (back in 2018). Best effort for the NGSIv2 part. Redoing the NGSIv2 part to support mongoc isn't even part of the roadmap. Some day, once the NGSI-LD part is free from dependencies of the old code, I can refresh the brqanch and get the new source code from Orion, free from the legacy driver. However, while that is highly interesting, it's not at the top on the backlog. NGSI.-LD Forwarding is. And a few other interesting NGSI-LD features.

If you want to run NGSIv2, we strongly advise you to use Orion, not Orion-LD. If you plan to migrate from NGSIv2 to NGSi-LD, then Orion-LD might be the better choice (as long as you don't depend on a feature post Aug 2018).

sjerman commented 1 year ago

Hi? Orion have updated the mongo driver. It now works with newer versions of mongo. How difficult is the merge do you think?

At present, it seems like:

I am just starting a development cycle and I should be using NGSI-LD but there is no way that I can run two versions of mongo.

kzangeli commented 1 year ago

I've been working quite some time on replqacing the old driver for the new. Now most of it is working but it is not yet enough tested so the old legacy driver is still default and you have to use an option to use the new one.

You don't need to run two different versions of mongo to support both mongo drivers. The database server is one thing, the client library is another thing. The server supports both drivers and another 50 drivers I imagine.

"lots of things break" ...

What problems have you seen? That is very interesting but I'd need you to describe the errors for me to be able to fix the problem. Preferribly as github issues. Very important for me to get this info and be able to fix whatever issues you've found!

_FYI: I run Orion-LD with ORIONLD_MONGOCONLY (only new driver) or ORIONLDEXPERMENTAL (hybrid mode - both old and new driver is used, but the new driver takes precedence over the old) and I have ~850 functests covering most of the functionality. It's impossible to cover it all in such a generic implementation but yeah, most of it definitely works just fine.

sjerman commented 1 year ago

Apologies for the vague language. I could have phrased it better :)

For me the key feature that needs to work is subscriptions. Do you believe that works?

I ran into a few issues using the existing tutorials.IoT-Agent .. I'll document them.

kzangeli commented 1 year ago

I don't know of any problems with subscriptions and mongoc (new driver). If I did ... I'd fix them!!! :) Would be great if you could tell me exactly what's wrong and I'll get right into that ! Thanks for reporting!

sjerman commented 1 year ago

OK thanks. I am using MongoDB6...

So :) - maybe I'm getting there. Running with experimental =true and MONGOCONLY=true the server starts...

I can run:

Steves-MacBook-Pro:iot-agent sjerman$ curl -sk  'https://192.168.1.222/ngsi-ld/ex/v1/version'
{
  "Orion-LD version": "1.1.2",
  "based on orion": "1.15.0-next",
  "kbase version": "0.8",
  "kalloc version": "0.8",
  "khash version": "0.8",
  "kjson version": "0.8.2",
  "microhttpd version": "0.9.75-0",
  "rapidjson version": "1.0.2",
  "libcurl version": "7.61.1",
  "libuuid version": "UNKNOWN",
  "mongocpp version": "1.1.3",
  "mongoc version": "1.22.0",
  "bson version": "1.22.0",
  "mongodb server version": "",
  "boost version": "1_66",
  "openssl version": "OpenSSL 1.1.1k  FIPS 25 Mar 2021",
  "branch": "",
  "cached subscriptions": 0,
  "Next File Descriptor": 20
}

And the following works:

 POST 'https://192.168.1.222/ngsi-ld/v1/entityOperations/upsert'
GET 'https://192.168.1.222/ngsi-ld/v1/entities/urn:ngsi-ld:Person:person004'

This doesn't:

GET 'https://192.168.1.222/ngsi-ld/v1/entities?type=Person'
{"type":"https://uri.etsi.org/ngsi-ld/errors/OperationNotSupported","title":"Not Implemented","detail":"this request does not support the new mongoc driver"}

I also changed the IoT-Agent demo to use experimental=true/MONGOCONLY=true and it has issues:

2023-03-16T12:20:36.060Z tutorial:command-listener sendCommand: tractor001 start
2023-03-16T12:20:36.061Z tutorial:command-listener {"method":"PATCH","url":"http://orion:1026/ngsi-ld/v1/entities/urn:ngsi-ld:Device:tractor001/attrs/start","headers":{"Content-Type":"application/json","NGSILD-Tenant":"openiot","NGSILD-Path":"/","fiware-service":"openiot","fiware-servicepath":"/","Link":"<http://context/ngsi-context.jsonld>; rel=\"http://www.w3.org/ns/json-ld#context\"; type=\"application/ld+json\""},"body":{"type":"Property","value":" "},"json":true}
GET /js/water.txt 304 13.771 ms - -
2023-03-16T12:20:36.078Z tutorial:command-listener {
  type: 'https://uri.etsi.org/ngsi-ld/errors/OperationNotSupported',
  title: 'Not Implemented',
  detail: 'this request does not support the new mongoc driver'
}
sjerman commented 1 year ago

Also (from demo):

POST /ngsi-ld/v1/entityOperations/upsert
{"type":"https://uri.etsi.org/ngsi-ld/errors/OperationNotSupported","title":"Not Implemented","detail":"this request does not support the new mongoc driver"}
rschwebel commented 1 year ago

_FYI: I run Orion-LD with ORIONLD_MONGOCONLY (only new driver)

I find it a bit unfortunate that this is a runtime option. So while I want to get rid of the old driver, I had to package it for ptxdist anyway, including all dependencies.

If you had too much spare time, a variant that makes it possible to eliminate the old driver on compile time would be much appreciated :^)

(Otherwhise I'll just ignore it, as I've now ported the dependencies anyway)

kzangeli commented 1 year ago

Yeah, I'd love to throw away all the old stuff ... I can't though, as the old NGSIv2 code is all using the legacy driver, and my boss REALLY wants Orion-LD to support both NGSI-LD and NGSIv2. Some day I'll fix the problem (by merging with the new code of Orion), but ... that's a difficult task and I have too much on my plate right now ... this year and next are fully booked. So, this will happen, but it's gonna take a loooooong time before it's done.

As you might have understood by now, by setting the ORIONLD_MONGOCONLY env var to true, all NGSIv2 requests fail.

Once the new "mongoc code" is thoroughly tested, I'll make "-experimental" the default and you won't need to set any env var to use mongoc. But, that's not the same as mongoc-only. The legacy driver is still in use for the older NGSIv2 requests.

rschwebel commented 11 months ago

Hi Ken,

Any news regarding plans to get rid of the MongoDB Legacy C++ Driver?

kzangeli commented 11 months ago

Any news regarding plans to get rid of the MongoDB Legacy C++ Driver?

Yeah, some news I do have. But first, my CTO doesn't want this. He wants Orion-LD to serve both NGSIv2 and NGSI-LD for a smoother transition to NGSI-LD.

That said, some time not too far in time I'm going to have to do something about this. Some companies have problems with the licensing of Orion (e.g. Amazon WS => they can't use the broker in Garnet). So, reasons to do this are piling up. I took a quick look at how much work it would be to completely get rid of all the old source code from Orion, including a complete rewrite of 4-5 libraries, and I came up with some 4 months full.time ... So, even if I started today, it would probably take me close to a year or so to have this done. It's not just stripping out all the NGSIv2 endpoints, it's to be 100% free of Orion/NGSIv2 source code. Also, I'd need two version of the broker, one NGSI-LD only and one with both APIs, to avoid getting sacked :)

In January we start a new European project where I believe I'll get another very good reason for an only-LD Orion-LD.

So, hopefully I will start on this soon.