search

FIWARE / context.Orion-LD

Context Broker and CEF building block for context data management which supports both the NGSI-LD and the NGSI-v2 APIs
https://www.etsi.org/deliver/etsi_gs/CIM/001_099/009/01.06.01_60/gs_CIM009v010601p.pdf
GNU Affero General Public License v3.0
51 stars 43 forks source link

Why use Postgres for Temporal Data in Orion-LD #1337

Open sjerman opened 1 year ago

sjerman commented 1 year ago

I wondered why postgres was used for TRoE storage. Seems like it would have been simpler to use the TimeSeries collections in MongoDB 5/6. With the changes to newer versions of the client libraries, any thoughts of moving to support Mongo TimeSeries?

kzangeli commented 1 year ago

Yeah, I tend to agree. When we first started to think about supporting TRoE in Orion-LD (wasn't meant to be supported initially), mongodb still didn't have any support for timeseries. Plus, we had certain expertise in Postgres/timescaleDB, so, we picked that one. Personally I'd love to move to mongodb, but, not my call.

Right now we have no plans in that direction, sorry.

sjerman commented 1 year ago

Hi. @kzangeli - Would you be open to a PR to add this support? I don't want to add another DB to our stack :)

It's too long since I did C++, but we have a guy that does work with it. He might need a few pointers about how to add it.

kzangeli commented 1 year ago

I'd be more than happy to accept help!! Just, one important detail.

No C++ !!!

I'm moving away from the bad C++ of Orion into a pure C implementation of Orion-LD. I never liked C++ you see.

I would have to spend quite some time with this guy, but I'm more than happy to do so.

Just one thing I need you to know: this TRoE thing, it's not a small little thing you can implement in a few weeks. We're talking minimum 6 months full-time here. My honest guess would be more like 12 months.

Still interested? ;-)

sjerman commented 1 year ago

Thanks for the reply. I don't think I can bite off that much work. Would that be for both the query component and the push to TRoE?

What about just implementing the push part (looking at the tutorials, is that what has been implemented for timescale?).

kzangeli commented 1 year ago

Sure, that's what Orion-LD does right now, Another component, Mintaka takes care of the queries. But that you already know.

There's more to it though - we have defined 5-6 services to add TRoE data, delete, update etc (in NGSI-LD). That's meant for fixing errors or just populate the TRoE DB without going "the normal" way.

This is not hard though, the query part, and especially the federation part of the queries is the hard part.

Put your guy in contact with me and I'll teach him all he needs to know. Count a few weeks full-time for education before he can start

kzangeli commented 1 year ago

And yes, 6-12 months, that was for "the works". It took me about one month to implement the push to postgres, but, I had to restructure quite a few things as I destroyed my incoming trees while using them (no one else needed them after me, so I made the broker a bit faster that way. Had to change all that as I now had a use for the data non-destroyed. But, of course, I know the source code by heart as I've implemented it all. Not gonna be that easy for your guy. I will help him continuously though. Also, of those 5-6 extra API endpoints, I only implemented two (still haven't implemented the missing ones). They are "nice to have". We'd start with the "real thing", which is "simply" to push every update to the current state to the historical database.

I will need quite some time to show him how the broker works, especially he'll need to become a master of the tree structure that is output from my JSON parser, as I use that tree and those nodes pretty much everywhere (I avoid to make unnecessary copies like that).

Looking forward to it :)