Support for non-SQL backends

[m13t]

Is your feature request related to a problem? Please describe.

Really like the software that’s being built, but it would be nice if the database abstraction supported more than just SQL via Pop. It would be great if the was a more friendly interface so that we can easier write backends for things like DynamoDB or MongoDB for example.

Describe the solution you'd like

To be able to possibly write plugins for the DBA layer that conform to a generic interface for communicating with a data store. This way the community can drive the databases supported by the application and better integrate with their existing stack.

Describe alternatives you've considered

None so far.

Additional context

N/A

[aeneasr]

The interface is well defined. There are tons and tons of database-agnostic acceptance tests (for example for identities).

Writing storage adapters is really hard, especially for Key/Value stores like DynamoDB. I generally advise against that because the effort to keep up with updates and changes is immense - unless you have a dedicated FT employee with deep knowledge of the domain and a budget of at least a week per month just for this.

If you're scared about scalability, don't be. You can easily scale to GitHub levels with classical RDBMS (GitHub still uses MySQL heavily) and we have many people running our stacks against billions of API requests with SQL.

Given that this will not be addressed by us, and that it's unrealistic that this will be implemented by a third party due to the early stage of the project, I'll be closing this. If you do intend on contributing significantly and over a long period of time to this effort, feel free to comment and we'll reopen.

[m13t]

Thanks for the quick response. It wasn’t about issues with traditional RDBMS at all, it was more about allowing the freedom of choice for implementers rather than imposing a limited choice. I’m not suggesting that Ory should implement these themselves, but maybe an easier way for adopters to write additional Persisters and have them loaded without maintaining a fork with their code in.

[aeneasr]

We've allowed this in the past for ORY Hydra and our observation was that people never updated their software versions because it takes a lot of time to implement schema changes. This means that there are several instances running with published CVEs just because they chose to implement this on their own :) That's why we're generally very cautious with implementing features that make this easier :D

[m13t]

I appreciate where you are coming from with that. We’re very heavy users of Mongo and our current identity solution (10-15m req/day) is sat on top of that, amongst various other elements such as reporting etc. Whilst we’re not in a position to move at the moment, I’m keen to track the maturity of this project 😀

[aeneasr]

Yeah I totally understand the point around being able to not support specific stacks very well while not so much others. If you ever weigh in on adopting ORY Kratos or related products try to keep in mind that investing the time to write and maintain adapters is much higher than operating another piece of complexity (e.g. PostgreSQL), which in itself is already quite time intense.

At some point we will be providing additional databases, especially as we look to scale out our soon to be announced cloud service. But first we have to run into real operation / scalability challenges to justify writing another database adapter. Generally I think that distributed K/V stores are required for high-scale multi-region. We're not sure yet if we ever commit to e.g. AWS DynamoDB as there are alternatives like Cassandra, CockroachDB, and others.

[m13t]

I suppose it’s that fine line of creating a reusable set of components that gives the flexibility for most use cases but not giving people too much rope to hang themselves with. As you say, maintaining more adapters is time intensive so forking and implementing one yourself is risky business.

Appreciate the chat and comments 👍🏼

[NurdinDev]

Interesting discussion, I reach this thread because I'm searching if there is a way to use Kratos with Mongo, and it seems not, but I'm curious to know what is the side effects if I used both databases? (Postgress for ORY) and (Mongo for my Database) and I'll store the userID also in my database like if I'm used Auth0, all the users are stored in their database.

Appreciate any feedback whether if it's a good or a bad approach to use ORY services in this way.

[aeneasr]

It's a perfectly fine approach! The userID in Ory Kratos is stable (never changes)

[forresthopkinsa]

Just to give my 2¢: NoSQL databases like DynamoDB make way more sense for serverless applications, at least due to their pricing model. A small project can't afford to keep a relational DB instance constantly running.

[aeneasr]

If you can't afford a SQL server, a viable option would be to run Ory Kratos with SQLite and store it on your server's disk. I would not recommend this for any system with a lot of traffic, and it needs constant back ups of the SQLite database. Vendor locked-in databases such as DynamoDB do not offer any benefit over SQLite for Ory Kratos (except tons of work to get this done 😓 )

[forresthopkinsa]

SQLite is not a terrible recommendation for a small project. It does, however, impose some serious scaling limitations.

Say you're running an application completely in Lambdas using Dynamo. You inherit practically infinite scalability (assuming perfect code), while also costing nothing until you reach a high amount of adoption:

• Lambda gives 1M free requests and 3.2M free seconds of compute per month
• Dynamo gives 25 GB free storage and 200M free requests per month

The most Lambda-friendly relational DB would be Aurora Serverless, but that can't practically scale to zero due to really long cold starts. That pretty much just leaves Dynamo.

I completely sympathize with your argument about vendor lock-in and the required effort. I don't have any counterargument for those things. I understand this isn't going to happen any time soon. I just want to put it out there that the locked-in option is very attractive, cost-wise, for small but scalable projects, and in my case it disqualifies Kratos despite how much I like it.