Split api-server "database" pkg using repositories pattern

[Pitasi]

api-server has a common package called database: https://github.com/EmerisHQ/demeris-api-server/tree/main/api/database.

The exported database.Database is used across every http handler, some examples:

• https://github.com/EmerisHQ/demeris-api-server/blob/339d31e63b526cf6eb995f362b3748e29e636e0a/api/chains/chains.go#L60
• https://github.com/EmerisHQ/demeris-api-server/blob/339d31e63b526cf6eb995f362b3748e29e636e0a/api/chains/chains.go#L159
• https://github.com/EmerisHQ/demeris-api-server/blob/339d31e63b526cf6eb995f362b3748e29e636e0a/api/account/api.go#L184

In the optic of not sharing a single database someday, I keep this questions in mind:

• what if instead of accessing the tracelistener db we need to make an API call?
• what if cns changes database and we cannot mix tracelistener and cns in our queries?

That's why I think we should make it easier for us to change our "storage layer" implementation, and better separate it from the business logic (= the handlers). This would also allow us to write real unit tests that don't depend on any database.

DoD

• [ ] identify which "repositories" can be implemented (at the very least cns and tracelistener, but maybe we can go even more granular)
• [ ] make http handlers take the repositories interfaces as a dependency, instead of a real database.Database

[tbruyelle]

Currently, this shared database pattern allows us to join data easily. If we go for splitting the data into separate databases, that means we first have to query the different services to gather the required data, and then build our sql query. So after a full rewrite of the sql queries, we'll necessarily have a bad impact of performances, because of the network latency we didn't have before.

I'm OK with the idea that a shared database isn't ideal in a micro-service architecture, because it basically creates a big SPOF, and makes migrations more difficult. But then I have an other question, do we really need a micro-service architecture ? If for instance we consider cns and api-server, I'm not sure to get why they deserve their own service. Maybe they could be merged ?

In addition to the performance, I fear that once we split up the databases, it will slow down development, mainly because writing a single sql query is faster than fetching data in various services.

So conclusion I think we should consider the options carefully. WDYT @Pitasi ?

[Pitasi]

Ok let's put this ticket on hold, it actually is part of my "internal" roadmap for achieving a certain end state, but I understand we need to better discuss this end state with the team 😄