raymondfeng
commented
7 years ago The current version of loopback-datasource-juggler is overloaded with many responsibilities, such as:
- A type system for model definitions
- A JSON-based language for typical query and mutation
- Validation
- Relation (may go beyond models backed by the same database)
- Navigation and inclusion of related data following relations
- Bind models to datasources to mix in CRUD
- Map CRUD operations to connectors (which in turn implements the translation from our JSON based DSL to SQL or NoSQL commands
- Proxy for other forms of APIs, such as REST/Swagger, SOAP WS, or gRPC
- Intent-based operation hooks
When we discuss what should be or should not be supported out of box in LB next, these different perspectives should be separated to avoid confusions.
We have planned for refactoring the juggler into separate modules. The @loopback/repository package for loopback-next is also a starting point for the effort.
bajtos
jannyHou
kjdelisle
ExTheSea
virkt25
I've been thinking about the persistence story in LoopBack over the last few months and have been wondering if the juggler is the right approach for the framework.
Warning: Opinions Ahead -- Please don your 3D glasses now.
Background
In the current iteration of loopback and loopback-datasource-juggler, we define relations between Models to provide much of the interoperability that users have come to rely on for making their REST APIs. These relations are supported by a Domain-Specific Language (DSL) that leverages that metadata to generate queries against datasources.
The Everything ORM
In many cases, our mapping of the DSL has required us to make an ORM-like set of facilities that generate query strings (in the case of SQL) or translate our DSL into one understood by the target database/database drivers (MongoDB, as an example). Support for this idea falls short of what many in the development community would expect for each of the individual use cases; SQL queries are often limited to primitive, inefficient operations and some NoSQL query objects do not accurately represent their original intention when translated accordingly. In many cases, users must fall back on basic methods that do not take advantage of the metadata our relations and DSL represent.
In each of these cases, whether or not it was our intent, we have implied to the community that we would shoulder the burden of providing a reasonably-complete and efficient set of query-generation tools to work hand-in-hand with our DSL to provide fine-grained control to various datasources.
In my opinion, the very idea of having our own DSL gives this impression; what purpose does it serve otherwise?
The Extensibility Problem (Routing around the Damage)
In the case of loopback-next, our extreme extensibility is a massive advantage, but it comes with the a significant "downside"; members of the community will simply avoid the Juggler if it doesn't meet their needs, and those that find it sufficient will keep using it.
"Great! Why's that a problem?"
Our Choice Informs Our Design
The decision of which approach we consider to be the best practice will have a direct effect on the design of the other core components of the framework. As an example, one of the other things that loopback 3.x gives you with its relations is automatic generation of REST APIs based on your models. "Foo hasMany Bar"? Great! We'll generate all of the
/foos/{id}/barsroutes for you!If we decide that there will be no Juggler for 4.x, then that probably means no relations, too. That doesn't mean we won't have a story for generating those API definitions in some other way, but it does give you two very different approaches.
"We can keep relations without using the DSL or the juggler for persistence!", you might say. But would you want to create all of the relationships between your Models in a declarative format just to generate the shape of your REST APIs? Typing
/foos/{id}/barsinto a Swagger editor isn't any more difficult than making those relations. The combined value of getting that for free as well as being able to leverage that metadata to generate queries against datasources is what makes the sales pitch for the idea of having those relations to begin with.DSL? I Prefer Cable, Thanks
Convincing community developers to invest great effort into creating these middleware layers that translate our DSL into commands for various database drivers is a tough sell when ready-made ORMs can be liberally sprinkled into your code in loopback-next.
Many of them come with their own Model and relationship engines (sequelize being the example everyone is tired of me bringing up whenever we discuss this), and since they tend to specialize in their chosen domain, they're often very efficient, well-written ORMs that do one (or a handful) of things extremely well.
Competing with this would require us to once again take on the burden of building all of these connectors ourselves, only this time, we'd have to make damn sure that we're making solid-quality SQL statements, and properly translating our MongoDB queries because developers will simply swap to using the underlying drivers if they're not already in too deep. Combine this with the fact that we just don't have the resources to build all of this in a timely fashion and the value-for-money of this proposition suddenly seems thin at best.
What Other Circus Acts Are We Good At?
The juggler is definitely something that distinguishes us from other frameworks that are value-adds on Express.js, Koa.js and so on. I find myself wondering what other killer features we could provide that existing frameworks don't. It might be that what we need to differentiate ourselves in a meaningful way doesn't even have to be a particular feature; if our framework and tooling do a better, faster job of getting people from idea to API, and gives you easy flexibility to add all sorts of awesome components, wouldn't that be our killer feature?
Trade Chainsaws for Bowling Pins
It might be that someone who's much more creative/intelligent than me knows of a way we can have our cake and eat it too; if a design for the new juggler provides a low-effort API to implement that bridges the gap between driver and DSL, then I'm definitely on board with the idea.
You may now remove your 3D glasses.
I definitely want to hear everyone's thoughts on this, both the validity/invalidity of my concerns, as well as approaches we could take to solve this problem. Thanks for reading!