waterline-query-docs: Non Database environments

sailsbot commented 7 years ago

From @gaurav21r on February 18, 2016 6:15

@particlebanana @mikermcneil From my experience of using Watreline Standalone for quite some time now (and a little bit of loopback ORM), I think this time around a larger focus should be on non Database Environments, like REST API, SOAP Web Services etc.

Waterline is quite fantastic in delivering what Mike had written in the docs:

In other words, Waterline is not just an ORM for your database. It is a purpose-agnostic, open standard and toolset for integrating with all kinds of RESTful services, datasources, and devices, whether it's LDAP, Neo4J, or a lamp.

If the current set officially maintained and a few well maintained community adapters (MySQL, PostgreSQL, MongoDB, MS SQL Server, Oracle, Redis) are migrated to this new interface, we would've already got around 98% of Databases developers use.

I think the real question is how can the query language be architected in a way to support say Github API, Slack Realtime API, All Rest APIs in general? How about Elasticsearch?

Copied from original issue: sailshq/waterline-query-docs#2

sailsbot commented 7 years ago

From @mikermcneil on February 20, 2016 15:8

@gaurav21r thanks for the note-- great points and questions.

Waterline Drivers

this time around

The purpose of the project documented in this repo is to provide a lower-level (but tightly specced) abstraction that sits between adapters and the database. To begin with, we're going to call out to these drivers from within existing adapters themselves-- the goal being to normalize and solidify things-- no major changes in usage (except case sensitivity changes in WL 0.12). Longer term, the goal of drivers is to empower adapter maintainers with custom userland<->adapter communication, enabling the development of db-specific features like transactions, sharding, replicas, etc. For more about the plan/progress for WL drivers so far, also check this out: https://github.com/node-machine/waterline-driver-interface#waterline-driver-interface (just updated in the wee hours of this morning-- planning on migrating a lot of the docs there into this repo over time)

Other APIs

I think the real question is how can the query language be architected in a way to support say Github API, Slack Realtime API, All Rest APIs in general? How about Elasticsearch?

My experience since originally building Waterline has been that 3rd party APIs like Slack/GitHub/etc rarely implement a true RESTful interface in a way that is comparable with a database. There are two main issues:

no describe() -- with the notable exception of SOAP/WSDL, most miscellaneous APIs don't expose their documentation information in a declarative manner. So using them in the standard usage of Waterline is a no-go. The first time this really occurred to me was here, a little over 2 years ago when I was working on a Twitter adapter (i.e. if I have to call Twitter.findTweets() and Twitter.findLocations(), how is this any different from just exposing the endpoints?) This is a natural segway to the next issue:
The exposed endpoints in APIs inevitably end up being use-case-specific. For example, in GitHub's case, despite being one of the purest REST APIs I know, AND/OR filtering varies even between different keys in the same criteria in the same endpoint (see https://github.com/mikermcneil/machinepack-github/blob/master/machines/search-issues.js for an example of what I mean)

What I've realized now is that this specificity is actually good! APIs that aren't databases should work this way-- it's so much easier to figure out what endpoint you need to hit (assuming there's good documentation) than it is to learn GitHub's data model (plus GitHub might not want to allow you to do CRUD on some of their models anyway). That said, I still wholeheartedly believe it's possible to make these sorts of APIs usable in a declarative manner-- I've just learned the hard way now that CRUD is not the right way to do it in the general case... instead, we need a more flexible abstraction. This, along with our team's other struggles against the same chains of semantic relevancy in other plugin systems (generators/Skipper/view engines,session stores, socket.io adapters, etc) was the genesis of the node-machine project.

This (the WL driver API) is the first step towards unifying the Waterline adapter interface with the machine specification-- starting with the lowest-level operations (see interface layers for a possible approach for standardizing higher-level functionality).

Which brings us to your other point re: datasources:

CRUD APIs

If the current set officially maintained and a few well maintained community adapters (MySQL, PostgreSQL, MongoDB, MS SQL Server, Oracle, Redis) are migrated to this new interface, we would've already got around 98% of Databases developers use.

Completely with you here! True CRUD interfaces work great w/ the ORM, and they're a natural fit:

MySQL
PostgreSQL
MongoDB
MS SQL Server
Oracle
sqlite3
Redis
DB2
and so forth-- plus LDAP systems, and the occasional rare API that is ~90-99% purely RESTful, allows for flexible filtering, and also allows CRUD on resources itself (i.e. basically functioning as a database). These APIs are useful in an ORM-y way and are ideal adapters.

Roadmap & Vision

So to summarize:

The current plan

machinepack <=> WL driver
                  |-----> WL adapter  -------<>--- WL core
                                                      |--------> Sails core

_(much nicer diagram here)_

Long term

The long-term goal here is to provide the same level of standardization documented at the driver level--all the way up the stack. But this time, instead of trying to cast everything to look like CRUD, it is built on small, functional building blocks that are flexible enough to represent literally anything.

For example, a logical next step would be:

machinepack <=> WL driver <=> WL adapter  -------<>--- WL core
                                                         |--------> Sails core

As well as:

machinepack <=> Skipper adapter   -------<>--- Skipper core
                                                    |--------> Sails core

Meanwhile, any API that doesn't fit any existing higher level interface can be modeled, documented, tested, and error-checked as a machinepack. And since the building blocks are there, anyone can structure their own higher level interface, rather than waiting on us: all it takes is uniformity. i.e. if two machinepacks have a subset of the same machines, each one with the same inputs/exits, then those two machinepacks share the same interface by definition.

P.S. (sorry for the long ramble! I get excited about this stuff)

sailsbot commented 7 years ago

From @mikermcneil on March 2, 2016 13:46

i.e. if two machinepacks have a subset of the same machines, each one with the same inputs/exits, then those two machinepacks share the same interface by definition.

I should clarify that only the semantic properties of their inputs/exits have to be compatible (i.e. the moreInfoUrl doesn't matter). Also, it's worth pointing out that I didn't explain that quite right: "compatible" doesn't mean "the same". For example:

Since numbers are JSON-compatible values, the following input definition would be considered compatible with the other input definition below it (i.e. if it imagined that as an interface). However, the opposite is not true (since JSON-compatible values are not necessarily numbers).

// in `GroceryStore.buyWhippedCream()`
inputs: {
  cans: {
    description: 'The number of cans to buy.',
    example: 4000
  }
}

// in `DarkAlley.buyWhippedCream()`
inputs: {
  cans: {
    description:
      'An ASCII-art string or artful dictionary describing your interpretation '+
      'of the burning cans we saw together that fateful night.',
    example: '*'
  }
}

So because the relation is not reflexive, we can't assume that these two machines are actually instances of a previously unrecognized shared interface (which is good, because they are totally not.)

sailsbot commented 7 years ago

Originally

https://github.com/sailshq/waterline-query-docs

Refs

@mikermcneil mikermcneil referenced this issue in balderdashy/waterline on Mar 9, 2016 Open Deep populate #1052 @mikermcneil mikermcneil referenced this issue in balderdashy/waterline on Apr 20, 2016 Closed Thoughts about waterline #1165 @atiertant atiertant referenced this issue in Atlantis-Software/offshore on Apr 25, 2016 Open continuation of deep populate #24

sailsbot commented 7 years ago

@sailsbot: Hello, I'm a repo bot-- nice to meet you!

It has been 30 days since there have been any updates or new comments on this page. If this issue has been resolved, feel free to disregard the rest of this message and simply close the issue if possible. On the other hand, if you are still waiting on a patch, please post a comment to keep the thread alive (with any new information you can provide).

If no further activity occurs on this thread within the next 3 days, the issue will automatically be closed.

Thanks so much for your help!

balderdashy / sails