Improve Graphql endpoint to support dynamic schemas

[cesarvarela]

With the advent of #2047, we must perform complex queries on the classifications collection and support dynamic graphQL schemas built from the definitions found in the taxas collection.

Current Architecture

Classifications collection schema

Update Option 1

Update Option 2

[kepae]

Per offline (online?) discussion, the need for dynamic graphQL queries results from:

• The serialized JSON values (value_json) in the classifications documents don't come with a "type" sibling.
• Any such type information has to come from the taxa document the classification derives from, using either the mongto_type or display_type fields on the field_list of the taxa.
• This means properly displaying the "types" of fields for queries is very dynamic, and stressfully so. This is essentially writing an interpreter.

The questions are:

• where should this "interpreter" live?
• how can it be tested and maintained?

Option 1

• might be faster and simpler for today's particular feature, and it would only serve that one feature.
• It seems that we could implement the custom resolver such that it handles each classification record specifically and sanitizes it, and essentially "hard-code" the interpretation of the types? is that correct @cesarvarela ? https://www.mongodb.com/docs/atlas/app-services/graphql/custom-resolvers/#define-the-resolver-function
• Code lives in Atlas App Services, not GitHub. This makes it more difficult to test and maintain.

Option 2

• would be expanding commitment to maintaining a public graphql endpoint using Netlify functions.
• The main advantage, as I see it, is that the code would live in our repo and would be easier to test.

Option 3

• Create a dedicated API server. This could benefit other projects going forward, such as Risk Checklisting (@lmcnulty)
• AIID now depends on a server other than MongoDB Atlas. AIID can still be run "serverless" to a degree, and without a separate API server, but also without this feature or others that we are discussing.

[cesarvarela]

Code lives in Atlas App Services, not GitHub. This makes it more difficult to test and maintain.

Although code is run there, it is being version controlled: https://github.com/responsible-ai-collaborative/aiid/tree/master/site/realm and deployed via https://github.com/responsible-ai-collaborative/aiid/blob/master/.github/workflows/realm-production.yml

We also have some unit tests for them: https://github.com/responsible-ai-collaborative/aiid/tree/master/site/gatsby-site/cypress/e2e/unit/functions

@kepae

[kepae]

Good point, thanks for the definitive correction on that -- we already maintain, test, and push code to realm/app services here in the repo.

[cesarvarela]

From yesterday's discussion:

Conclusions

• We have to run queries against a dataset that will get increasingly bigger
• We are storing classifications in MongoDB so we have no other option (at least without Rube Goldberging this) than to use Mongo's query engine
• To properly use the query engine, we have to store classification attributes with the correct data type instead of using the JSON.stringify value

Actions

• Add a value field to classification attributes by creating a migration that stores the JSON.parsed value from the value_json attribute
• Create a custom resolver that takes care of classification upserts so we always keep a value field up to date
• Create a custom resolver that receives a MongoDB query expression, forwards it to the query engine (collection.find), and returns the results

Once this is done, we can go back to #2047