Async Concept API

[alexjpwalker]

Problem to Solve

Chained Concept API calls are very inefficient. Consider the example of Workbase, which takes in an arbitrary Graql query, and for every answer, loads its attribute ownerships and its linked relations.

Currently, each one of these calls must be executed serially over the network, resulting in a colossal number of network roundtrips.

Proposed Solution

We should introduce an async Concept API into client-java and client-python. (client-nodejs already has one, so admittedly Workbase is a poor example)

Additional Information

It's also possible that an async Concept API is not in fact the right solution here. Maybe we should look at the common use cases (eg: loading connected attributes and connected relations) and introduce them as query options, or some other kind of query extension.

[haikalpribadi]

Loading all the relations of a concept could pull a lot of data with one concept, @alexjpwalker. I'm not sure this is an efficient behaviour for concept API.

[haikalpribadi]

Can you elaborate further the problematic scenario that inspired this feature request?

[alexjpwalker]

Can you elaborate further the problematic scenario that inspired this feature request?

@haikalpribadi In Workbase, the user passes in an arbitrary TypeQL query.

Say they do: match $x isa thing;

Then Workbase iterates over all the answers and loads their edges in the graph (sub, rel, has edges) so that the user can visualise the links between vertices.

It does so using Concept API, because parsing an arbitrary TypeQL query and adding into it the ability to load connected concepts is a hard problem.

I believe GraMi (TypeDB Loader) suffers the same issue, it has to take in arbitrary user-written TypeQL queries and load connected concepts.

[haikalpribadi]

The behaviour you see in Workbase is very specific to Workbase. The act of querying a concept and immediately needing all of its neighbours is definitely not general to all TypeDB users. Think about it: when I query for X, I always get all that X is connected to - that makes no sense at all, and just comes with performance cost.

As for GraMi / TypeDB Loader, I don't understand why that would be the case. Can you elaborate with more detail how the behaviour in GraMi requires every concept to load its neighbours, @alexjpwalker ? cc: @hkuich

[alexjpwalker]

I haven't looked deeply into it but I remember Grami/TypeDB Loader needing to do potentially thousands of Concept API methods for some reason to do with processing arbitrary TypeQL queries. I think the scope of this problem reaches beyond Workbase.

[alexjpwalker]

I will close this issue as we now use Concept API in an async manner in Studio using CompletableFuture and it seems to be holding up perfectly fine.