Improve BatchCreateFactor

[zachkinstner]

The BatchCreateFactor function needs to be improved. Accomplishing this will not be trivial, and may require RexConnect and/or Weaver changes, so I'm creating this issue to keep things organized.

[zachkinstner]

For each Factor, the following tasks need to occur (in roughly this order):

• Validate all parameters
• Obtain/verify the Primary Artifact
• Obtain/verify the Related Artifact
• Obtain/verify the Descriptor's RefinesPrimary Artifact (if present)
• Obtain/verify the Descriptor's RefinesRelated Artifact (if present)
• Obtain/verify the Descriptor's RefinesType Artifact (if present)
• Obtain/verify the Vector's Axis Artifact (if present)
• Add the new Factor
• Add the MemberCreatesFactor edge/VCI
• Add the FactorUsesPrimaryArtifact edge/VCI
• Add the FactorUsesRelatedArtifact edge/VCI
• Add the FactorDescriptorRefinesPrimaryArtifact edge/VCI (if present)
• Add the FactorDescriptorRefinesRelatedArtifact edge/VCI (if present)
• Add the FactorDescriptorRefinesTypeArtifact edge/VCI (if present)
• Add the FactorVectorUsesAxisArtifact edge/VCI (if present)
• Return the new Factor's vertex ID

Upon each Artifact "verify" step, if the Artifact does not exist, then all the remaining tasks in the sequence should be skipped, and the entire sequence should return an error/value that indicates the issue.

[zachkinstner]

The original implementation did this:

• Validated parameters
• Attempted to obtain all Artifacts
  • Checked in-memory cache
  • Sent RexConnect request with a command for each Artifact ID (sometimes hundreds)
  • Added a "not found" error response to each applicable batch response item
  • Cached in-memory Artifact-existence values
• Created Factor vertex and edges
  • Sent RexConnect request with, for each Factor:
  • One command for adding the Factor, saving it to a variable, and returning the new vertex ID
  • One command for the three mandatory Artifact edges
  • One command for each of the four optional Artifact edges

Notes:

• The in-memory Artifact cache doesn't really help much, since the Artifact will always need to be loaded into a variable on the database side.
• The query for adding a Factor can change lengths due to many optional parameters. This means re-compilation of many similar queries.
• Combining three edge-creation queries into one command is bad enough, but becomes much worse when VCI support is included.

[zachkinstner]

I'm going to invent a theoretical RexConnect feature, which might help solve this problem: conditional command execution. It would work by checking the response of a specified command for a value of false, zero, or null.

[zachkinstner]

New implementation idea (skipping optional edges for now):

Cmd ID	Cond?	Name	Pseudo-Query
Mem0		Get Mem Once	m=g.V('MId',_P0).next();
F0.0		Get Prim Art	pa=g.V('AId',_P0); if(pa){ pa=pa.next(); }; pa;
F0.1	F0.0	Get Rel Art	ra=g.V('AId',_P0); if(ra){ ra=ra.next(); }; ra;
F0.2	F0.1	Add Factor	f=g.addVertex([...]); f.id;
F0.3	F0.1	Add Mem Edge	...VCI...; g.addEdge(m,f,'Creates',[...VCI...]);
F0.4	F0.1	Add Prim Edge	...VCI...; g.addEdge(f,pa,'UsesPrimary',[...VCI...]);
F0.5	F0.1	Add Rel Edge	...VCI...; g.addEdge(f,ra,'UsesRelated',[...VCI...]);

Notes:

• The Fx.y pattern increases x with each Factor, and y with each command.
• For each Factor, record which command ID stores the new Factor vertex ID (it's F0.2 above).
• Use of the new RexConnect conditional execution:
  • The command after an Artifact command is dependent upon its result.
  • All commands after the last Artifact command (A) are conditional upon A.
  • Thus, any missing Artifact short-circuits the remaining commands.

[zachkinstner]

This could be done without a new RexConnect feature:

1. Initialize a boolean variable: pass=true;
2. If any Artifact is missing: pass=false;
3. Before each query, do: if(pass){ return null; };

This is solution is not very elegant, and it adds an extra command for each Factor. If an Artifact is missing, then this solution still executes several extra commands that will all return null (a minor point).

[zachkinstner]

The query for adding a Factor can change lengths due to many optional parameters. This means re-compilation of many similar queries.

To avoid this, create a property map over several commands:

1. Initialize props with all mandatory Factor properties
2. For each applicable element E, add mandatory properties of E to props
3. For each applicable optional Factor and element property, add property to props
4. Create new Factor: g.addVertex(props);

This approach minimizes the number of possible (unique, parameterized) query scripts, and makes those queries much shorter. Thus, those queries are faster to compile, and take less memory to cache. Note that only the optional properties are added individually. The mandatory properties are added in bunches (since the order/quantity doesn't change).

[zachkinstner]

This is also a great test case for RexConnect vs. RexProClient performance.

These BatchCreateFactor improvements would be simpler to implement with RexProClient, since each query is executed individually. The logic can occur entirely in the application code (instead of needing conditional RexConnect commands, etc.), so if an Artifact isn't found, the application code can simply set the error response and move on.

The assumed downside to that approach is the quantity of round-trips to the database, and the de/serialization that occurs each time. The performance tests would determine its actual impact.

[zachkinstner]

The integration test has ~300 factors, with 20 factors in each request, and 3-degree parallelism. It executes in 2.5 to 3.0 seconds. I'm curious to see the performance impact on the production environment.