Reconsider ingest infrastructure data architecture

[ofanobilbao]

We need to consider the way in which we link our domain entities together. The current implementation is causing performance issues in scale tests.

Acceptance criteria:

• investigate performance issues in the ingest-core bottlenecked by mongoDB I/O
• investigate ways of improving the data architecture
• write up a proposal for a change in architecture that addresses the issues, listing pros and cons, as well as changes required per components

Replaces HCA-985

[ofanobilbao]

On 5th June 2018 Rolando commented:

Proposal: all links between metadata entities are in the backwards direction i.e entities reference their provenance/inputs

Pros:

• Process creation is fast, assuming that Processes don’t have a large number of inputs (we haven’t observed that yet; Processes seems to have 1 input in general, 2 for analysis processes)
• Simplification of how things are linked together(always backwards)
• the repeated table scan in findAssays() is now a constant time operation

Cons:

• If there are processes with a large number of inputs, Process creation takes a long time
• Increased complexity for spreadsheet conversion as spreadsheets reference in the forward direction (edited)

[ofanobilbao]

On 10th August 2018 Norman commented:

Rolando is this done?

[ofanobilbao]

On 10th August 2018 Rolando commented:

Norman Morrison Still to do. However, as part of the coming push to ingest huge datasets, the work involved in this ticket will be either (a) completed as part of optimization efforts or (b) invalid if we can handle the datasets fine.