Exascale networking - updates

[ialarmedalien]

Some updates to the stored queries, document schema, and added some tests and data to liven things up.

• The Jacobson lab have provided a new file of edge data, with all previous datasets merged into a single file. I have therefore rejigged the parser to use this file as the source.
• Also added in a bit more error checking to ensure that dodgy data doesn't make its way into Arango.
• Added parser tests and test data.

Schema changes:

• merged the phenotype and gene documents so we just refer to nodes and edges
• added docs / names / examples for all fields
• reworked the current queries to fit requirements -- all queries now return edges and nodes, with a configurable 'distance' variable

[ialarmedalien]

Overall looks good and I don't have any adamant requests. I do think moving the djornl importer into a second module and wrapping it in a class are two extraneous layers. I also had some tangential thoughts about running importers, which may or may not affect your testing setup:

importers should be able to run without the user having access to the local data and without checking in the data to the repo, meaning they would be responsible for downloading the raw data over network We could use either blobstore or file cache server to store the data, and when we download the data, the importer should validate a checksum The RE API should check whether imports have run, and if not, then kick them off. This could be an explicit poke

A couple of thoughts / responses to your thoughts and responses:

• I want the Jacobson people to redo the node (gene/phenotype) metadata that they provide, and ideally give us an idea of any other data types that they're going to be using, so that we can create a more robust way of storing the data, either in a single collection or across a set of collections;
• I also want the Jacobson data to be stored somewhere that can be accessed in the same manner as the GO / ENVO / etc. ontologies, so there's no "special treatment" required for the data set. At present, it's in a GDrive folder, which is far from ideal, and I don't really want to have to write some special importer for Google Drive. Setting up something more sustainable (with version control and that is easy to access programmatically) would be much better; if their data isn't top secret, maybe a GitHub repo would work.
• I'm not attached to having the parser in a separate file and class; the reasons for moving it were (a) having an informative file name, esp. when more importers get added, and (b) making it easier to test different configurations.