Maestro v 1.1.9dev1 Compatibility

[bgunnar5]

In this branch I got the latest version of Maestro (v 1.1.9dev1) working with Merlin. To do this I had to:

1. Modify how MerlinSpec objects are created/validated

• Maestro v 1.1.7dev1 and later uses json schema validation to check the formatting of their yaml spec files. This was breaking our code when we tried to use a later version of Maestro since we had no way of modifying their schema file. The fix for this was to create our own json schema file that is roughly the same as Maestro's but with Merlin specific sections (like the merlin block).
• Leaving the schema mostly the same as theirs means we can make use of the validation functions that they already wrote and solely focus on Merlin specific validations in our code. You'll see this in the specification.py file under the verify() method, where I created validity checks for the merlin and batch blocks but use Maestro's validity checks for the rest of the blocks.

2. Create tests for the new validity checks

• I wrote a test class that creates a single yaml file that can be modified and reset on a test-by-test basis so adding new blocks and tests for new blocks in the future is easy to integrate

3. Unpack Maestro's ExecutionGraph that we were storing inside our own DAG object

• This is where the main problem points of using later versions of Maestro were coming from. When a Maestro study is staged, it resets the detect_cycles() method in their ExecutionGraph object to be a nested function called _pass_detect_cycles(). Pickle struggles with nested functions which resulted in us having to lock in a Maestro version from before they added this functionality.
• Merlin uses exactly two attributes from the ExecutionGraph: its adjacency table and its values dict. Since these are the only two things we use from the ExecutionGraph object, unpacking the object seemed like a reasonable option.

Other modifications in this branch include:

• Adding instructions in the Contributing page of the docs on how to add new blocks/fields to the merlin spec file
• Modifying the Workflow Specification page in the docs to put the "samples" field back inside the "merlin" block
• Adding a section in the Variables page to briefly explain what the $(LAUNCHER) variable is

Note: The only docs files that were changed modified were merlin_variables.rst, merlin_specification.rst, and merlin_developer.rst (just the end). For some reason, the commits I made to the last branch I worked on (the documentation edits) showed up as changes here as well.

[bgunnar5]

This branch also resolves issue #318

[lucpeterson]

@koning and @ryannova this has the python version changes too

[koning]

Does this allow tasks to be json serializable?

[bgunnar5]

@koning No we still use only pickle serialization. The only place json is used is for schema validation when we're creating the MerlinSpec object. I'll be playing around with json serialization in the future but most likely not on this branch

[koning]

Nice improvement Brian.