Better configuration system, configurable `spark-submit` and env vars

[timodonnell]

• Simplified configuration system. No more special handling of paths: there's just an automatic THIS_DIR substitution available that gives the absolute path to the directory of the current JSON file. No more ConfigDict object. We now resolve all substitutions when loading the config and return a plain dict.
• spark-submit is now configured in demeter.json. May have to update for gcloud.
• Running on a cluster now supports arbitrary environment variables. demter.json sets YARN_CONF_DIR and SPARK_HOME

[arahuja]

Looks fine to me, the variable substitution seems a bit weird, does it add much over just using environment variables directly?

[timodonnell]

Thanks @arahuja . Pushed a commit addressing these.

I'd like to find an off the shelf configuration library that does variable substitition but haven't found one unfortunately (YAML doesn't do it), so I think I'll stick with the current hacky system for now