Open dhutchis opened 8 years ago
For 2), it isn't necessary to copy the Myria jar or config file to anywhere but the coordinator. REEF takes care of deploying all necessary files and configuration to the workers. For 4), if we add more per-worker parameters, it might be time to switch to YAML. As for reading a password from a file, I think that's best handled with environment variables. We could discuss in a separate issue how to override settings in the Myria configuration file with environment variables.
Per-worker settings for 4) don't matter for the LocalRuntime mode, since it only runs on one node anyway. They ought to matter for the YarnClientConfiguration mode. Unfortunately I could only get Yarn working to the point that the Myria driver talks to Yarn, but no coordinators or workers seem to start.
HPC clusters like TX-E1 present a small handful of challenges. Items 1 through 3 are solved; this issue is about item 4.
scp
to copy the myria jar and deployment to every worker node. ExportGRADLE_USER_HOME
to some place on the local disk to do the gradle build with file locking.yarn.resourcemanager.hostname
, add nodes to theslaves
file, and use thesbin/start-yarn.sh
scripts. This works for multi-node setup too.There are steps we can take to solve item 4. We should expand the
deployment.cfg
file to include these options, and pass them down to theWorkers
and to theJdbcInfo
classes.Here is an example of the logic we might add to JdbcInfo's constructor, except that it shouldn't be hardcoded: