Have loop apps use environment variables when missing args

[kwalcock]

The added arguments are named

REST_CONSUMER_INPUT_DIR
REST_CONSUMER_OUTPUT_DIR
REST_CONSUMER_DONE_DIR

EIDOS_INPUT_DIR
EIDOS_OUTPUT_DIR
EIDOS_DONE_DIR
EIDOS_THREADS

REST_PRODUCER_INPUT_DIR
REST_PRODUCER_DONE_DIR

The oddball kafka consumer uses a setting kafka.app.output.dir which can be set via KAFKA_APP_OUTPUT_DIR. (It seemed long ago like the convention for kafka apps was to put everything into configuration files, so I did.)

In all these cases, there is a FileUtils.ensureDirsExist to create the directories if something else hasn't yet.

[kwalcock]

FYI, @yanzv, please see the comment above.

[kwalcock]

This isn't quite right yet because all those shell programs set command line parameters. They need to be adjusted before this will work.

[kwalcock]

The input to the entire system probably needs to be baseDir, threads, and memory. Everything else is derived from them, so the values above can be used, but if they don't exist, then a default value can be generated.

[yanzv]

@kwalcock this is great. Will also have to update the start-loop.sh script and possibly remove the command line parameters. Regarding KAFKA_APP_OUTPUT_DIR, is this directory an input to the rest consumer? Some of these directories are output for one app and input for another. I am thinking if there is a way to consolidate these parameters under a different key EIDOS_xxxxx but not sure if that make sense in this case.

[kwalcock]

I do have a general configuration file that coordinates the directories that the separate parts share.

[kwalcock]

@yanzv, the configuration that you mention is implemented in loop_sbt_configure.sh. You may or may not want to add something similar to start-loop.sh.

[yanzv]

@yanzv, the configuration that you mention is implemented in loop_sbt_configure.sh. You may or may not want to add something similar to start-loop.sh.

@kwalcock Sure will check it out. Thanks