questions on manual_install.sh

[rkazak]

I am opening this issue for manual_install.sh so that I can bounce ideas on it with you as I come across them?

[rkazak]

issue 1 The script should be re-executable and as it successfully passes an install stage remember and not try to re-do that stage.

issue 2 A check in some initialization function could check java is available before starting the main part of the process, I recently 'cleaned' my machine and have resisted installing arbitrary tools...

issue 3 Should we be using the 'pip' that's installed as part of anaconda, if so do we need to source the updated .bash_profile to get the recent modifications?

[rjurney]

Thanks for your help with this! I will make a pass at improving the script and get back to you.

[rkazak]

You are very welcome. I too may take a stab at some of this.

issue 4 We need a way to restore the original .bash_profile for when things go wrong, else this too becomes a source of errors.

[rkazak]

issue 5 We should also check if any other pre-requisites we are going to install are not already available on the host.

[rkazak]

enhancement 1 We should provide a choice between anaconda and miniconda if it's practical during installation.

[rkazak]

enhancement 2 I would suggest that rather than updating .bash_profile, put all that into your own file and source it through .bash_profile?

[rkazak]

issue 6 Why is the airflow directory outside the project_root ?

[rkazak]

issue 7 line 126 is cd $PROJECT_HOME# to $PROJECT_HOME

• should be cd $PROJECT_HOME # to $PROJECT_HOME

[rkazak]

issue 8 I had to up the version of zeppelin to 0.7.0 as the 0.6.2 is no longer there. also updated kafka from 2.11-0.10.1.1 to 2.12-0.10.1.1

[rkazak]

issue 9

• line 139 elasticsearch/bin/elasticsearch 2>1 > /dev/null & # re-run if you shutdown your computer
• should be elasticsearch/bin/elasticsearch 2>&1 > /dev/null & # re-run if you shutdown your computer

[rjurney]

@rkazak First of all, thanks for your help! I am really thrilled at your participation :)

1) That is a good idea. I just created a ticket for this feature: https://github.com/rjurney/Agile_Data_Code_2/issues/14 2) Another good idea: https://github.com/rjurney/Agile_Data_Code_2/issues/15 3) I think not, because we set the environment variables for path, etc. right before we alter .bash_profile 4) Agreed. We should cp it to .back and have a uninstall script. https://github.com/rjurney/Agile_Data_Code_2/issues/16 5 & 6) I'm not sure about this. How would I check that a valid Hadoop is installed? This bring up an issue... in the other install scripts, software is installed in the /home/ubuntu or /home/vagrant directories and only the code is in Agile_Data_Code_2. I was hesitant to install so much software outside the project directory for a manual install, so everything installs into Agile_Data_Code_2. What do you think is the right thing to do here? Anyway, we should be consistent so I created https://github.com/rjurney/Agile_Data_Code_2/issues/17 E1) Not a bad idea... actually, for the manual install maybe we just choose Anaconda? The only reason we use miniconda for the VMs is that it was taking an hour or more to download and users can't tell why that is and give up. This isn't as likely for a manual install where it is clearer what is going on? What do you think?

[rjurney]

@rkazak I fixed Issue #14 in https://github.com/rjurney/Agile_Data_Code_2/commit/62878c1a4e41d17d063112f49c824b4896952a0a Can you please test it out?

[rjurney]

@rkazak I fixed issue #15 in https://github.com/rjurney/Agile_Data_Code_2/commit/46b244d0bc29c29cf2334c42671e4c58f52d5023 Can you please test it out?

[rjurney]

@rkazak I made the backup of .bash_profile re: #16 in https://github.com/rjurney/Agile_Data_Code_2/commit/bdab034406542f326a4ba2b8bb18a6f707cd4cb1 Can you test it out? I am waiting to do the uninstall script later.

[rjurney]

Please see the individual issues, which are resolved. Please re-open this issue if you run into more issues.