search

crowdrec / idomaar

CrowdRec reference framework
Apache License 2.0
32 stars 12 forks source link

Check on newsreel evaluator on aws #64

Closed davidemalagoli closed 8 years ago

davidemalagoli commented 8 years ago

Hi, We've finished the tests on machine creation on aws, mainly modifying:

now it is possible to fire up a machine with "vagrant up --provider=aws", you need to set the following env variables: AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_PRIVATEKEY_PATH

i've tested the flow with mahout/spark evaluator and the communication works as expected. Could you please test it with newsreel eval? The branch is "aws".

When this will be ready we can ship both the current code and the already provisioned ami to speed up the process.

If you want I can create a IAM account to use the crowdrec account.

Thanks

andras-sereny commented 8 years ago

How do I get the access keys?

andras-sereny commented 8 years ago

This fails for me:

/c/dev/repo/idomaar/datastreammanager$ vagrant up --provider=aws Bringing machine 'default' up with 'aws' provider... ==> default: Box 'dummy' could not be found. Attempting to find and install... default: Box Provider: aws default: Box Version: >= 0 ==> default: Adding box 'dummy' (v0) for provider: aws default: Downloading: https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box ==> default: Box download is resuming from prior download progress default: Progress: 100% (Rate: 1342k/s, Estimated time remaining: --:--:--) The box you attempted to add doesn't match the provider you specified.

Is this Ubuntu box supposed to work on EC2?

davidemalagoli commented 8 years ago

Hi, sorry perhaps I've not specified to use AWS branch, that branch contains the new Vagrantfile that has the aws section.

On Tue, Dec 22, 2015 at 3:23 PM, András Serény notifications@github.com wrote:

This fails for me:

/c/dev/repo/idomaar/datastreammanager$ vagrant up --provider=aws Bringing machine 'default' up with 'aws' provider... ==> default: Box 'dummy' could not be found. Attempting to find and install... default: Box Provider: aws default: Box Version: >= 0 ==> default: Adding box 'dummy' (v0) for provider: aws default: Downloading: https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box ==> default: Box download is resuming from prior download progress default: Progress: 100% (Rate: 1342k/s, Estimated time remaining: --:--:--) The box you attempted to add doesn't match the provider you specified.

Is this Ubuntu box supposed to work on EC2?

— Reply to this email directly or view it on GitHub https://github.com/crowdrec/idomaar/issues/64#issuecomment-166628120.

Davide Malagoli Mobile: +39 345 6539950

andras-sereny commented 8 years ago

I'm using the aws branch and the aws section is indeed in the Vagrantfile. Still, I get the above error.

davidemalagoli commented 8 years ago

I've tried right now and it works correctly please:

andras-sereny commented 8 years ago

It does seem to me the box https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box defined in the Vagrantfile won't work with the aws provider.

Don't you happen to have a box named 'dummy' added to Vagrant locally?

andras-sereny commented 8 years ago

When I use the public Vagrant box lattice/ubuntu-trusty-64, it starts up fine, so it's not an aws auth or similiar issue. However, this box is not suitable for Idomaar (e.g. has no java), so it'd nice to know the actual box you're using.

davidemalagoli commented 8 years ago

Is strange cause the provider should fire the ami configured with AWS_AMI parameters ( AWS_AMI = 'ami-5189a661', that is an official ubuntu image from http://cloud-images.ubuntu.com/releases/14.04/release-20150325/) and it must not use a vagrant box. Java is installed via shell command (/datastreammanager/vagrant/01.java.sh)

Perhaps you need to execute the following command to add the dummy box?

vagrant box add dummy https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box

On Mon, Jan 11, 2016 at 3:18 PM, András Serény notifications@github.com wrote:

When I use the public Vagrant box lattice/ubuntu-trusty-64, it starts up fine, so it's not an aws auth or similiar issue. However, this box is not suitable for Idomaar (e.g. has no java), so it'd nice to know the actual box you're using.

— Reply to this email directly or view it on GitHub https://github.com/crowdrec/idomaar/issues/64#issuecomment-170564211.

Davide Malagoli Mobile: +39 345 6539950

andras-sereny commented 8 years ago

Indeed, adding the dummy box fixes this, thanks.

andras-sereny commented 8 years ago

After a few fixes (7d3676fd, e1989f90), it almost works. I think the error now is in my demo computing env, but I'll confirm this tomorrow.

Interestingly, there is no trace of a Zookeeper release 3.4.7, http://zookeeper.apache.org/releases.html, (referenced from Idomaar), so I downgraded the ZK dependency to 3.4.6.

andras-sereny commented 8 years ago

Indeed, it's all fine now.