search

marrowp / segue

Automatically exported from code.google.com/p/segue
0 stars 0 forks source link

EMR gives an AWS 404 related to S3 #14

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?
1. Load EMR
2. setCredentials(xxx,xxx)
3. emr.test <- createCluster(numInstances=2 ) 
4. emr.result <- emrlapply(emr.test, data, jdm, taskTimeout=10)

What is the expected output? 

I would expect it to push my data/function to AWS EMR somehow (right?) and 
return the results of a regression and some basic math that's in my function 
called "jdm".

What do you see instead?

Status Code: 404, AWS Service: Amazon S3, AWS Request ID: 2A48E32A867E7C9C, AWS 
Error Code: NoSuchBucket, AWS Error Message: The specified bucket does not 
exist, S3 Extended Request ID: 
Fbxkh0YrGn26+ORg0Vh4aUUB1OU7Ubiq5SyaAsDHB7aatujn1jS3lH5mTjxRC95V

What version of the product are you using? On what operating system?

segue_0.05; Linux

Please provide any additional information below.

> emr.handle <- createCluster(numInstances=2 )
STARTING - 2013-10-29 22:53:51
STARTING - 2013-10-29 22:54:22
STARTING - 2013-10-29 22:54:53
STARTING - 2013-10-29 22:55:25
STARTING - 2013-10-29 22:55:56
STARTING - 2013-10-29 22:56:27
STARTING - 2013-10-29 22:56:58
STARTING - 2013-10-29 22:57:29
BOOTSTRAPPING - 2013-10-29 22:58:00
BOOTSTRAPPING - 2013-10-29 22:58:32
WAITING - 2013-10-29 22:59:03
Your Amazon EMR Hadoop Cluster is ready for action. 
Remember to terminate your cluster with stopCluster().
Amazon is billing you!

> emr.result <- emrlapply(emr.handle, data, jdm, taskTimeout=10)
Error in .jcall("RJavaTools", "Ljava/lang/Object;", "invokeMethod", cl,  : 
  Status Code: 404, AWS Service: Amazon S3, AWS Request ID: 2A48E32A867E7C9C, AWS Error Code: NoSuchBucket, AWS Error Message: The specified bucket does not exist, S3 Extended Request ID: Fbxkh0YrGn26+ORg0Vh4aUUB1OU7Ubiq5SyaAsDHB7aatujn1jS3lH5mTjxRC95V

Original issue reported on code.google.com by millerin...@gmail.com on 30 Oct 2013 at 5:39

GoogleCodeExporter commented 9 years ago 
I am getting a similar issue except my instances start however rstudio shows a 
"shutting-down" status. What's even more odd is that the EC2 instances continue 
to run in the EC2 console:

> myCluster <- createCluster(numInstances=3, location = "us-east-1a")
STARTING - 2014-02-03 23:12:55
STARTING - 2014-02-03 23:13:25
STARTING - 2014-02-03 23:13:56
STARTING - 2014-02-03 23:14:26
STARTING - 2014-02-03 23:14:57
STARTING - 2014-02-03 23:15:28
STARTING - 2014-02-03 23:15:59
SHUTTING_DOWN - 2014-02-03 23:16:29
SHUTTING_DOWN - 2014-02-03 23:17:00
SHUTTING_DOWN - 2014-02-03 23:17:30

Here is the STDERR description:
+ sudo tee /etc/apt/sources.list.d/R.list
+ gpg --keyserver pgp.mit.edu --recv-key 381BA480
gpg: directory `/home/hadoop/.gnupg' created
gpg: new configuration file `/home/hadoop/.gnupg/gpg.conf' created
gpg: WARNING: options in `/home/hadoop/.gnupg/gpg.conf' are not yet active 
during this run
gpg: keyring `/home/hadoop/.gnupg/secring.gpg' created
gpg: keyring `/home/hadoop/.gnupg/pubring.gpg' created
gpg: requesting key 381BA480 from hkp server pgp.mit.edu
gpg: /home/hadoop/.gnupg/trustdb.gpg: trustdb created
gpg: key 381BA480: public key "Johannes Ranke (CRAN Debian archive) 
<jranke@uni-bremen.de>" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1
+ gpg -a --export 381BA480
+ sudo apt-key add jranke_cran.asc
+ sudo apt-get update
W: A error occurred during the signature verification. The repository is not 
updated and the previous index files will be used. GPG error: 
http://http.us.debian.org testing Release: The following signatures couldn't be 
verified because the public key is not available: NO_PUBKEY 8B48AD6246925553

W: Failed to fetch http://http.us.debian.org/debian/dists/testing/Release  

W: Some index files failed to download, they have been ignored, or old ones 
used instead.
+ sudo DEBIAN_FRONTEND=noninteractive apt-get install -t testing --yes 
--force-yes gcc
Failed to fetch 
http://http.us.debian.org/debian/pool/main/e/eglibc/libc-bin_2.13-24_amd64.deb  
404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/e/eglibc/libc6-i386_2.13-24_amd64.deb
  404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/g/gcj-4.4/gcj-4.4-base_4.4.6-11_amd64
.deb  404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/g/gcc-4.6/gcc-4.6-base_4.6.2-11_amd64
.deb  404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/e/eglibc/multiarch-support_2.13-24_am
d64.deb  404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/g/gcc-4.6/libstdc++6_4.6.2-11_amd64.d
eb  404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/g/gmp/libgmp10_5.0.2+dfsg-2_amd64.deb
  404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/m/mpfr4/libmpfr4_3.1.0-3_amd64.deb  
404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/g/gcc-4.4/g++-4.4_4.4.6-14_amd64.deb 
 404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/g/gcc-4.4/libstdc++6-4.4-dev_4.4.6-14
_amd64.deb  404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/g/gcc-4.6/libgomp1_4.6.2-11_amd64.deb
  404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/g/gcc-4.6/libquadmath0_4.6.2-11_amd64
.deb  404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/g/gcc-4.6/libgfortran3_4.6.2-11_amd64
.deb  404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/g/gcc-4.4/gcc-4.4-base_4.4.6-14_amd64
.deb  404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/g/gcc-4.4/cpp-4.4_4.4.6-14_amd64.deb 
 404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/g/gcc-4.4/gcc-4.4_4.4.6-14_amd64.deb 
 404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/g/gcc-4.6/libgcc1_4.6.2-11_amd64.deb 
 404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/g/gcc-4.6/lib32stdc++6_4.6.2-11_amd64
.deb  404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/g/gcc-4.6/lib32gcc1_4.6.2-11_amd64.de
b  404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/g/gcc-4.4/gfortran-4.4_4.4.6-14_amd64
.deb  404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/m/make-dfsg/make_3.81-8.1_amd64.deb  
404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/e/eglibc/libc6-dev_2.13-24_amd64.deb 
 404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/e/eglibc/libc-dev-bin_2.13-24_amd64.d
eb  404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/e/eglibc/locales_2.13-24_all.deb  
404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/e/eglibc/libc6_2.13-24_amd64.deb  
404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/b/binutils/binutils_2.22-4_amd64.deb 
 404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/g/gcc-4.6/cpp-4.6_4.6.2-11_amd64.deb 
 404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/g/gcc-defaults/cpp_4.6.2-4_amd64.deb 
 404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/g/gcc-4.6/gcc-4.6_4.6.2-11_amd64.deb 
 404  Not Found [IP: 64.50.236.52 80]
Failed to fetch 
http://http.us.debian.org/debian/pool/main/g/gcc-defaults/gcc_4.6.2-4_amd64.deb 
 404  Not Found [IP: 64.50.236.52 80]
E: Unable to fetch some archives, maybe run apt-get update or try with 
--fix-missing?

Original comment by vedjai...@gmail.com on 3 Feb 2014 at 11:31

GoogleCodeExporter commented 9 years ago 
Hi,
I have the same problem today. Using Segue 0.05 on EMR. 
There is any solution?

Tks

Original comment by thiagoca...@gmail.com on 28 Oct 2014 at 8:38

GoogleCodeExporter commented 9 years ago 
Having the same issue as well if anyone knows of a workaround please post.

Thanks..

Original comment by gdelga...@gmail.com on 20 Nov 2014 at 4:04