aws / sagemaker-spark

A Spark library for Amazon SageMaker.
https://aws.github.io/sagemaker-spark/
Apache License 2.0
300 stars 128 forks source link

fix: upgrade pyspark #135

Closed icywang86rui closed 3 years ago

icywang86rui commented 3 years ago

Issue #, if available:

Description of changes:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot commented 3 years ago

AWS CodeBuild CI Report

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot commented 3 years ago

AWS CodeBuild CI Report

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot commented 3 years ago

AWS CodeBuild CI Report

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot commented 3 years ago

AWS CodeBuild CI Report

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

e13h commented 3 years ago

I would love to see pyspark upgraded in this package. I noticed that the builds have failed, but after a closer look at the build logs, it seems like an unrelated issue to me.

It looks like the CI tool doesn't have the right permissions to read from AWS Secrets Manager.

From the logs:

[Container] 2021/04/20 23:17:10 Running command GITHUB_OAUTH_TOKEN=$(aws secretsmanager get-secret-value --secret-id /codebuild/github/oauth --query SecretString --output text)

An error occurred (AccessDeniedException) when calling the GetSecretValue operation: User: arn:aws:sts::142577830533:assumed-role/DevBuildStackPDX-PullRequestBuildRole4B40B95C-G4AE4UBWNU2K/AWSCodeBuild-0d42dff0-ffe3-4d5e-8b10-10f6d45ec659 is not authorized to perform: secretsmanager:GetSecretValue on resource: /codebuild/github/oauth

[Container] 2021/04/20 23:17:10 Command did not exit successfully GITHUB_OAUTH_TOKEN=$(aws secretsmanager get-secret-value --secret-id /codebuild/github/oauth --query SecretString --output text) exit status 255
[Container] 2021/04/20 23:17:10 Phase complete: PRE_BUILD State: FAILED
[Container] 2021/04/20 23:17:10 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: GITHUB_OAUTH_TOKEN=$(aws secretsmanager get-secret-value --secret-id /codebuild/github/oauth --query SecretString --output text). Reason: exit status 255
sagemaker-bot commented 3 years ago

AWS CodeBuild CI Report

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

icywang86rui commented 3 years ago

I would love to see pyspark upgraded in this package. I noticed that the builds have failed, but after a closer look at the build logs, it seems like an unrelated issue to me.

It looks like the CI tool doesn't have the right permissions to read from AWS Secrets Manager.

From the logs:

[Container] 2021/04/20 23:17:10 Running command GITHUB_OAUTH_TOKEN=$(aws secretsmanager get-secret-value --secret-id /codebuild/github/oauth --query SecretString --output text)

An error occurred (AccessDeniedException) when calling the GetSecretValue operation: User: arn:aws:sts::142577830533:assumed-role/DevBuildStackPDX-PullRequestBuildRole4B40B95C-G4AE4UBWNU2K/AWSCodeBuild-0d42dff0-ffe3-4d5e-8b10-10f6d45ec659 is not authorized to perform: secretsmanager:GetSecretValue on resource: /codebuild/github/oauth

[Container] 2021/04/20 23:17:10 Command did not exit successfully GITHUB_OAUTH_TOKEN=$(aws secretsmanager get-secret-value --secret-id /codebuild/github/oauth --query SecretString --output text) exit status 255
[Container] 2021/04/20 23:17:10 Phase complete: PRE_BUILD State: FAILED
[Container] 2021/04/20 23:17:10 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: GITHUB_OAUTH_TOKEN=$(aws secretsmanager get-secret-value --secret-id /codebuild/github/oauth --query SecretString --output text). Reason: exit status 255

Yeah. I fixed the permission issue. At least the scala part of the unit tests passed now. I am trying to figure out why the pyspark tests are failing.

sagemaker-bot commented 3 years ago

AWS CodeBuild CI Report

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot commented 3 years ago

AWS CodeBuild CI Report

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot commented 3 years ago

AWS CodeBuild CI Report

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot commented 3 years ago

AWS CodeBuild CI Report

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository