search

aws / sagemaker-spark-container

The SageMaker Spark Container is a Docker image used to run data processing workloads with the Spark framework on Amazon SageMaker.
Apache License 2.0
36 stars 74 forks source link

Remove python sdk and fix history integ test failure due to s3 eventual consistency #26

Closed guoqiao1992 closed 4 years ago

guoqiao1992 commented 4 years ago

Issue #, if available:

  1. Spark Processor was released in Sagemaker python sdk.
  2. History server integration tests failed indeterministically. We suspect that it's due to s3 eventual consistency.

Description of changes:

  1. Remove the python sdk inside container.
  2. Try to get s3 file stats first before starting history server

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.