cmaureir
commented
1 year ago Thanks for the detailed description @can-sun :+1:
I've set the upload limit for sagemaker-feature-store-pyspark to 400 MB on PyPI and TestPyPI. Please be mindful of the frequency of releases at that size.
have a nice week!
Project URL
https://pypi.org/project/sagemaker-feature-store-pyspark https://test.pypi.org/project/sagemaker-feature-store-pyspark
Does this project already exist?
New Limit
400
Update issue title
Which indexes
PyPI, TestPyPI
About the project
This project is managed by Amazon SageMaker FeatureStore team which acts as a spark connector for customers ingest data to SageMaker FeatureStore in stream/batch. For more information about this, aws official documentation can be found here: https://docs.aws.amazon.com/sagemaker/latest/dg/batch-ingestion-spark-connector-setup.html
Reasons for the request
For the next release 1.1.0 we have to support multiple spark versions and jars for different spark versions will be uploaded together with the python package which will break the current default limit. Our estimate for the new limit should be between 300~400M. In the future we will optimize this project and move all of the jars to a public S3 bucket we own, however process needs a security review. For now we want to increase such a limit to unblock ourselves for the next release so that customers can onboard to this new tool in different platforms.
Code of Conduct