search

aws-samples / sagemaker-studio-sparkmagic-lib

MIT No Attribution
7 stars 8 forks source link

Paginate list_instances call #8

Closed varun196 closed 3 years ago

varun196 commented 3 years ago

Description

EMR connection will fail if master node is not retrieved in the non-paginated list_instances call. This commits paginates the call, reformats the file using black and bounces PyPI version.

resolves #2

Testing Done

Added unit test.

❯ python3 -m black . && python3 -m pytest -vrfpP .                                                                                                                                          ─╯
All done! ✨ 🍰 ✨
10 files left unchanged.
===================================================================================== test session starts =====================================================================================
platform linux -- Python 3.7.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /workplace/shahvar/Astra/src/LooseLeafSparkMagicImage/SMStudioSparkMagicLib
plugins: anyio-3.3.0
collected 12 items

sagemaker_studio_sparkmagic_lib/tests/emr_test.py::test_get_cluster_happy_case_non_kerberos PASSED                                                                                      [  8%]
sagemaker_studio_sparkmagic_lib/tests/emr_test.py::test_get_cluster_happy_case_non_kerberos_with_pagination PASSED                                                                      [ 16%]
sagemaker_studio_sparkmagic_lib/tests/emr_test.py::test_get_cluster_happy_case_kerberos PASSED                                                                                          [ 25%]
sagemaker_studio_sparkmagic_lib/tests/emr_test.py::test_get_bad_cluster PASSED                                                                                                          [ 33%]
sagemaker_studio_sparkmagic_lib/tests/emr_test.py::test_cluster_dedicated_krb_cluster PASSED                                                                                            [ 41%]
sagemaker_studio_sparkmagic_lib/tests/emr_test.py::test_cross_realm_krb_cluster PASSED                                                                                                  [ 50%]
sagemaker_studio_sparkmagic_lib/tests/emr_test.py::test_external_kdc_cluster PASSED                                                                                                     [ 58%]
sagemaker_studio_sparkmagic_lib/tests/kerberos_test.py::test_generated_krb_conf PASSED                                                                                                  [ 66%]
sagemaker_studio_sparkmagic_lib/tests/utils_test.py::test_get_domain_search_without_search_line PASSED                                                                                  [ 75%]
sagemaker_studio_sparkmagic_lib/tests/utils_test.py::test_get_domain_search_happy_case PASSED                                                                                           [ 83%]
sagemaker_studio_sparkmagic_lib/tests/utils_test.py::test_get_domain_search_resolv_does_not_exist PASSED                                                                                [ 91%]
sagemaker_studio_sparkmagic_lib/tests/utils_test.py::test_get_emr_endpoint_url PASSED                                                                                                   [100%]

Also verified by installing on Studio and trying to connect to a cluster with 100 instances.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

edwardps commented 3 years ago

Can you bounce the version and also add description in the changelog?