[CCI]Added Poetry Management For Testing

roma2023 commented 1 year ago

Description

Poetry provides an automated and efficient way to manage project dependencies, virtual environments, and packaging. It cuts out redundant layers of bash scripts and dev requirements. Also, it significantly simplifies the project's onboarding process for contributors.

This PR aims to integrate Poetry management tool, so that testing would just run after:

poetry install
poetry run pytest

Issues Resolved

Targets issue #392

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov[bot] commented 1 year ago

Codecov Report

Merging #496 (c25a922) into main (c8b04a5) will not change coverage. The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #496   +/-   ##
=======================================
  Coverage   70.92%   70.92%           
=======================================
  Files          81       81           
  Lines        7732     7732           
=======================================
  Hits         5484     5484           
  Misses       2248     2248

roma2023 commented 1 year ago

I would really appreciate any guidance on this project. Locally I am passing 900 tests and failing 21. And those fails are mostly associated with the absence of modules like botocore. But they seem to be already installed and enlisted in poetry.lock.

Here are more details on the fails:

FAILED test_opensearchpy/test_connection.py::TestUrllib3Connection::test_aws_signer_as_http_auth - ModuleNotFoundError: No module named 'botocore'
FAILED test_opensearchpy/test_connection.py::TestUrllib3Connection::test_aws_signer_when_service_is_specified - ModuleNotFoundError: No module named 'botocore'
FAILED test_opensearchpy/test_connection.py::TestSignerWithFrozenCredentials::test_aws_signer_as_http_auth - ModuleNotFoundError: No module named 'botocore'
FAILED test_opensearchpy/test_connection.py::TestSignerWithFrozenCredentials::test_aws_signer_frozen_credentials_as_http_auth - ModuleNotFoundError: No module named 'botocore'      
FAILED test_opensearchpy/test_connection.py::TestSignerWithFrozenCredentials::test_aws_signer_when_service_is_specified - ModuleNotFoundError: No module named 'botocore'
FAILED test_opensearchpy/test_transport.py::TestTransport::test_failed_connection_will_be_marked_as_dead - TypeError: '<' not supported between instances of 'DummyConnection' and 'DummyConnection'
FAILED test_opensearchpy/test_transport.py::TestTransport::test_resurrected_connection_will_be_marked_as_live_on_success - TypeError: '<' not supported between instances of 'DummyConnection' and 'DummyConnection'
FAILED test_opensearchpy/test_async/test_connection.py::TestAIOHttpConnection::test_uses_given_ca_certs - AssertionError: Expected 'load_verify_locations' to be called once. Called 
3 times.
FAILED test_opensearchpy/test_async/test_connection.py::TestAIOHttpConnection::test_uses_default_ca_certs - AssertionError: Expected 'load_verify_locations' to be called once. Called 3 times.
FAILED test_opensearchpy/test_async/test_connection.py::TestAIOHttpConnection::test_uses_no_ca_certs - AssertionError: Expected 'load_verify_locations' to not have been called. Called 2 times.
FAILED test_opensearchpy/test_async/test_signer.py::TestAsyncSigner::test_aws_signer_async_as_http_auth - ModuleNotFoundError: No module named 'botocore'
FAILED test_opensearchpy/test_async/test_signer.py::TestAsyncSigner::test_aws_signer_async_when_service_is_specified - ModuleNotFoundError: No module named 'botocore'
FAILED test_opensearchpy/test_async/test_signer.py::TestAsyncSignerWithFrozenCredentials::test_aws_signer_async_as_http_auth - ModuleNotFoundError: No module named 'botocore'       
FAILED test_opensearchpy/test_async/test_signer.py::TestAsyncSignerWithFrozenCredentials::test_aws_signer_async_when_service_is_specified - ModuleNotFoundError: No module named 'botocore'
FAILED test_opensearchpy/test_async/test_signer.py::TestAsyncSignerWithFrozenCredentials::test_aws_signer_async_frozen_credentials_as_http_auth - ModuleNotFoundError: No module named 'botocore'
FAILED test_opensearchpy/test_async/test_transport.py::TestTransport::test_failed_connection_will_be_marked_as_dead - TypeError: '<' not supported between instances of 'DummyConnection' and 'DummyConnection'
FAILED test_opensearchpy/test_async/test_transport.py::TestTransport::test_resurrected_connection_will_be_marked_as_live_on_success - TypeError: '<' not supported between instances 
of 'DummyConnection' and 'DummyConnection'
FAILED test_opensearchpy/test_async/test_server/test_plugins/test_index_management.py::TestIndexManagementPlugin::test_create_policy - opensearchpy.exceptions.ConflictError: ConflictError(409, 'version_conflict_engine_exception', '[example-policy]: version conflict, document already exists (current version [1])')
FAILED test_opensearchpy/test_async/test_server/test_plugins/test_index_management.py::TestIndexManagementPlugin::test_delete_policy - opensearchpy.exceptions.ConflictError: ConflictError(409, 'version_conflict_engine_exception', '[example-policy]: version conflict, document already exists (current version [1])')
FAILED test_opensearchpy/test_async/test_server/test_plugins/test_index_management.py::TestIndexManagementPlugin::test_get_policy - opensearchpy.exceptions.ConflictError: ConflictError(409, 'version_conflict_engine_exception', '[example-policy]: version conflict, document already exists (current version [1])')
FAILED test_opensearchpy/test_async/test_server/test_plugins/test_index_management.py::TestIndexManagementPlugin::test_update_policy - opensearchpy.exceptions.ConflictError: ConflictError(409, 'version_conflict_engine_exception', '[example-policy]: version conflict, document already exists (current version [1])')
ERROR test_opensearchpy/test_async/test_server/test_clients.py::TestUnicode::test_indices_analyze - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_clients.py::TestBulk::test_bulk_works_with_string_body - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_clients.py::TestBulk::test_bulk_works_with_bytestring_body - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_clients.py::TestYarlMissing::test_aiohttp_connection_works_without_yarl - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestStreamingBulk::test_actions_remain_unchanged - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestStreamingBulk::test_all_documents_get_inserted - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestStreamingBulk::test_documents_data_types - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestStreamingBulk::test_all_errors_from_chunk_are_raised_on_failure - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestStreamingBulk::test_different_op_types - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestStreamingBulk::test_transport_error_can_becaught - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestStreamingBulk::test_rejected_documents_are_retried - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestStreamingBulk::test_rejected_documents_are_retried_at_most_max_retries_times - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestStreamingBulk::test_transport_error_is_raised_with_max_retries - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestBulk::test_bulk_works_with_single_item - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestBulk::test_all_documents_get_inserted - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestBulk::test_stats_only_reports_numbers - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestBulk::test_errors_are_reported_correctly - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestBulk::test_error_is_raised - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestBulk::test_ignore_error_if_raised - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestBulk::test_errors_are_collected_properly - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestScan::test_order_can_be_preserved - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestScan::test_all_documents_are_read - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestScan::test_scroll_error - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestScan::test_initial_search_error - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestScan::test_no_scroll_id_fast_route - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestScan::test_logger - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestScan::test_clear_scroll - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestScan::test_scan_auth_kwargs_forwarded[kwargs0] - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestScan::test_scan_auth_kwargs_forwarded[kwargs1] - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestScan::test_scan_auth_kwargs_forwarded[kwargs2] - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestScan::test_scan_auth_kwargs_favor_scroll_kwargs_option - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestReindex::test_reindex_passes_kwargs_to_scan_and_bulk - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestReindex::test_reindex_accepts_a_query - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestReindex::test_all_documents_get_moved - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_helpers/test_actions.py::TestParentChildReindex::test_children_are_reindexed_correctly - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_plugins/test_index_management.py::TestIndexManagementPlugin::test_create_policy - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_plugins/test_index_management.py::TestIndexManagementPlugin::test_delete_policy - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_plugins/test_index_management.py::TestIndexManagementPlugin::test_get_policy - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_async/test_server/test_plugins/test_index_management.py::TestIndexManagementPlugin::test_update_policy - opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'no permissions for [] and User [name=admin, backend_roles=[admin], requestedT...
ERROR test_opensearchpy/test_server/test_helpers/test_count.py::test_count_prefetch
================================================= 21 failed, 925 passed, 51 skipped, 298 warnings, 40 errors in 158.14s (0:02:38) ===========

dblock commented 1 year ago

I am seeing unit tests all pass, but integration tests fail with a different reason:

2023-09-14T22:05:15.0933043Z Obtaining file:///home/runner/work/opensearch-py/opensearch-py/opensearch-py
2023-09-14T22:05:15.0952579Z   Installing build dependencies: started
2023-09-14T22:05:16.7193915Z   Installing build dependencies: finished with status 'done'
2023-09-14T22:05:16.7204616Z   Checking if build backend supports build_editable: started
2023-09-14T22:05:16.8009044Z   Checking if build backend supports build_editable: finished with status 'done'
2023-09-14T22:05:16.8019630Z   Getting requirements to build editable: started
2023-09-14T22:05:16.8806743Z   Getting requirements to build editable: finished with status 'done'
2023-09-14T22:05:16.8816509Z   Preparing editable metadata (pyproject.toml): started
2023-09-14T22:05:17.0905706Z   Preparing editable metadata (pyproject.toml): finished with status 'error'
2023-09-14T22:05:17.0969317Z   error: subprocess-exited-with-error
2023-09-14T22:05:17.0969626Z   
2023-09-14T22:05:17.0971923Z   × Preparing editable metadata (pyproject.toml) did not run successfully.
2023-09-14T22:05:17.0972856Z   │ exit code: 1
2023-09-14T22:05:17.0973388Z   ╰─> [16 lines of output]
2023-09-14T22:05:17.0974431Z       Traceback (most recent call last):
2023-09-14T22:05:17.0975224Z         File "/home/runner/.local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
2023-09-14T22:05:17.0975844Z           main()
2023-09-14T22:05:17.0976621Z         File "/home/runner/.local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
2023-09-14T22:05:17.0977155Z           json_out['return_val'] = hook(**hook_input['kwargs'])
2023-09-14T22:05:17.1017122Z         File "/home/runner/.local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 181, in prepare_metadata_for_build_editable
2023-09-14T22:05:17.1017919Z           return hook(metadata_directory, config_settings)
2023-09-14T22:05:17.1020731Z         File "/tmp/pip-build-env-brbl5pu7/overlay/local/lib/python3.10/dist-packages/poetry/core/masonry/api.py", line 42, in prepare_metadata_for_build_wheel
2023-09-14T22:05:17.1023283Z           builder = WheelBuilder(poetry)
2023-09-14T22:05:17.1025296Z         File "/tmp/pip-build-env-brbl5pu7/overlay/local/lib/python3.10/dist-packages/poetry/core/masonry/builders/wheel.py", line 61, in __init__
2023-09-14T22:05:17.1026262Z           super().__init__(poetry, executable=executable)
2023-09-14T22:05:17.1027209Z         File "/tmp/pip-build-env-brbl5pu7/overlay/local/lib/python3.10/dist-packages/poetry/core/masonry/builders/builder.py", line 85, in __init__
2023-09-14T22:05:17.1027785Z           self._module = Module(
2023-09-14T22:05:17.1028508Z         File "/tmp/pip-build-env-brbl5pu7/overlay/local/lib/python3.10/dist-packages/poetry/core/masonry/utils/module.py", line 69, in __init__
2023-09-14T22:05:17.1029157Z           raise ModuleOrPackageNotFound(
2023-09-14T22:05:17.1029974Z       poetry.core.masonry.utils.module.ModuleOrPackageNotFound: No file/folder found for package package1
2023-09-14T22:05:17.1030463Z       [end of output]
2023-09-14T22:05:17.1030679Z   
2023-09-14T22:05:17.1031024Z   note: This error originates from a subprocess, and is likely not a problem with pip.
2023-09-14T22:05:17.1032033Z error: metadata-generation-failed

Are you seeing something else?

Also I notice some of our yml's say "ruby=...", we should fix that :)

dblock commented 10 months ago

Closing this as stale. Please feel free to reopen if you finish it.

opensearch-project / opensearch-py