search

transferwise / pipelinewise-target-redshift

Singer.io Target for Amazon Redshift - PipelineWise compatible
https://transferwise.github.io/pipelinewise/
Other
12 stars 65 forks source link

[AP-473] switch to pipelinewise-singer-python to customize logging #41

Closed Samira-El closed 4 years ago

koszti commented 4 years ago

Test results look fine, and all of them passing 

==================================================================================================================== test session starts ====================================================================================================================
platform darwin -- Python 3.7.6, pytest-5.3.0, py-1.8.0, pluggy-0.13.1 -- /Users/peter.kosztolanyi/workspaces/transferwise/pipelinewise-target-redshift/venv/bin/python3
cachedir: .pytest_cache
rootdir: /Users/peter.kosztolanyi/workspaces/transferwise/pipelinewise-target-redshift
collected 33 items                                                                                                                                                                                                                                          

tests/integration/test_target_redshift.py::TestTargetRedshift::test_invalid_json PASSED                                                                                                                                                               [  3%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_message_order PASSED                                                                                                                                                              [  6%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_loading_tables PASSED                                                                                                                                                             [  9%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_loading_tables_with_metadata_columns PASSED                                                                                                                                       [ 12%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_loading_tables_with_defined_parallelism PASSED                                                                                                                                    [ 15%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_loading_tables_with_defined_slice_number PASSED                                                                                                                                   [ 18%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_loading_tables_with_gzip_compression PASSED                                                                                                                                       [ 21%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_loading_tables_with_bz2_compression PASSED                                                                                                                                        [ 24%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_loading_tables_with_hard_delete PASSED                                                                                                                                            [ 27%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_loading_with_multiple_schema PASSED                                                                                                                                               [ 30%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_loading_unicode_characters PASSED                                                                                                                                                 [ 33%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_loading_long_text PASSED                                                                                                                                                          [ 36%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_non_db_friendly_columns PASSED                                                                                                                                      [ 39%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_nested_schema_unflattening PASSED                                                                                                                                   [ 42%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_nested_schema_flattening PASSED                                                                                                                                     [ 45%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_column_name_change PASSED                                                                                                                                           [ 48%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_grant_privileges PASSED                                                                                                                                             [ 51%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_custom_copy_options PASSED                                                                                                                                          [ 54%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_copy_using_aws_environment PASSED                                                                                                                                   [ 57%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_copy_using_role_arn PASSED                                                                                                                                          [ 60%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_invalid_custom_copy_options PASSED                                                                                                                                  [ 63%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_logical_streams_from_pg_with_hard_delete_and_default_batch_size_should_pass PASSED                                                                                  [ 66%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_logical_streams_from_pg_with_hard_delete_and_batch_size_of_5_should_pass PASSED                                                                                     [ 69%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_logical_streams_from_pg_with_hard_delete_and_batch_size_of_5_and_no_records_should_pass PASSED                                                                      [ 72%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_flush_streams_with_no_intermediate_flushes PASSED                                                                                                                   [ 75%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_flush_streams_with_intermediate_flushes PASSED                                                                                                                      [ 78%]
tests/integration/test_target_redshift.py::TestTargetRedshift::test_flush_streams_with_intermediate_flushes_on_all_streams PASSED                                                                                                       [ 81%]
tests/unit/test_db_sync.py::TestTargetRedshift::test_config_validation PASSED                                                                                                                                                           [ 84%]
tests/unit/test_db_sync.py::TestTargetRedshift::test_column_type_mapping PASSED                                                                                                                                                         [ 87%]
tests/unit/test_db_sync.py::TestTargetRedshift::test_stream_name_to_dict PASSED                                                                                                                                                         [ 90%]
tests/unit/test_db_sync.py::TestTargetRedshift::test_flatten_schema PASSED                                                                                                                                                              [ 93%]
tests/unit/test_db_sync.py::TestTargetRedshift::test_flatten_record PASSED                                                                                                                                                              [ 96%]
tests/unit/test_target_rs.py::TestTargetRedshift::test_persist_lines_with_40_records_and_batch_size_of_20_expect_flushing_once PASSED                                                                                                   [100%]

================================================================================================= 33 passed, 4 warnings in 670.84s (0:11:10) ==================================================================================================
koszti commented 4 years ago

Output looks nice

cat tests/integration/resources/messages-with-three-streams.json | target-redshift --config config.json 
time=2020-02-11 08:39:14 name=target_redshift level=INFO message=Caching available catalog objects in redshift...
time=2020-02-11 08:39:15 name=target_redshift level=INFO message=Table 'local_dev1.test_table_one' exists
time=2020-02-11 08:39:15 name=target_redshift level=INFO message=Table 'local_dev1.test_table_two' exists
time=2020-02-11 08:39:15 name=target_redshift level=INFO message=Table 'local_dev1.test_table_three' exists
time=2020-02-11 08:39:15 name=target_redshift level=INFO message=Uploading 2 rows to S3
time=2020-02-11 08:39:15 name=target_redshift level=INFO message=Target S3 bucket: pipe-redshift-test, local file: /var/folders/rq/877vn4v537lblxrx19wgc_352x0wzf/T/tmpn_ytar0a.csv.1, S3 key: redshift-imports-test/pipelinewise_tap_mysql_test-test_table_two_20200210-215915-319469.csv.1
time=2020-02-11 08:39:15 name=target_redshift level=INFO message=Uploading 1 rows to S3
time=2020-02-11 08:39:15 name=target_redshift level=INFO message=Target S3 bucket: pipe-redshift-test, local file: /var/folders/rq/877vn4v537lblxrx19wgc_352x0wzf/T/tmp2vr555sc.csv.1, S3 key: redshift-imports-test/pipelinewise_tap_mysql_test-test_table_one_20200210-215915-319313.csv.1
time=2020-02-11 08:39:15 name=target_redshift level=INFO message=Uploading 3 rows to S3
time=2020-02-11 08:39:15 name=target_redshift level=INFO message=Target S3 bucket: pipe-redshift-test, local file: /var/folders/rq/877vn4v537lblxrx19wgc_352x0wzf/T/tmpl40vf_6s.csv.1, S3 key: redshift-imports-test/pipelinewise_tap_mysql_test-test_table_three_20200210-215915-319665.csv.1
time=2020-02-11 08:39:15 name=target_redshift level=INFO message=Loading 3 rows into 'local_dev1.stg_test_table_three'
time=2020-02-11 08:39:15 name=target_redshift level=INFO message=Loading 1 rows into 'local_dev1.stg_test_table_one'
time=2020-02-11 08:39:15 name=target_redshift level=INFO message=Loading 2 rows into 'local_dev1.stg_test_table_two'
time=2020-02-11 08:39:16 name=target_redshift level=INFO message=Deleting redshift-imports-test/pipelinewise_tap_mysql_test-test_table_three_20200210-215915-319665.csv.1 from S3
time=2020-02-11 08:39:16 name=target_redshift level=INFO message=Deleting redshift-imports-test/pipelinewise_tap_mysql_test-test_table_one_20200210-215915-319313.csv.1 from S3
time=2020-02-11 08:39:23 name=target_redshift level=INFO message=Deleting redshift-imports-test/pipelinewise_tap_mysql_test-test_table_two_20200210-215915-319469.csv.1 from S3
time=2020-02-11 08:39:23 name=target_redshift level=INFO message=Emitting state {"currently_syncing": null, "bookmarks": {"tap_mysql_test-test_table_one": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_two": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_three": {"initial_full_table_complete": true}}}
{"currently_syncing": null, "bookmarks": {"tap_mysql_test-test_table_one": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_two": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_three": {"initial_full_table_complete": true}}}