search

transferwise / pipelinewise-target-redshift

Singer.io Target for Amazon Redshift - PipelineWise compatible
https://transferwise.github.io/pipelinewise/
Other
12 stars 65 forks source link

Fix MutableMapping for python 3.10 #160

Closed Limess closed 8 months ago

Limess commented 2 years ago

Context

Support Python >= 3.10

This fails on Python 3.10:

[2022-05-25, 16:19:55 BST] {{subprocess.py:92}} INFO -   File "/virtualenvs/pipelinewise-target-redshift/bin/target-redshift", line 8, in <module>
[2022-05-25, 16:19:55 BST] {{subprocess.py:92}} INFO -     sys.exit(main())
[2022-05-25, 16:19:55 BST] {{subprocess.py:92}} INFO -   File "/virtualenvs/pipelinewise-target-redshift/lib/python3.10/site-packages/target_redshift/__init__.py", line 447, in main
[2022-05-25, 16:19:55 BST] {{subprocess.py:92}} INFO -     persist_lines(config, singer_messages, table_cache)
[2022-05-25, 16:19:55 BST] {{subprocess.py:92}} INFO -   File "/virtualenvs/pipelinewise-target-redshift/lib/python3.10/site-packages/target_redshift/__init__.py", line 162, in persist_lines
[2022-05-25, 16:19:55 BST] {{subprocess.py:92}} INFO -     primary_key_string = stream_to_sync[stream].record_primary_key_string(o['record'])
[2022-05-25, 16:19:55 BST] {{subprocess.py:92}} INFO -   File "/virtualenvs/pipelinewise-target-redshift/lib/python3.10/site-packages/target_redshift/db_sync.py", line 360, in record_primary_key_string
[2022-05-25, 16:19:55 BST] {{subprocess.py:92}} INFO -     flatten = flatten_record(record, self.flatten_schema, max_level=self.data_flattening_max_level)
[2022-05-25, 16:19:55 BST] {{subprocess.py:92}} INFO -   File "/virtualenvs/pipelinewise-target-redshift/lib/python3.10/site-packages/target_redshift/db_sync.py", line 164, in flatten_record
[2022-05-25, 16:19:55 BST] {{subprocess.py:92}} INFO -     if isinstance(v, collections.MutableMapping) and level < max_level:
[2022-05-25, 16:19:55 BST] {{subprocess.py:92}} INFO - AttributeError: module 'collections' has no attribute 'MutableMapping'

See https://stackoverflow.com/a/71902541/2738092 for context.

Checklist

craigastill commented 1 year ago

In absence of a Maintainer review (12 months at time of writing this), this PR is a cleaner fix for the imports vs #228 .

There's currently no automated validation for the fix. My suggestion would be to add a python 3.10 CI job.

Limess commented 8 months ago

Re-raised as https://github.com/transferwise/pipelinewise-target-redshift/pull/298 as I had raised from our forks master branch.