search

transferwise / pipelinewise-target-postgres

Singer.io Target for PostgreSQL - PipelineWise compatible
https://transferwise.github.io/pipelinewise/
Other
21 stars 38 forks source link

Compatibility with Python 3.8 #45

Open tchiotludo opened 3 years ago

tchiotludo commented 3 years ago

Running the target on python3.8 will lead to this error : 

Traceback (most recent call last):
  File "/tmp/singer-tap2/bin/target-postgres", line 5, in <module>
    from target_postgres import main
  File "/tmp/singer-tap2/lib/python3.8/site-packages/target_postgres/__init__.py", line 13, in <module>
    from joblib import Parallel, delayed, parallel_backend
  File "/tmp/singer-tap2/lib/python3.8/site-packages/joblib/__init__.py", line 119, in <module>
    from .parallel import Parallel
  File "/tmp/singer-tap2/lib/python3.8/site-packages/joblib/parallel.py", line 28, in <module>
    from ._parallel_backends import (FallbackToBackend, MultiprocessingBackend,
  File "/tmp/singer-tap2/lib/python3.8/site-packages/joblib/_parallel_backends.py", line 22, in <module>
    from .executor import get_memmapping_executor
  File "/tmp/singer-tap2/lib/python3.8/site-packages/joblib/executor.py", line 14, in <module>
    from .externals.loky.reusable_executor import get_reusable_executor
  File "/tmp/singer-tap2/lib/python3.8/site-packages/joblib/externals/loky/__init__.py", line 12, in <module>
    from .backend.reduction import set_loky_pickler
  File "/tmp/singer-tap2/lib/python3.8/site-packages/joblib/externals/loky/backend/reduction.py", line 125, in <module>
    from joblib.externals import cloudpickle  # noqa: F401
  File "/tmp/singer-tap2/lib/python3.8/site-packages/joblib/externals/cloudpickle/__init__.py", line 3, in <module>
    from .cloudpickle import *
  File "/tmp/singer-tap2/lib/python3.8/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 152, in <module>
    _cell_set_template_code = _make_cell_set_template_code()
  File "/tmp/singer-tap2/lib/python3.8/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 133, in _make_cell_set_template_code
    return types.CodeType(
TypeError: an integer is required (got type bytes)

Moving to python 3.7 works well. Seems to be a common issue with 3.8

larskoe commented 3 years ago

@tchiotludo I ran into the same issue. This is fixed in #35 but it was never pinned to a new version. Workaround I found was to use the actual pip_url when installing the target. Like so: git+https://github.com/transferwise/pipelinewise-target-postgres.git. This will ensure that joblib==0.16.0.

@koszti @Samira-El the latest version 2.1.0 doesn't have the fix from #35 in it yet. I see you that you both were active on that PR. Could one of you release a new version with the new changes?

Samira-El commented 3 years ago

Hey guys, I made a new release with the bump in joblib version https://github.com/transferwise/pipelinewise-target-postgres/releases/tag/v2.1.1

astrojuanlu commented 2 years ago

I guess this issue can be closed then?