Failing to install from source on MacOS

rickybasse commented 5 years ago

I am having toubles installing the tool from source following the guidelines

Expected Behavior

The tool should complete the steps and successfully install

Actual Behavior

Executing ./install.sh --acceptlicenses results in:

...
Collecting PyMySQL==0.7.11 (from pipelinewise==0.10.1)
  Using cached https://files.pythonhosted.org/packages/c6/42/c54c280d8418039bd2f61284f99cb6d9e0eae80383fc72ceb6eac67855fe/PyMySQL-0.7.11-py2.py3-none-any.whl
Collecting psycopg2==2.8.2 (from pipelinewise==0.10.1)
  Using cached https://files.pythonhosted.org/packages/23/7e/93c325482c328619870b6cd09370f6dbe1148283daca65115cd63642e60f/psycopg2-2.8.2.tar.gz
    ERROR: Command errored out with exit status 1:
     command: /Users/rbassetto/babbel/repos/pipelinewise/.virtualenvs/pipelinewise/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/sh/gf_0zlrs6pl42pd9jg3yj8n80000gn/T/pip-install-vt2pyf42/psycopg2/setup.py'"'"'; __file__='"'"'/private/var/folders/sh/gf_0zlrs6pl42pd9jg3yj8n80000gn/T/pip-install-vt2pyf42/psycopg2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
         cwd: /private/var/folders/sh/gf_0zlrs6pl42pd9jg3yj8n80000gn/T/pip-install-vt2pyf42/psycopg2/
    Complete output (23 lines):
    running egg_info
    creating pip-egg-info/psycopg2.egg-info
    writing pip-egg-info/psycopg2.egg-info/PKG-INFO
    writing dependency_links to pip-egg-info/psycopg2.egg-info/dependency_links.txt
    writing top-level names to pip-egg-info/psycopg2.egg-info/top_level.txt
    writing manifest file 'pip-egg-info/psycopg2.egg-info/SOURCES.txt'

    Error: pg_config executable not found.

    pg_config is required to build psycopg2 from source.  Please add the directory
    containing pg_config to the $PATH or specify the full executable path with the
    option:

        python setup.py build_ext --pg-config /path/to/pg_config build ...

    or with the pg_config option in 'setup.cfg'.

    If you prefer to avoid building psycopg2 from source, please install the PyPI
    'psycopg2-binary' package instead.

    For further information please check the 'doc/src/install.rst' file (also at
    <http://initd.org/psycopg/docs/install.html>).

    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Steps to Reproduce the Problem

clone the repository
cd into the just cloned directory
execute ./install.sh --acceptlicenses

Specifications

OS: MacOS Mojave Version 10.14.6

Let me know if you need any other context information.

koszti commented 5 years ago

can you try install postgres dependencies with Homebrew? (https://brew.sh/)

brew install postgres and ./install.sh --acceptlicenses should do the trick

rickybasse commented 5 years ago

After installing postgres dependencies the process was able to collect psycopg2 but failed during the package installation phase.

The logs are quite verbose but this should be enough to give you some context:

...

Installing collected packages: argparse, tabulate, PyYAML, attrs, six, pyrsistent, jsonschema, MarkupSafe, jinja2, pycparser, cffi, bcrypt, asn1crypto, cryptography, pynacl, paramiko, ansible, joblib, idna, PyMySQL, psycopg2, jmespath, urllib3, python-dateutil, docutils, botocore, s3transfer, boto3, azure-common, certifi, chardet, requests, azure-nspkg, azure-storage, future, entrypoints, keyring, pyasn1, ijson, pyOpenSSL, pycryptodomex, pyjwt, pytz, pyasn1-modules, snowflake-connector-python, more-itertools, wcwidth, py, pyparsing, packaging, atomicwrites, zipp, importlib-metadata, pluggy, pytest, pytest-dependency, coverage, python-dotenv, pipelinewise
  Running setup.py install for tabulate ... done
  Running setup.py install for PyYAML ... done
  Running setup.py install for pyrsistent ... done
  Running setup.py install for ansible ... done
  Running setup.py install for psycopg2 ... error
    ERROR: Command errored out with exit status 1:
     command: /Users/rbassetto/babbel/repos/pipelinewise/.virtualenvs/pipelinewise/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/sh/gf_0zlrs6pl42pd9jg3yj8n80000gn/T/pip-install-nn6i3t5r/psycopg2/setup.py'"'"'; __file__='"'"'/private/var/folders/sh/gf_0zlrs6pl42pd9jg3yj8n80000gn/T/pip-install-nn6i3t5r/psycopg2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/sh/gf_0zlrs6pl42pd9jg3yj8n80000gn/T/pip-record-7_1y3wpc/install-record.txt --single-version-externally-managed --compile --install-headers /Users/rbassetto/babbel/repos/pipelinewise/.virtualenvs/pipelinewise/include/site/python3.7/psycopg2
         cwd: /private/var/folders/sh/gf_0zlrs6pl42pd9jg3yj8n80000gn/T/pip-install-nn6i3t5r/psycopg2/
    Complete output (144 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.14-x86_64-3.7
    creating build/lib.macosx-10.14-x86_64-3.7/psycopg2
    copying lib/_json.py -> build/lib.macosx-10.14-x86_64-3.7/psycopg2
    copying lib/extras.py -> build/lib.macosx-10.14-x86_64-3.7/psycopg2
    copying lib/compat.py -> build/lib.macosx-10.14-x86_64-3.7/psycopg2
    copying lib/errorcodes.py -> build/lib.macosx-10.14-x86_64-3.7/psycopg2
    copying lib/tz.py -> build/lib.macosx-10.14-x86_64-3.7/psycopg2
    copying lib/_range.py -> build/lib.macosx-10.14-x86_64-3.7/psycopg2
    copying lib/_ipaddress.py -> build/lib.macosx-10.14-x86_64-3.7/psycopg2
    copying lib/_lru_cache.py -> build/lib.macosx-10.14-x86_64-3.7/psycopg2
    copying lib/__init__.py -> build/lib.macosx-10.14-x86_64-3.7/psycopg2
    copying lib/extensions.py -> build/lib.macosx-10.14-x86_64-3.7/psycopg2
    copying lib/errors.py -> build/lib.macosx-10.14-x86_64-3.7/psycopg2
    copying lib/sql.py -> build/lib.macosx-10.14-x86_64-3.7/psycopg2
    copying lib/pool.py -> build/lib.macosx-10.14-x86_64-3.7/psycopg2
    running build_ext
    building 'psycopg2._psycopg' extension
    creating build/temp.macosx-10.14-x86_64-3.7
    creating build/temp.macosx-10.14-x86_64-3.7/psycopg
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -DPSYCOPG_VERSION=2.8.2 (dt dec pq3 ext lo64) -DPG_VERSION_NUM=110005 -DHAVE_LO64=1 -I/Users/rbassetto/babbel/repos/pipelinewise/.virtualenvs/pipelinewise/include -I/Users/rbassetto/.pyenv/versions/3.7.3/include/python3.7m -I. -I/usr/local/Cellar/postgresql/11.5_1/include -I/usr/local/Cellar/postgresql/11.5_1/include/server -c psycopg/psycopgmodule.c -o build/temp.macosx-10.14-x86_64-3.7/psycopg/psycopgmodule.o
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -DPSYCOPG_VERSION=2.8.2 (dt dec pq3 ext lo64) -DPG_VERSION_NUM=110005 -DHAVE_LO64=1 -I/Users/rbassetto/babbel/repos/pipelinewise/.virtualenvs/pipelinewise/include -I/Users/rbassetto/.pyenv/versions/3.7.3/include/python3.7m -I. -I/usr/local/Cellar/postgresql/11.5_1/include -I/usr/local/Cellar/postgresql/11.5_1/include/server -c psycopg/green.c -o build/temp.macosx-10.14-x86_64-3.7/psycopg/green.o
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -DPSYCOPG_VERSION=2.8.2 (dt dec pq3 ext lo64) -DPG_VERSION_NUM=110005 -DHAVE_LO64=1 -I/Users/rbassetto/babbel/repos/pipelinewise/.virtualenvs/pipelinewise/include -I/Users/rbassetto/.pyenv/versions/3.7.3/include/python3.7m -I. -I/usr/local/Cellar/postgresql/11.5_1/include -I/usr/local/Cellar/postgresql/11.5_1/include/server -c psycopg/pqpath.c -o build/temp.macosx-10.14-x86_64-3.7/psycopg/pqpath.o
    psycopg/pqpath.c:135:17: warning: implicit conversion from enumeration type 'ConnStatusType' to different enumeration type 'ExecStatusType' [-Wenum-conversion]
                    PQstatus(conn->pgconn) : PQresultStatus(*pgres)));
                    ^~~~~~~~~~~~~~~~~~~~~~
    psycopg/pqpath.c:1710:11: warning: code will never be executed [-Wunreachable-code]
        ret = 1;
              ^
    psycopg/pqpath.c:1815:17: warning: implicit conversion from enumeration type 'ConnStatusType' to different enumeration type 'ExecStatusType' [-Wenum-conversion]
                    PQstatus(curs->conn->pgconn) : PQresultStatus(curs->pgres)));
                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~

...

koszti commented 5 years ago

this looks similar to https://stackoverflow.com/questions/55839111/installing-psycopg2-fails-on-macos-with-unclear-error-message when installing psycopg2 fails. But to confirm we need to see the full error message. Maybe your openssl or some other library install path is not in sync with pg_config.

Is there anything like ld: library not found ... in the full output of the error message? If yes then you can try to compare the missing path in the error message to the output of pg_config. This might give a clue.

rickybasse commented 5 years ago

Sorry @koszti for the late reply, I moved to another topic and I didn't had time to continue with the investigation. Should we leave this one open anyway or you prefer to close it?

koszti commented 4 years ago

solved by switching to psycopg2-binary starting from PPW 0.16.0 by commit https://github.com/transferwise/pipelinewise/pull/397/files#diff-2eeaed663bd0d25b7e608891384b7298R28

psycopg2-binary doesn't require a compiler or external libraries.

resba commented 1 year ago

For those coming here from the google, if your trying to install psycopg2-binary on an M1/M2 macbook, use the most latest version of psycopg2-binary <3

transferwise / pipelinewise