sfc-gh-tmathew
commented
7 months ago @binitbhaskar, It does not appear to be a bug with schemachange.
Schemachange is supported on python 3.8, 3.9 and 3.10. Your docker container seems to be running Python 3.12.
Can you use python 3.10 instead of python 3.12 in your gitlab pipeline?
Describe the bug While running the docker command in gitlab cicd to run schemachange, we are getting error. Please find the gitlab stage details below
deploy-to-eu-staging-snowflake: stage: snowflake-staging when: manual image: docker:latest services:
Steps to reproduce the behavior:
Expected behavior A clear and concise description of what you expected to happen.
Screenshots
Schemachange (please complete the following information):
Additional context Please find the details logs in the gitlab cicd
$ docker run --rm --name schemachange-script -v "$PWD":/usr/src/schemachange -w /usr/src/schemachange -e SNOWFLAKE_USER -e SNOWFLAKE_ROLE -e SNOWFLAKE_PRIVATE_KEY_PATH -e SNOWFLAKE_PRIVATE_KEY -e SNOWFLAKE_AUTHENTICATOR python:3 /bin/bash -c "export OPENBLAS_NUM_THREADS=1 && pip install schemachange --upgrade --progress-bar off --debug && schemachange -f contents/ -a SNOWFLAKE_URL -u $SNOWFLAKE_USER -r $SNOWFLAKE_ROLE -w TEST_WH -d TEST_DB -c TEST_DB.SCHEMACHANGE.CHANGE_HISTORY" Unable to find image 'python:3' locally 3: Pulling from library/python 8457fd5474e7: Pulling fs layer 13baa2029dde: Pulling fs layer 325c5bf4c2f2: Pulling fs layer 7e18a660069f: Pulling fs layer 98a59f0ffede: Pulling fs layer 72c7f17f2221: Pulling fs layer 2f40b346325a: Pulling fs layer f3f08e04e337: Pulling fs layer
Digest: sha256:7b8d65a924f596eb65306214f559253c468336bcae09fd575429774563460caf Status: Downloaded newer image for python:3 Collecting schemachange Obtaining dependency information for schemachange from https://files.pythonhosted.org/packages/91/36/828c2caac9933af76893e3f73fe5e63f0b100e49bf1aa52c48160ceb87fd/schemachange-3.6.0-py3-none-any.whl.metadata Downloading schemachange-3.6.0-py3-none-any.whl.metadata (33 kB) Collecting jinja2~=3.0 (from schemachange) Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB) Collecting pandas~=1.3 (from schemachange) Downloading pandas-1.5.3.tar.gz (5.2 MB) Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: still running... Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'done' Collecting pyyaml~=6.0 (from schemachange) Obtaining dependency information for pyyaml~=6.0 from https://files.pythonhosted.org/packages/b4/33/720548182ffa8344418126017aa1d4ab4aeec9a2275f04ce3f3573d8ace8/PyYAML-6.0.1-cp[31](https://gitlab.com/sinch/sinch-projects/enterprise-and-messaging/beehive/finance/rating/genesis/financial-data-pipeline-selfserve/-/jobs/5489018214#L31)2-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata Downloading PyYAML-6.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB) Collecting snowflake-connector-python<4.0,>=2.8 (from schemachange) Downloading snowflake-connector-python-3.4.0.tar.gz (719 kB) Installing build dependencies: started Installing build dependencies: finished with status 'error' error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully. │ exit code: 1 ╰─> [331 lines of output] Collecting setuptools>=40.6.0 Obtaining dependency information for setuptools>=40.6.0 from https://files.pythonhosted.org/packages/bb/26/7945080113158354380a12ce26873dd6c1ebd88d47f5bc24e2c5bb38c16a/setuptools-68.2.2-py3-none-any.whl.metadata Using cached setuptools-68.2.2-py3-none-any.whl.metadata (6.3 kB) Collecting wheel Obtaining dependency information for wheel from https://files.pythonhosted.org/packages/fa/7f/4c07234086edbce4a0a446209dc0cb08a19bb206a3ea53b2f56a403f983b/wheel-0.41.3-py3-none-any.whl.metadata Using cached wheel-0.41.3-py3-none-any.whl.metadata (2.2 kB) Collecting cython Obtaining dependency information for cython from https://files.pythonhosted.org/packages/0a/f5/a629b0decba74c6cd1f9e830d1e55b89b6be15507d26c7cc14bc87275b28/Cython-3.0.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata Downloading Cython-3.0.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.2 kB) Collecting pyarrow<10.1.0,>=10.0.1 Downloading pyarrow-10.0.1.tar.gz (994 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 994.1/994.1 kB 15.4 MB/s eta 0:00:00 Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'done' Collecting numpy>=1.16.6 (from pyarrow<10.1.0,>=10.0.1) Obtaining dependency information for numpy>=1.16.6 from https://files.pythonhosted.org/packages/07/c0/ccbb2a4c75b283d6100400a907087bfa4d89cee9df73fa6af85268115d81/numpy-1.26.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata Using cached numpy-1.26.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB) Using cached setuptools-68.2.2-py3-none-any.whl (807 kB) Using cached wheel-0.41.3-py3-none-any.whl (65 kB) Using cached Cython-3.0.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.5 MB) Using cached numpy-1.26.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.9 MB) Building wheels for collected packages: pyarrow Building wheel for pyarrow (pyproject.toml): started Building wheel for pyarrow (pyproject.toml): finished with status 'error' error: subprocess-exited-with-error
note: This error originates from a subprocess, and is likely not a problem with pip. [notice] A new release of pip is available: 23.2.1 -> 23.3.1 [notice] To update, run: pip install --upgrade pip ╭───────────────────── Traceback (most recent call last) ─────────────────────────────────╮ │ │ │ args = [ │ │ │ │ │ '/usr/local/bin/python', │ │ │ │ │ │ │ │ │ '/usr/local/lib/python3.12/site-packages/pip/pip-run… │ │ │ │ │ 'install', │ │ │ │ │ '--ignore-installed', │ │ │ │ │ '--no-user', │ │ │ │ │ '--prefix', │ │ │ │ │ '/tmp/pip-build-env-21a4wy6w/overlay', │ │ │ │ │ '--no-warn-script-location', │ │ │ │ │ '--no-binary', │ │ │ │ │ ':none:', │ │ │ │ │ ... +9 │ │ │ │ ] │ │ │ │ extra_environ = { │ │ │ │ │ '_PIP_STANDALONE_CERT': │ │ │ │ '/usr/local/lib/python3.12/site-packages/pip/_vendor/c… │ │ │ │ } │ │ │ │ finder = <pip._internal.index.package_finder.PackageFinder │ │ │ │ object at 0x7f3aeb1fc0b0> │ │ │ │ format_control = 'only_binary' │ │ │ │ formats = set() │ │ │ │ index_urls = ['https://pypi.org/simple'] │ │ │ │ kind = 'build dependencies' │ │ │ │ pip_runnable = '/usr/local/lib/python3.12/site-packages/pip/__pip-run… │ │ │ │ prefix = <pip._internal.build_env._Prefix object at │ │ │ │ 0x7f3ae9b0b380> │ │ │ │ requirements = [ │ │ │ │ │ 'setuptools>=40.6.0', │ │ │ │ │ 'wheel', │ │ │ │ │ 'cython', │ │ │ │ │ 'pyarrow>=10.0.1,<10.1.0' │ │ │ │ ] │ │ │ │ spinner = <pip._internal.cli.spinners.NonInteractiveSpinner │ │ │ │ object at 0x7f3ae9b0b590> │ │ │ ╰──────────────────────────────────────────────────────────────────────────╯ │ │ │ │ /usr/local/lib/python3.12/site-packages/pip/_internal/utils/subprocess.py:22 │ │ 4 in call_subprocess │ │ │ │ 221 │ │ │ │ │ extra={"markup": True}, │ │ 222 │ │ │ │ ) │ │ 223 │ │ │ │ │ ❱ 224 │ │ │ raise error │ │ 225 │ │ elif on_returncode == "warn": │ │ 226 │ │ │ subprocess_logger.warning( │ │ 227 │ │ │ │ 'Command "%s" had error code %s in %s', │ │ │ │ ╭───────────────────────────────── locals ─────────────────────────────────╮ │ │ │ all_output = [ │ │ │ │ │ 'Collecting setuptools>=40.6.0\n', │ │ │ │ │ ' Obtaining dependency information for │ │ │ │ setuptools>=40.6.0 from https://files.pyt'+134, │ │ │ │ │ ' Using cached │ │ │ │ setuptools-68.2.2-py3-none-any.whl.metadata (6.3 │ │ │ │ kB)\n', │ │ │ │ │ 'Collecting wheel\n', │ │ │ │ │ ' Obtaining dependency information for wheel │ │ │ │ from https://files.pythonhosted.org'+116, │ │ │ │ │ ' Using cached │ │ │ │ wheel-0.41.3-py3-none-any.whl.metadata (2.2 │ │ │ │ kB)\n', │ │ │ │ │ 'Collecting cython\n', │ │ │ │ │ ' Obtaining dependency information for │ │ │ │ cython from https://files.pythonhosted.or'+159, │ │ │ │ │ ' Downloading │ │ │ │ Cython-3.0.5-cp312-cp312-manylinux_2_17_x86_64.m… │ │ │ │ │ 'Collecting pyarrow<10.1.0,>=10.0.1\n', │ │ │ │ │ ... +321 │ │ │ │ ] │ │ │ │ cmd = [ │ │ │ │ │ '/usr/local/bin/python', │ │ │ │ │ │ │ │ │ '/usr/local/lib/python3.12/site-packages/pip/p… │ │ │ │ │ 'install', │ │ │ │ │ '--ignore-installed', │ │ │ │ │ '--no-user', │ │ │ │ │ '--prefix', │ │ │ │ │ '/tmp/pip-build-env-21a4wy6w/overlay', │ │ │ │ │ '--no-warn-script-location', │ │ │ │ │ '--no-binary', │ │ │ │ │ ':none:', │ │ │ │ │ ... +9 │ │ │ │ ] │ │ │ │ command_desc = 'pip subprocess to install build dependencies' │ │ │ │ cwd = None │ │ │ │ env = { │ │ │ │ │ 'HOSTNAME': '4ffeab1ce008', │ │ │ │ │ 'PYTHON_VERSION': '3.12.0', │ │ │ │ │ 'PWD': '/usr/src/schemachange', │ │ │ │ │ 'OPENBLAS_NUM_THREADS': '1', │ │ │ │ │ 'HOME': '/root', │ │ │ │ │ 'LANG': 'C.UTF-8', │ │ │ │ │ 'GPG_KEY': │ │ │ │ '7169605F62C751356D054A26A821E680E5FA6305', │ │ │ │ │ 'SHLVL': '1', │ │ │ │ │ 'PYTHON_PIP_VERSION': '23.2.1', │ │ │ │ │ 'PYTHON_GET_PIP_SHA256': │ │ │ │ '22b849a10f86f5ddf7ce148ca2a31214504ee6c83ef6268… │ │ │ │ │ ... +5 │ │ │ │ } │ │ │ │ error = <InstallationSubprocessError(reference='subproce… │ │ │ │ message='[green]pip subprocess to install build │ │ │ │ dependencies[/] did not run successfully.\nexit │ │ │ │ code: 1', context='See above for output.', │ │ │ │ note_stmt='This error originates from a │ │ │ │ subprocess, and is likely not a problem with │ │ │ │ pip.', hint_stmt=None)> │ │ │ │ extra_environ = { │ │ │ │ │ '_PIP_STANDALONE_CERT': │ │ │ │ '/usr/local/lib/python3.12/site-packages/pip/_ve… │ │ │ │ } │ │ │ │ extra_ok_returncodes = [] │ │ │ │ line = '' │ │ │ │ log_failed_cmd = True │ │ │ │ log_subprocess = <bound method VerboseLogger.verbose of │ │ │ │ <VerboseLogger pip.subprocessor (INFO)>> │ │ │ │ on_returncode = 'raise' │ │ │ │ output = 'Collecting setuptools>=40.6.0\n Obtaining │ │ │ │ dependency information for setuptools>'+30381 │ │ │ │ proc = <Popen: returncode: 1 args: │ │ │ │ ['/usr/local/bin/python', │ │ │ │ '/usr/local/lib/python...> │ │ │ │ proc_had_error = True │ │ │ │ show_stdout = False │ │ │ │ showing_subprocess = False │ │ │ │ spinner = <pip._internal.cli.spinners.NonInteractiveSpinner │ │ │ │ object at 0x7f3ae9b0b590> │ │ │ │ stdout_only = False │ │ │ │ unset_environ = [] │ │ │ │ use_spinner = True │ │ │ │ used_level = 15 │ │ │ ╰──────────────────────────────────────────────────────────────────────────╯ │ ╰──────────────────────────────────────────────────────────────────────────────╯ InstallationSubprocessError: pip subprocess to install build dependencies exited with 1 Cleaning up project directory and file based variables 00:01 ERROR: Job failed: exit code 1