USCDataScience / sparkler

Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.
http://irds.usc.edu/sparkler/
Apache License 2.0
410 stars 143 forks source link

Fix sparkler CI build #222

Open lewismc opened 3 years ago

lewismc commented 3 years ago

The CI build if failing

#10 35.54 Collecting scipy==1.4.1
#10 35.56   Downloading scipy-1.4.1.tar.gz (24.6 MB)
#10 38.26   Installing build dependencies: started
#10 104.0   Installing build dependencies: still running...
#10 165.2   Installing build dependencies: still running...
#10 221.0   Installing build dependencies: finished with status 'done'
#10 221.0   Getting requirements to build wheel: started
#10 221.3   Getting requirements to build wheel: finished with status 'done'
#10 221.3     Preparing wheel metadata: started
#10 221.8     Preparing wheel metadata: finished with status 'error'
#10 221.8     ERROR: Command errored out with exit status 1:
#10 221.8      command: /usr/local/bin/python /usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpifpjdxhq
#10 221.8          cwd: /tmp/pip-install-93_jp9d1/scipy_3982779a15134ff5a98a4a76a9e0844e
#10 221.8     Complete output (139 lines):
#10 221.8     setup.py:418: UserWarning: Unrecognized setuptools command ('dist_info --egg-base /tmp/pip-modern-metadata-cld5f994'), proceeding with generating Cython sources and expanding templates
#10 221.8       warnings.warn("Unrecognized setuptools command ('{}'), proceeding with "
#10 221.8     Running from scipy source directory.
#10 221.8     lapack_opt_info:
#10 221.8     lapack_mkl_info:
#10 221.8     customize UnixCCompiler
#10 221.8       libraries mkl_rt not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
#10 221.8       NOT AVAILABLE
#10 221.8     
#10 221.8     openblas_lapack_info:
#10 221.8     customize UnixCCompiler
#10 221.8     customize UnixCCompiler
#10 221.8       libraries openblas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
#10 221.8       NOT AVAILABLE
#10 221.8     
#10 221.8     openblas_clapack_info:
#10 221.8     customize UnixCCompiler
#10 221.8     customize UnixCCompiler
#10 221.8       libraries openblas,lapack not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
#10 221.8       NOT AVAILABLE
#10 221.8     
#10 221.8     flame_info:
#10 221.8     customize UnixCCompiler
#10 221.8       libraries flame not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
#10 221.8       NOT AVAILABLE
#10 221.8     
#10 221.8     atlas_3_10_threads_info:
#10 221.8     Setting PTATLAS=ATLAS
#10 221.8     customize UnixCCompiler
#10 221.8       libraries lapack_atlas not found in /usr/local/lib
#10 221.8     customize UnixCCompiler
#10 221.8       libraries tatlas,tatlas not found in /usr/local/lib
#10 221.8     customize UnixCCompiler
#10 221.8       libraries lapack_atlas not found in /usr/lib
#10 221.8     customize UnixCCompiler
#10 221.8       libraries tatlas,tatlas not found in /usr/lib
#10 221.8     customize UnixCCompiler
#10 221.8       libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
#10 221.8     customize UnixCCompiler
#10 221.8       libraries tatlas,tatlas not found in /usr/lib/x86_64-linux-gnu
#10 221.8     <class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
#10 221.8       NOT AVAILABLE
#10 221.8     
#10 221.8     atlas_3_10_info:
#10 221.8     customize UnixCCompiler
#10 221.8       libraries lapack_atlas not found in /usr/local/lib
#10 221.8     customize UnixCCompiler
#10 221.8       libraries satlas,satlas not found in /usr/local/lib
#10 221.8     customize UnixCCompiler
#10 221.8       libraries lapack_atlas not found in /usr/lib
#10 221.8     customize UnixCCompiler
#10 221.8       libraries satlas,satlas not found in /usr/lib
#10 221.8     customize UnixCCompiler
#10 221.8       libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
#10 221.8     customize UnixCCompiler
#10 221.8       libraries satlas,satlas not found in /usr/lib/x86_64-linux-gnu
#10 221.8     <class 'numpy.distutils.system_info.atlas_3_10_info'>
#10 221.8       NOT AVAILABLE
#10 221.8     
#10 221.8     atlas_threads_info:
#10 221.8     Setting PTATLAS=ATLAS
#10 221.8     customize UnixCCompiler
#10 221.8       libraries lapack_atlas not found in /usr/local/lib
#10 221.8     customize UnixCCompiler
#10 221.8       libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
#10 221.8     customize UnixCCompiler
#10 221.8       libraries lapack_atlas not found in /usr/lib
#10 221.8     customize UnixCCompiler
#10 221.8       libraries ptf77blas,ptcblas,atlas not found in /usr/lib
#10 221.8     customize UnixCCompiler
#10 221.8       libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
#10 221.8     customize UnixCCompiler
#10 221.8       libraries ptf77blas,ptcblas,atlas not found in /usr/lib/x86_64-linux-gnu
#10 221.8     <class 'numpy.distutils.system_info.atlas_threads_info'>
#10 221.8       NOT AVAILABLE
#10 221.8     
#10 221.8     atlas_info:
#10 221.8     customize UnixCCompiler
#10 221.8       libraries lapack_atlas not found in /usr/local/lib
#10 221.8     customize UnixCCompiler
#10 221.8       libraries f77blas,cblas,atlas not found in /usr/local/lib
#10 221.8     customize UnixCCompiler
#10 221.8       libraries lapack_atlas not found in /usr/lib
#10 221.8     customize UnixCCompiler
#10 221.8       libraries f77blas,cblas,atlas not found in /usr/lib
#10 221.8     customize UnixCCompiler
#10 221.8       libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
#10 221.8     customize UnixCCompiler
#10 221.8       libraries f77blas,cblas,atlas not found in /usr/lib/x86_64-linux-gnu
#10 221.8     <class 'numpy.distutils.system_info.atlas_info'>
#10 221.8       NOT AVAILABLE
#10 221.8     
#10 221.8     accelerate_info:
#10 221.8       NOT AVAILABLE
#10 221.8     
#10 221.8     lapack_info:
#10 221.8     customize UnixCCompiler
#10 221.8       libraries lapack not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
#10 221.8       NOT AVAILABLE
#10 221.8     
#10 221.8     /tmp/pip-build-env-5cgjibsz/overlay/lib/python3.9/site-packages/numpy/distutils/system_info.py:1712: UserWarning:
#10 221.8         Lapack (http://www.netlib.org/lapack/) libraries not found.
#10 221.8         Directories to search for the libraries can be specified in the
#10 221.8         numpy/distutils/site.cfg file (section [lapack]) or by setting
#10 221.8         the LAPACK environment variable.
#10 221.8       if getattr(self, '_calc_info_{}'.format(lapack))():
#10 221.8     lapack_src_info:
#10 221.8       NOT AVAILABLE
#10 221.8     
#10 221.8     /tmp/pip-build-env-5cgjibsz/overlay/lib/python3.9/site-packages/numpy/distutils/system_info.py:1712: UserWarning:
#10 221.8         Lapack (http://www.netlib.org/lapack/) sources not found.
#10 221.8         Directories to search for the sources can be specified in the
#10 221.8         numpy/distutils/site.cfg file (section [lapack_src]) or by setting
#10 221.8         the LAPACK_SRC environment variable.
#10 221.8       if getattr(self, '_calc_info_{}'.format(lapack))():
#10 221.8       NOT AVAILABLE
#10 221.8     
#10 221.8     Traceback (most recent call last):
#10 221.8       File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
#10 221.8         main()
#10 221.8       File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
#10 221.8         json_out['return_val'] = hook(**hook_input['kwargs'])
#10 221.8       File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 133, in prepare_metadata_for_build_wheel
#10 221.8         return hook(metadata_directory, config_settings)
#10 221.8       File "/tmp/pip-build-env-5cgjibsz/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 161, in prepare_metadata_for_build_wheel
#10 221.8         self.run_setup()
#10 221.8       File "/tmp/pip-build-env-5cgjibsz/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 253, in run_setup
#10 221.8         super(_BuildMetaLegacyBackend,
#10 221.8       File "/tmp/pip-build-env-5cgjibsz/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 145, in run_setup
#10 221.8         exec(compile(code, __file__, 'exec'), locals())
#10 221.8       File "setup.py", line 540, in <module>
#10 221.8         setup_package()
#10 221.8       File "setup.py", line 536, in setup_package
#10 221.8         setup(**metadata)
#10 221.8       File "/tmp/pip-build-env-5cgjibsz/overlay/lib/python3.9/site-packages/numpy/distutils/core.py", line 137, in setup
#10 221.8         config = configuration()
#10 221.8       File "setup.py", line 435, in configuration
#10 221.8         raise NotFoundError(msg)
#10 221.8     numpy.distutils.system_info.NotFoundError: No lapack/blas resources found.
#10 221.8     ----------------------------------------
#10 221.8 WARNING: Discarding https://files.pythonhosted.org/packages/04/ab/e2eb3e3f90b9363040a3d885ccc5c79fe20c5b8a3caa8fe3bf47ff653260/scipy-1.4.1.tar.gz#sha256=dee1bbf3a6c8f73b6b218cb28eed8dd13347ea2f87d572ce19b289d6fd3fbc59 (from https://pypi.org/simple/scipy/) (requires-python:>=3.5). Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpifpjdxhq Check the logs for full command output.
#10 221.8 ERROR: Could not find a version that satisfies the requirement scipy==1.4.1
#10 221.8 ERROR: No matching distribution found for scipy==1.4.1
#10 ERROR: executor failed running [/bin/sh -c pip install -r requirements.txt && mkdir /models && mkdir /images]: exit code: 1
------
 > [5/6] RUN pip install -r requirements.txt && mkdir /models && mkdir /images:
------
Dockerfile:8
--------------------
   6 |     COPY webui/requirements.txt /projects/sce-domain-discovery/webui/
   7 |     
   8 | >>> RUN pip install -r requirements.txt && mkdir /models && mkdir /images
   9 |     
  10 |     COPY . /projects/sce-domain-discovery/
--------------------
error: failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c pip install -r requirements.txt && mkdir /models && mkdir /images]: exit code: 1
Error: buildx call failed with: error: failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c pip install -r requirements.txt && mkdir /models && mkdir /images]: exit code: 1