Unable to run in CI pipeline

[mapattacker]

Hi, I am wondering if anyone tried to use lib this in a CI/CD pipeline.

This is my gitlab-ci job

integration-tests:
  stage: test
  image: python:3.8.3
  before_script:
    - pip install pytest==6.2.3 pytest-docker==0.10.3 requests==2.25.1
  script:
    - pytest tests/integration_tests/ -v -s

My test works in local environment, but in the CI job it seems like it can't detect docker. I even tried added in manual installation for docker & docker-compose but the result is still the same.

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
391command = 'docker-compose -f "/builds/recommendation-engine/product_association/docker-compose-test.yml" -p "pytest5314" up --build -d'
392success_codes = (0,)
393    def execute(command, success_codes=(0,)):
394        """Run a shell command."""
395        try:
396            output = subprocess.check_output(command, stderr=subprocess.STDOUT, shell=True)
397            status = 0
398        except subprocess.CalledProcessError as error:
399            output = error.output or b""
400            status = error.returncode
401            command = error.cmd
402    
403        if status not in success_codes:
404>           raise Exception(
405                'Command {} returned {}: """{}""".'.format(
406                    command, status, output.decode("utf-8")
407                )
408            )
409E           Exception: Command docker-compose -f "/builds/imda_dsl/ai-shopfloor/retail-recommendation-engine/product_association/docker-compose-test.yml" -p "pytest5314" up --build -d returned 1: """Traceback (most recent call last):
410E             File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
411E               httplib_response = self._make_request(
412E             File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 394, in _make_request
413E               conn.request(method, url, **httplib_request_kw)
414E             File "/usr/local/lib/python3.8/http/client.py", line 1240, in request
415E               self._send_request(method, url, body, headers, encode_chunked)
416E             File "/usr/local/lib/python3.8/http/client.py", line 1286, in _send_request
417E               self.endheaders(body, encode_chunked=encode_chunked)
418E             File "/usr/local/lib/python3.8/http/client.py", line 1235, in endheaders
419E               self._send_output(message_body, encode_chunked=encode_chunked)
420E             File "/usr/local/lib/python3.8/http/client.py", line 1006, in _send_output
421E               self.send(msg)
422E             File "/usr/local/lib/python3.8/http/client.py", line 946, in send
423E               self.connect()
424E             File "/usr/local/lib/python3.8/site-packages/docker/transport/unixconn.py", line 30, in connect
425E               sock.connect(self.unix_socket)
426E           FileNotFoundError: [Errno 2] No such file or directory
427E           
428E           During handling of the above exception, another exception occurred:
429E           
430E           Traceback (most recent call last):
431E             File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
432E               resp = conn.urlopen(
433E             File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
434E               retries = retries.increment(
435E             File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 532, in increment
436E               raise six.reraise(type(error), error, _stacktrace)
437E             File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise
438E               raise value.with_traceback(tb)
439E             File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
440E               httplib_response = self._make_request(
441E             File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 394, in _make_request
442E               conn.request(method, url, **httplib_request_kw)
443E             File "/usr/local/lib/python3.8/http/client.py", line 1240, in request
444E               self._send_request(method, url, body, headers, encode_chunked)
445E             File "/usr/local/lib/python3.8/http/client.py", line 1286, in _send_request
446E               self.endheaders(body, encode_chunked=encode_chunked)
447E             File "/usr/local/lib/python3.8/http/client.py", line 1235, in endheaders
448E               self._send_output(message_body, encode_chunked=encode_chunked)
449E             File "/usr/local/lib/python3.8/http/client.py", line 1006, in _send_output
450E               self.send(msg)
451E             File "/usr/local/lib/python3.8/http/client.py", line 946, in send
452E               self.connect()
453E             File "/usr/local/lib/python3.8/site-packages/docker/transport/unixconn.py", line 30, in connect
454E               sock.connect(self.unix_socket)
455E           urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
456E           
457E           During handling of the above exception, another exception occurred:
458E           
459E           Traceback (most recent call last):
460E             File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
461E               return self.version(api_version=False)["ApiVersion"]
462E             File "/usr/local/lib/python3.8/site-packages/docker/api/daemon.py", line 181, in version
463E               return self._result(self._get(url), json=True)
464E             File "/usr/local/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner
465E               return f(self, *args, **kwargs)
466E             File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 237, in _get
467E               return self.get(url, **self._set_request_timeout(kwargs))
468E             File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 555, in get
469E               return self.request('GET', url, **kwargs)
470E             File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
471E               resp = self.send(prep, **send_kwargs)
472E             File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
473E               r = adapter.send(request, **kwargs)
474E             File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
475E               raise ConnectionError(err, request=request)
476E           requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
477E           
478E           During handling of the above exception, another exception occurred:
479E           
480E           Traceback (most recent call last):
481E             File "/usr/local/bin/docker-compose", line 8, in <module>
482E               sys.exit(main())
483E             File "/usr/local/lib/python3.8/site-packages/compose/cli/main.py", line 81, in main
484E               command_func()
485E             File "/usr/local/lib/python3.8/site-packages/compose/cli/main.py", line 200, in perform_command
486E               project = project_from_options('.', options)
487E             File "/usr/local/lib/python3.8/site-packages/compose/cli/command.py", line 60, in project_from_options
488E               return get_project(
489E             File "/usr/local/lib/python3.8/site-packages/compose/cli/command.py", line 152, in get_project
490E               client = get_client(
491E             File "/usr/local/lib/python3.8/site-packages/compose/cli/docker_client.py", line 41, in get_client
492E               client = docker_client(
493E             File "/usr/local/lib/python3.8/site-packages/compose/cli/docker_client.py", line 170, in docker_client
494E               client = APIClient(use_ssh_client=not use_paramiko_ssh, **kwargs)
495E             File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 197, in __init__
496E               self._version = self._retrieve_server_version()
497E             File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
498E               raise DockerException(
499E           docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
500E           """.

Thank you.

[n1ngu]

AFAIU, you need to request the docker service in you job https://docs.gitlab.com/ee/ci/services/#using-services-with-docker-run-docker-in-docker-side-by-side

Same goes for Bitbucket Pipelines https://support.atlassian.com/bitbucket-cloud/docs/run-docker-commands-in-bitbucket-pipelines/

[mapattacker]

thanks @n1ngu, forgot about the dind~

It took make several hours to get the pipeline to work. If anyone is interested:

integration-tests:
  stage: test
  image: docker/compose
  services:
    - docker:dind
  before_script:
    - apk update && apk add py-pip python3-dev libffi-dev openssl-dev gcc libc-dev rust cargo make
    - /usr/bin/python3.8 -m pip install --upgrade pip
  script:
    - pip3 install pytest==6.2.3 requests==2.25.1 pytest-docker==0.10.3 
    - pytest tests/integration_tests/ -v -s