Unexpected error - An error occured () when pulling/fetching from docker container

[VodopaDev]

Hi!

I have been using k8s for a side project that uses git and dvc. When using dvc from the k8s host, I have no trouble getting data from the dvc s3-backend (which is also in the k8s cluster).
But the Jenkins docker container cannot pull or fetch data from the s3 backend. I assume the env variables are correct as the logs below show that dvc has no error at the Collecting information from remote cache... step.

What is really strange is that dvc fails without a specific error (()).
I also ran the command without --run-cache and outcome.
Does someone as an idea of what I could do?

root@jenkins-78cd7f8696-nmsk5:~/mlops_code# dvc pull --run-cache -v

2021-03-17 11:17:14,705 DEBUG: Check for update is enabled.
2021-03-17 11:17:14,866 DEBUG: Preparing to download data from 's3://dvc'
2021-03-17 11:17:14,866 DEBUG: Preparing to collect status from s3://dvc
2021-03-17 11:17:14,867 DEBUG: Collecting information from local cache...
2021-03-17 11:17:14,868 DEBUG: Collecting information from remote cache...
2021-03-17 11:17:14,869 DEBUG: Matched '0' indexed hashes
2021-03-17 11:17:14,869 DEBUG: Querying 1 hashes via object_exists
2021-03-17 11:17:14,970 ERROR: unexpected error - An error occurred () when calling the ListObjects operation:
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/dvc/output/base.py", line 496, in get_dir_cache
    objects.check(self.odb, self.odb.get(self.hash_info))
  File "/usr/local/lib/python3.7/dist-packages/dvc/objects/__init__.py", line 18, in check
    odb.check(obj.hash_info)
  File "/usr/local/lib/python3.7/dist-packages/dvc/objects/db/base.py", line 109, in check
    obj.check(self)
  File "/usr/local/lib/python3.7/dist-packages/dvc/objects/file.py", line 41, in check
    self.path_info, self.fs, self.hash_info.name, odb.repo.state
  File "/usr/local/lib/python3.7/dist-packages/dvc/objects/stage.py", line 40, in get_file_hash
    errno.ENOENT, os.strerror(errno.ENOENT), path_info
FileNotFoundError: [Errno 2] No such file or directory: '/root/mlops_code/.dvc/cache/75/62d607a0d1b2c3517353062afa1d7d.dir'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/dvc/main.py", line 50, in main
    ret = cmd.run()
  File "/usr/local/lib/python3.7/dist-packages/dvc/command/data_sync.py", line 40, in run
    glob=self.args.glob,
  File "/usr/local/lib/python3.7/dist-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/dvc/repo/pull.py", line 38, in pull
    run_cache=run_cache,
  File "/usr/local/lib/python3.7/dist-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/dvc/repo/fetch.py", line 53, in fetch
    revs=revs,
  File "/usr/local/lib/python3.7/dist-packages/dvc/repo/__init__.py", line 397, in used_cache
    filter_info=filter_info,
  File "/usr/local/lib/python3.7/dist-packages/dvc/stage/__init__.py", line 621, in get_used_cache
    cache.update(out.get_used_cache(*args, **kwargs))
  File "/usr/local/lib/python3.7/dist-packages/dvc/output/base.py", line 611, in get_used_cache
    self.hash_info.value, self.collect_used_dir_cache(**kwargs),
  File "/usr/local/lib/python3.7/dist-packages/dvc/output/base.py", line 536, in collect_used_dir_cache
    self.get_dir_cache(jobs=jobs, remote=remote)
  File "/usr/local/lib/python3.7/dist-packages/dvc/output/base.py", line 501, in get_dir_cache
    **kwargs,
  File "/usr/local/lib/python3.7/dist-packages/dvc/data_cloud.py", line 92, in pull
    show_checksums=show_checksums,
  File "/usr/local/lib/python3.7/dist-packages/dvc/remote/base.py", line 56, in wrapper
    return f(obj, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/dvc/remote/base.py", line 491, in pull
    download=True,
  File "/usr/local/lib/python3.7/dist-packages/dvc/remote/base.py", line 328, in _process
    download=download,
  File "/usr/local/lib/python3.7/dist-packages/dvc/remote/base.py", line 176, in _status
    md5s, jobs=jobs, name=str(self.fs.path_info)
  File "/usr/local/lib/python3.7/dist-packages/dvc/remote/base.py", line 132, in hashes_exist
    return indexed_hashes + self.odb.hashes_exist(list(hashes), **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/dvc/objects/db/base.py", line 380, in hashes_exist
    remote_hashes = self.list_hashes_exists(hashes, jobs, name)
  File "/usr/local/lib/python3.7/dist-packages/dvc/objects/db/base.py", line 338, in list_hashes_exists
    ret = list(itertools.compress(hashes, in_remote))
  File "/usr/lib/python3.7/concurrent/futures/_base.py", line 586, in result_iterator
    yield fs.pop().result()
  File "/usr/lib/python3.7/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.7/dist-packages/dvc/objects/db/base.py", line 329, in exists_with_progress
    ret = self.fs.exists(path_info)
  File "/usr/local/lib/python3.7/dist-packages/dvc/fs/s3.py", line 231, in exists
    return self.isfile(path_info) or self.isdir(path_info)
  File "/usr/local/lib/python3.7/dist-packages/dvc/fs/s3.py", line 261, in isfile
    return path_info.path in self._list_paths(path_info)
  File "/usr/local/lib/python3.7/dist-packages/dvc/fs/s3.py", line 282, in _list_paths
    for obj_summary in obj_summaries:
  File "/usr/local/lib/python3.7/dist-packages/boto3/resources/collection.py", line 83, in __iter__
    for page in self.pages():
  File "/usr/local/lib/python3.7/dist-packages/boto3/resources/collection.py", line 166, in pages
    for page in pages:
  File "/usr/local/lib/python3.7/dist-packages/botocore/paginate.py", line 255, in __iter__
    response = self._make_request(current_kwargs)
  File "/usr/local/lib/python3.7/dist-packages/botocore/paginate.py", line 332, in _make_request
    return self._method(**current_kwargs)
  File "/usr/local/lib/python3.7/dist-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.7/dist-packages/botocore/client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred () when calling the ListObjects operation:
------------------------------------------------------------
2021-03-17 11:17:15,009 DEBUG: Version info for developers:
DVC version: 2.0.6 (pip)
---------------------------------
Platform: Python 3.7.3 on Linux-4.19.121-linuxkit-x86_64-with-debian-10.8
Supports: http, https, s3
Cache types: hardlink, symlink
Cache directory: overlay on overlay
Caches: local
Remotes: s3
Workspace directory: overlay on overlay
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2021-03-17 11:17:15,011 DEBUG: Analytics is enabled.
2021-03-17 11:17:15,048 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmp1531lprl']'
2021-03-17 11:17:15,050 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmp1531lprl']'

[VodopaDev]

Ups I posted the issue in the wrong repo.
Sorry!