Timeout when running a local experiment

[maltewae]

I get a Connection refused Error when I try to run an experiment locally. An out.json file is not created. I executed the following command: ./sebs.py local start 110.dynamic-html test out.json --config config/example.json --deployments 1

Based on the traceback the method list_buckets() does not work.

My system properties:

• Architecture: arm64 or x86_64 (emulated with Rosetta)
• OS: macOS 12.2.1 (21D62)
• Docker version 20.10.12
• Python version 3.7.13

SeBS version: https://github.com/spcl/serverless-benchmarks/commit/9dcbcc94de53de40d37595ced09006a4639981ac

Out.log:

17:44:52,337 INFO SeBS-aea0: Created experiment output at /Users/maltewae/src/github.com/spcl/serverless-benchmarks
17:44:52,338 INFO Local.Minio-8704: Minio storage ACCESS_KEY=...
17:44:52,338 INFO Local.Minio-8704: Minio storage SECRET_KEY=...
17:44:52,666 INFO Local.Minio-8704: Starting minio instance at 172.17.0.2:9000
17:46:54,421 INFO Local.Minio-8704: Stopping minio container at 172.17.0.2:9000
17:46:55,86 INFO Local.Minio-8704: Stopped minio container at 172.17.0.2:9000

Traceback:

ERROR:root:HTTPConnectionPool(host='172.17.0.2', port=9000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x14cb6e250>: Failed to establish a new connection: [Errno 61] Connection refused'))
Traceback (most recent call last):
  File "/Users/maltewae/src/github.com/spcl/serverless-benchmarks/python-venv/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/Users/maltewae/src/github.com/spcl/serverless-benchmarks/python-venv/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/Users/maltewae/src/github.com/spcl/serverless-benchmarks/python-venv/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/maltewae/src/github.com/spcl/serverless-benchmarks/python-venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
    chunked=chunked,
  File "/Users/maltewae/src/github.com/spcl/serverless-benchmarks/python-venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/maltewae/src/github.com/spcl/serverless-benchmarks/python-venv/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/Cellar/python@3.7/3.7.13/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1281, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/Cellar/python@3.7/3.7.13/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1327, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python@3.7/3.7.13/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1276, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python@3.7/3.7.13/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/local/Cellar/python@3.7/3.7.13/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 976, in send
    self.connect()
  File "/Users/maltewae/src/github.com/spcl/serverless-benchmarks/python-venv/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/Users/maltewae/src/github.com/spcl/serverless-benchmarks/python-venv/lib/python3.7/site-packages/urllib3/connection.py", line 187, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x14cb6e250>: Failed to establish a new connection: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./sebs.py", line 30, in __call__
    return self.main(*args, **kwargs)
  File "/Users/maltewae/src/github.com/spcl/serverless-benchmarks/python-venv/lib/python3.7/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/maltewae/src/github.com/spcl/serverless-benchmarks/python-venv/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/maltewae/src/github.com/spcl/serverless-benchmarks/python-venv/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/maltewae/src/github.com/spcl/serverless-benchmarks/python-venv/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/maltewae/src/github.com/spcl/serverless-benchmarks/python-venv/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "./sebs.py", line 77, in wrapper
    return func(*args, **kwargs)
  File "./sebs.py", line 357, in start
    storage=storage, size=benchmark_input_size
  File "/Users/maltewae/src/github.com/spcl/serverless-benchmarks/sebs/benchmark.py", line 517, in prepare_input
    storage.allocate_buckets(self.benchmark, buckets)
  File "/Users/maltewae/src/github.com/spcl/serverless-benchmarks/sebs/faas/storage.py", line 184, in allocate_buckets
    buckets = self.list_buckets(self.correct_name(benchmark))
  File "/Users/maltewae/src/github.com/spcl/serverless-benchmarks/sebs/local/storage.py", line 155, in list_buckets
    buckets = self.connection.list_buckets()
  File "/Users/maltewae/src/github.com/spcl/serverless-benchmarks/python-venv/lib/python3.7/site-packages/minio/api.py", line 406, in list_buckets
    headers=headers)
  File "/Users/maltewae/src/github.com/spcl/serverless-benchmarks/python-venv/lib/python3.7/site-packages/urllib3/poolmanager.py", line 376, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/Users/maltewae/src/github.com/spcl/serverless-benchmarks/python-venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 826, in urlopen
    **response_kw
  File "/Users/maltewae/src/github.com/spcl/serverless-benchmarks/python-venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 826, in urlopen
    **response_kw
  File "/Users/maltewae/src/github.com/spcl/serverless-benchmarks/python-venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 826, in urlopen
    **response_kw
  [Previous line repeated 2 more times]
  File "/Users/maltewae/src/github.com/spcl/serverless-benchmarks/python-venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 786, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/Users/maltewae/src/github.com/spcl/serverless-benchmarks/python-venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.17.0.2', port=9000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x14cb6e250>: Failed to establish a new connection: [Errno 61] Connection refused'))

[mcopik]

@maltewae Thanks! Can you tell me what is the version of Docker image running minio?

[maltewae]

My minio version: RELEASE.2022-03-22T02-05-10Z

[mcopik]

@maltewae Can you also tell me the hash of Docker image? I'm wondering if you're running amd64 or arm64 version.

[maltewae]

16a0b1807bf9c7f1c2ea8558ae067a0352614e26bd4e6a898688a16d30747e31

[wowu]

This timeout is caused by Docker container IPs being not reachable on macOS.

Per-container IP addressing is not possible The docker (Linux) bridge network is not reachable from the macOS host

https://docs.docker.com/desktop/mac/networking/#there-is-no-docker0-bridge-on-macos

For host <-> minio communication port forwarding should be used instead.

[mcopik]

@Wowu Thank you for your comment! I'm not an expert in MacOS and I could have missed that.

@maltewae Can you please verify it solved the issue? BTW, storage's deployment changed on master branch since our last discussion - see the README.

https://github.com/spcl/serverless-benchmarks/blob/master/sebs/storage/minio.py#L94

[wowu]

I managed to fix the timeout error by patching sebs/storage/minio.py:

diff --git a/sebs/storage/minio.py b/sebs/storage/minio.py
index 6c79f05..d2f12c4 100644
--- a/sebs/storage/minio.py
+++ b/sebs/storage/minio.py
@@ -115,7 +115,7 @@ class Minio(PersistentStorage):

     def get_connection(self):
         return minio.Minio(
-            self._cfg.address,
+            f"localhost:{self._cfg.mapped_port}",
             access_key=self._cfg.access_key,
             secret_key=self._cfg.secret_key,
             secure=False,

The point is to use port mapping for host connections on macOS. I suggest two solutions:

1. Use localhost:{mapped_port} instead of container address by default on every machine if it doesn't break anything
2. Use localhost:{mapped_port} instead of container address only if detected OS is macOS.

[mcopik]

The issue has been fixed by @Kaleab-git some time ago.