search

spcl / serverless-benchmarks

SeBS: serverless benchmarking suite for automatic performance analysis of FaaS platforms.
https://mcopik.github.io/projects/sebs/
BSD 3-Clause "New" or "Revised" License
150 stars 68 forks source link

socket.timeout: The write operation timed out - regression test on gcp #155

Closed jchigu closed 1 year ago

jchigu commented 1 year ago

I have done all the configurations for service accounts and creating a VM and network and setting organisational policies and the previous errors have disappeared. now experiencing a timeout error. see the following logs:\


[17:18:17.917912] SeBS-2391 Created experiment output at /root/serverless-benchmarks/regression-output
Select test test_110.dynamic-html
Select test test_120.uploader
Select test test_210.thumbnailer
Select test test_220.video-processing
Select test test_311.compression
Select test test_411.image-recognition
Select test test_501.graph-pagerank
Select test test_502.graph-mst
Select test test_503.graph-bfs
Select test test_504.dna-visualisation
[17:18:18.902908] GCPCredentials-0463 No cached credentials for GCP found, initialize!
[17:18:18.912193] GCPResources-dc4b No cached resources for GCP found, using user configuration.
[17:18:18.909665] GCPCredentials-f09c No cached credentials for GCP found, initialize!
[17:18:18.910959] GCPCredentials-a55c No cached credentials for GCP found, initialize!
[17:18:18.913519] GCPResources-9a61 No cached resources for GCP found, using user configuration.
[17:18:18.913792] GCPConfig-48bd Using user-provided config for GCP
[17:18:18.911359] GCPCredentials-6bf4 No cached credentials for GCP found, initialize!
[17:18:18.916666] GCPResources-4fef No cached resources for GCP found, using user configuration.
[17:18:18.916967] GCPConfig-820c Using user-provided config for GCP
[17:18:18.903536] GCPCredentials-72c8 No cached credentials for GCP found, initialize!
[17:18:18.917762] GCPResources-13f8 No cached resources for GCP found, using user configuration.
[17:18:18.918033] GCPConfig-9388 Using user-provided config for GCP
[17:18:18.909216] GCPCredentials-d25b No cached credentials for GCP found, initialize!
[17:18:18.918806] GCPResources-4c9d No cached resources for GCP found, using user configuration.
[17:18:18.919072] GCPConfig-99d9 Using user-provided config for GCP
[17:18:18.910077] GCPCredentials-0032 No cached credentials for GCP found, initialize!
[17:18:18.919831] GCPResources-6d5b No cached resources for GCP found, using user configuration.
[17:18:18.920100] GCPConfig-5025 Using user-provided config for GCP
[17:18:18.910539] GCPCredentials-f204 No cached credentials for GCP found, initialize!
[17:18:18.920996] GCPResources-65e1 No cached resources for GCP found, using user configuration.
[17:18:18.921255] GCPConfig-dde3 Using user-provided config for GCP
[17:18:18.911659] GCPCredentials-d63e No cached credentials for GCP found, initialize!
[17:18:18.922027] GCPResources-6cd3 No cached resources for GCP found, using user configuration.
[17:18:18.922296] GCPConfig-6021 Using user-provided config for GCP
[17:18:18.913226] GCPResources-ea09 No cached resources for GCP found, using user configuration.
[17:18:18.923038] GCPConfig-32fc Using user-provided config for GCP
[17:18:18.923973] GCPConfig-816f Using user-provided config for GCP
[17:18:18.924820] GCPCredentials-aada No cached credentials for GCP found, initialize!
[17:18:18.925092] GCPResources-4975 No cached resources for GCP found, using user configuration.
[17:18:18.956617] GCPConfig-7c21 Using user-provided config for GCP
Begin regression test of 502.graph-mst on gcp, region: west-europe1.
Begin regression test of 501.graph-pagerank on gcp, region: west-europe1.
Begin regression test of 110.dynamic-html on gcp, region: west-europe1.
Begin regression test of 311.compression on gcp, region: west-europe1.
[17:19:32.134888] Benchmark-7c42 Using cached benchmark 502.graph-mst at /root/serverless-benchmarks/regression-cache/502.graph-mst/gcp/python/3.7/502.graph-mst.zip
[17:19:32.138145] Benchmark-50f2 Using cached benchmark 311.compression at /root/serverless-benchmarks/regression-cache/311.compression/gcp/python/3.7/311.compression.zip
[17:19:32.138746] Benchmark-c6a6 Using cached benchmark 501.graph-pagerank at /root/serverless-benchmarks/regression-cache/501.graph-pagerank/gcp/python/3.7/501.graph-pagerank.zip
[17:19:32.152557] GCP-f3af Creating new function! Reason: function function-311_compression_python_3_7 not found in cache.
[17:19:32.139929] Benchmark-6931 Using cached benchmark 110.dynamic-html at /root/serverless-benchmarks/regression-cache/110.dynamic-html/gcp/python/3.7/110.dynamic-html.zip
[17:19:32.176822] GCP-1f09 Creating new function! Reason: function function-110_dynamic_html_python_3_7 not found in cache.
[17:19:32.175891] GCP-cfdd Creating new function! Reason: function function-501_graph_pagerank_python_3_7 not found in cache.
[17:19:32.148469] GCP-9c80 Creating new function! Reason: function function-502_graph_mst_python_3_7 not found in cache.
Begin regression test of 120.uploader on gcp, region: west-europe1.
Begin regression test of 503.graph-bfs on gcp, region: west-europe1.
[17:19:38.641698] Benchmark-af71 Using cached benchmark 503.graph-bfs at /root/serverless-benchmarks/regression-cache/503.graph-bfs/gcp/python/3.7/503.graph-bfs.zip
Begin regression test of 504.dna-visualisation on gcp, region: west-europe1.
[17:19:38.675163] Benchmark-87b2 Using cached benchmark 504.dna-visualisation at /root/serverless-benchmarks/regression-cache/504.dna-visualisation/gcp/python/3.7/504.dna-visualisation.zip
[17:19:38.676138] GCP-46a4 Creating new function! Reason: function function-504_dna_visualisation_python_3_7 not found in cache.
[17:19:38.708682] GCP-f2db Creating new function! Reason: function function-503_graph_bfs_python_3_7 not found in cache.
[17:19:38.681471] Benchmark-59fd Using cached benchmark 120.uploader at /root/serverless-benchmarks/regression-cache/120.uploader/gcp/python/3.7/120.uploader.zip
[17:19:38.717854] GCP-972e Creating new function! Reason: function function-120_uploader_python_3_7 not found in cache.
Begin regression test of 210.thumbnailer on gcp, region: west-europe1.
[17:19:39.086609] Benchmark-9def Using cached benchmark 210.thumbnailer at /root/serverless-benchmarks/regression-cache/210.thumbnailer/gcp/python/3.7/210.thumbnailer.zip
[17:19:39.140854] GCP-90fe Creating new function! Reason: function function-210_thumbnailer_python_3_7 not found in cache.
Begin regression test of 411.image-recognition on gcp, region: west-europe1.
Begin regression test of 220.video-processing on gcp, region: west-europe1.
[17:19:42.307743] Benchmark-09a0 Using cached benchmark 220.video-processing at /root/serverless-benchmarks/regression-cache/220.video-processing/gcp/python/3.7/220.video-processing.zip
[17:19:42.308512] Benchmark-a92a Using cached benchmark 411.image-recognition at /root/serverless-benchmarks/regression-cache/411.image-recognition/gcp/python/3.7/411.image-recognition.zip
[17:19:42.453461] GCP-79b4 Creating new function! Reason: function function-220_video_processing_python_3_7 not found in cache.
[17:19:42.458417] GCP-ac7c Creating new function! Reason: function function-411_image_recognition_python_3_7 not found in cache.

-------------

sebs.regression.GCPTestSequence.test_501.graph-pagerank: fail
sebs.regression.GCPTestSequence.test_501.graph-pagerank: Traceback (most recent call last):
  File "/root/serverless-benchmarks/python-venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/root/serverless-benchmarks/python-venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/root/serverless-benchmarks/python-venv/lib/python3.8/site-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.8/http/client.py", line 1256, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1050, in _send_output
    self.send(chunk)
  File "/usr/lib/python3.8/http/client.py", line 972, in send
    self.sock.sendall(data)
  File "/usr/lib/python3.8/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "/usr/lib/python3.8/ssl.py", line 1173, in send
    return self._sslobj.write(data)
socket.timeout: The write operation timed out```
mcopik commented 1 year ago

@jchigu Did you solve the issue? Unfortunately, we did not have the capacity to investigate this issue in depth.

Looks like another SSL error - the failure happens within a request to the Google cloud.

mcopik commented 1 year ago

@jchigu Does the issue still persist?