search

spcl / serverless-benchmarks

SeBS: serverless benchmarking suite for automatic performance analysis of FaaS platforms.
https://mcopik.github.io/projects/sebs/
BSD 3-Clause "New" or "Revised" License
149 stars 67 forks source link

Erroneous installation of Docker dependency #105

Closed mcopik closed 1 year ago

mcopik commented 1 year ago

The bug was raised via email by Justin Chigu.

(python-virtualenv) justinchigu@justinchigu-VirtualBox:~/serverless-benchmarks$ ./sebs.py benchmark invoke 110.dynamic-html test --config config/example.json --deployment aws --verbose
Traceback (most recent call last):
  File "/home/justinchigu/serverless-benchmarks/./sebs.py", line 13, in <module>
    import sebs
  File "/home/justinchigu/serverless-benchmarks/sebs/__init__.py", line 6, in <module>
    from .sebs import SeBS  # noqa
  File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 7, in <module>
    from sebs.local import Local
  File "/home/justinchigu/serverless-benchmarks/sebs/local/__init__.py", line 1, in <module>
    from .local import Local  # noqa
  File "/home/justinchigu/serverless-benchmarks/sebs/local/local.py", line 10, in <module>
    from sebs.local.config import LocalConfig
  File "/home/justinchigu/serverless-benchmarks/sebs/local/config.py", line 4, in <module>
    from sebs.faas.config import Config, Credentials, Resources
  File "/home/justinchigu/serverless-benchmarks/sebs/faas/__init__.py", line 1, in <module>
    from .system import System  # noqa
  File "/home/justinchigu/serverless-benchmarks/sebs/faas/system.py", line 8, in <module>
    from sebs.benchmark import Benchmark
  File "/home/justinchigu/serverless-benchmarks/sebs/benchmark.py", line 73, in <module>
    class Benchmark(LoggingBase):
  File "/home/justinchigu/serverless-benchmarks/sebs/benchmark.py", line 158, in Benchmark
    docker_client: docker.client,
AttributeError: module 'docker' has no attribute 'client'
(python-virtualenv) justinchigu@justinchigu-VirtualBox:~/serverless-benchmarks$

This looks oddly similar to prior issues when the package docker-py has been installed instead of `docker.

mcopik commented 1 year ago

@jchigu The error mentioned in the other issue (please keep all correspondence in one issue, thanks!) seems to suggest that your installation is broken:

Deployment aws not supported!

Did you run the full command ./install.py --aws --azure --gcp --openwhisk --local? Did it succeed?

Furthermore, the new log you posted indicates that you no longer experience the issue with the docker library. Do you know what resolved the issue?

mcopik commented 1 year ago

@jchigu Regarding the Docker problem, there was a potential issue that should be resolved on the dev branch in commit ab50a1a.

Regarding the current problem, I was not able to reproduce it - can you please post the last five lines of the following file: python-venv/bin/activate? It should look similar to this:

tail -n 5 python-venv/bin/activate
fi
export SEBS_WITH_AWS=TRUE
export SEBS_WITH_AZURE=TRUE
export SEBS_WITH_GCP=TRUE
export SEBS_WITH_OPENWHISK=TRUE
mcopik commented 1 year ago

@jchigu The environment you created is local and will not change overnight, so there is no need to recheck it.

Based on your comments from two days ago, you have two different environments created - python-venv created by our installation script and a different environment called python-virtualenv that you have activated. The former looks correct - all platforms should be activated. I realized that there was a typo in our documentation that might have created confusion and led to using multiple environments.

Can you please remove existing virtual environments and repeat the installation step as outlined in our documentation. I think it should work correctly now :-)

jchigu commented 1 year ago

Thank you. Now the installation error is solved.

May I now know the next steps that I need to follow. I have been advised to run all the benchmarks locally on my machine, so what steps do I need to take.

Here are some sample outputs that I am getting

OUTPUT 1 FOR AWS (python-venv) justinchigu@justinchigu-VirtualBox:~/serverless-benchmarks$ ./sebs.py benchmark regression test --config config/example.json --deployment aws Select test test_110.dynamic-html Select test test_120.uploader Select test test_210.thumbnailer Select test test_220.video-processing Select test test_311.compression Select test test_411.image-recognition Select test test_501.graph-pagerank Select test test_502.graph-mst Select test test_503.graph-bfs Select test test_504.dna-visualisation Begin regression test of 311.compression on aws, region: us-east-1. Begin regression test of 501.graph-pagerank on aws, region: us-east-1. Begin regression test of 504.dna-visualisation on aws, region: us-east-1. Begin regression test of 110.dynamic-html on aws, region: us-east-1. Begin regression test of 502.graph-mst on aws, region: us-east-1. Begin regression test of 210.thumbnailer on aws, region: us-east-1. Begin regression test of 411.image-recognition on aws, region: us-east-1. Begin regression test of 503.graph-bfs on aws, region: us-east-1. Begin regression test of 120.uploader on aws, region: us-east-1. Begin regression test of 220.video-processing on aws, region: us-east-1.

sebs.regression.AWSTestSequence.test_501.graph-pagerank: fail sebs.regression.AWSTestSequence.test_501.graph-pagerank: Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 268, in _raise_for_status response.raise_for_status() File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/images/spcleth/serverless-benchmarks:build.aws.python.3.7/json

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/sebs/benchmark.py", line 351, in install_dependencies self._docker_client.images.get(repo_name + ":" + image_name) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/models/images.py", line 335, in get return self.prepare_model(self.client.api.inspect_image(name)) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/utils/decorators.py", line 19, in wrapped return f(self, resource_id, *args, **kwargs) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/image.py", line 251, in inspect_image return self._result( File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 274, in _result self._raise_for_status(response) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 270, in _raise_for_status raise create_api_error_from_http_exception(e) from e File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception raise cls(e, response=response, explanation=explanation) from e docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.41/images/spcleth/serverless-benchmarks:build.aws.python.3.7/json: Not Found ("no such image: spcleth/serverless-benchmarks:build.aws.python.3.7: No such image: spcleth/serverless-benchmarks:build.aws.python.3.7")

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 268, in _raise_for_status response.raise_for_status() File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.41/images/create?tag=build.aws.python.3.7&fromImage=spcleth%2Fserverless-benchmarks

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/sebs/benchmark.py", line 359, in install_dependencies self._docker_client.images.pull(repo_name, image_name) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/models/images.py", line 465, in pull pull_log = self.client.api.pull( File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/image.py", line 429, in pull self._raise_for_status(response) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 270, in _raise_for_status raise create_api_error_from_http_exception(e) from e File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception raise cls(e, response=response, explanation=explanation) from e docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.41/images/create?tag=build.aws.python.3.7&fromImage=spcleth%2Fserverless-benchmarks: Internal Server Error ("Head "https://registry-1.docker.io/v2/spcleth/serverless-benchmarks/manifests/build.aws.python.3.7": read tcp 10.0.2.15:34386->44.205.64.79:443: read: connection reset by peer")

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 50, in test func = deployment_client.get_function( File "/home/justinchigu/serverless-benchmarks/sebs/faas/system.py", line 157, in getfunction rebuilt, = code_package.build(self.package_code) File "/home/justinchigu/serverless-benchmarks/sebs/benchmark.py", line 499, in build self.install_dependencies(self._output_dir) File "/home/justinchigu/serverless-benchmarks/sebs/benchmark.py", line 361, in install_dependencies raise RuntimeError("Docker pull of image {} failed!".format(image_name)) RuntimeError: Docker pull of image build.aws.python.3.7 failed!

sebs.regression.AWSTestSequence.test_411.image-recognition: fail sebs.regression.AWSTestSequence.test_411.image-recognition: Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 268, in _raise_for_status response.raise_for_status() File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/images/spcleth/serverless-benchmarks:build.aws.python.3.7/json

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/sebs/benchmark.py", line 351, in install_dependencies self._docker_client.images.get(repo_name + ":" + image_name) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/models/images.py", line 335, in get return self.prepare_model(self.client.api.inspect_image(name)) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/utils/decorators.py", line 19, in wrapped return f(self, resource_id, *args, **kwargs) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/image.py", line 251, in inspect_image return self._result( File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 274, in _result self._raise_for_status(response) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 270, in _raise_for_status raise create_api_error_from_http_exception(e) from e File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception raise cls(e, response=response, explanation=explanation) from e docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.41/images/spcleth/serverless-benchmarks:build.aws.python.3.7/json: Not Found ("no such image: spcleth/serverless-benchmarks:build.aws.python.3.7: No such image: spcleth/serverless-benchmarks:build.aws.python.3.7")

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 268, in _raise_for_status response.raise_for_status() File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.41/images/create?tag=build.aws.python.3.7&fromImage=spcleth%2Fserverless-benchmarks

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/sebs/benchmark.py", line 359, in install_dependencies self._docker_client.images.pull(repo_name, image_name) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/models/images.py", line 465, in pull pull_log = self.client.api.pull( File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/image.py", line 429, in pull self._raise_for_status(response) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 270, in _raise_for_status raise create_api_error_from_http_exception(e) from e File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception raise cls(e, response=response, explanation=explanation) from e docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.41/images/create?tag=build.aws.python.3.7&fromImage=spcleth%2Fserverless-benchmarks: Internal Server Error ("Head "https://registry-1.docker.io/v2/spcleth/serverless-benchmarks/manifests/build.aws.python.3.7": read tcp 10.0.2.15:44558->44.205.64.79:443: read: connection reset by peer")

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 50, in test func = deployment_client.get_function( File "/home/justinchigu/serverless-benchmarks/sebs/faas/system.py", line 157, in getfunction rebuilt, = code_package.build(self.package_code) File "/home/justinchigu/serverless-benchmarks/sebs/benchmark.py", line 499, in build self.install_dependencies(self._output_dir) File "/home/justinchigu/serverless-benchmarks/sebs/benchmark.py", line 361, in install_dependencies raise RuntimeError("Docker pull of image {} failed!".format(image_name)) RuntimeError: Docker pull of image build.aws.python.3.7 failed!

sebs.regression.AWSTestSequence.test_210.thumbnailer: fail sebs.regression.AWSTestSequence.test_210.thumbnailer: Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 268, in _raise_for_status response.raise_for_status() File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/images/spcleth/serverless-benchmarks:build.aws.python.3.7/json

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/sebs/benchmark.py", line 351, in install_dependencies self._docker_client.images.get(repo_name + ":" + image_name) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/models/images.py", line 335, in get return self.prepare_model(self.client.api.inspect_image(name)) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/utils/decorators.py", line 19, in wrapped return f(self, resource_id, *args, **kwargs) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/image.py", line 251, in inspect_image return self._result( File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 274, in _result self._raise_for_status(response) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 270, in _raise_for_status raise create_api_error_from_http_exception(e) from e File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception raise cls(e, response=response, explanation=explanation) from e docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.41/images/spcleth/serverless-benchmarks:build.aws.python.3.7/json: Not Found ("no such image: spcleth/serverless-benchmarks:build.aws.python.3.7: No such image: spcleth/serverless-benchmarks:build.aws.python.3.7")

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 268, in _raise_for_status response.raise_for_status() File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.41/images/create?tag=build.aws.python.3.7&fromImage=spcleth%2Fserverless-benchmarks

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/sebs/benchmark.py", line 359, in install_dependencies self._docker_client.images.pull(repo_name, image_name) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/models/images.py", line 465, in pull pull_log = self.client.api.pull( File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/image.py", line 429, in pull self._raise_for_status(response) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 270, in _raise_for_status raise create_api_error_from_http_exception(e) from e File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception raise cls(e, response=response, explanation=explanation) from e docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.41/images/create?tag=build.aws.python.3.7&fromImage=spcleth%2Fserverless-benchmarks: Internal Server Error ("Head "https://registry-1.docker.io/v2/spcleth/serverless-benchmarks/manifests/build.aws.python.3.7": read tcp 10.0.2.15:44572->44.205.64.79:443: read: connection reset by peer")

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 50, in test func = deployment_client.get_function( File "/home/justinchigu/serverless-benchmarks/sebs/faas/system.py", line 157, in getfunction rebuilt, = code_package.build(self.package_code) File "/home/justinchigu/serverless-benchmarks/sebs/benchmark.py", line 499, in build self.install_dependencies(self._output_dir) File "/home/justinchigu/serverless-benchmarks/sebs/benchmark.py", line 361, in install_dependencies raise RuntimeError("Docker pull of image {} failed!".format(image_name)) RuntimeError: Docker pull of image build.aws.python.3.7 failed!

sebs.regression.AWSTestSequence.test_120.uploader: fail sebs.regression.AWSTestSequence.test_120.uploader: Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 268, in _raise_for_status response.raise_for_status() File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/images/spcleth/serverless-benchmarks:build.aws.python.3.7/json

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/sebs/benchmark.py", line 351, in install_dependencies self._docker_client.images.get(repo_name + ":" + image_name) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/models/images.py", line 335, in get return self.prepare_model(self.client.api.inspect_image(name)) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/utils/decorators.py", line 19, in wrapped return f(self, resource_id, *args, **kwargs) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/image.py", line 251, in inspect_image return self._result( File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 274, in _result self._raise_for_status(response) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 270, in _raise_for_status raise create_api_error_from_http_exception(e) from e File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception raise cls(e, response=response, explanation=explanation) from e docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.41/images/spcleth/serverless-benchmarks:build.aws.python.3.7/json: Not Found ("no such image: spcleth/serverless-benchmarks:build.aws.python.3.7: No such image: spcleth/serverless-benchmarks:build.aws.python.3.7")

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 268, in _raise_for_status response.raise_for_status() File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.41/images/create?tag=build.aws.python.3.7&fromImage=spcleth%2Fserverless-benchmarks

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/sebs/benchmark.py", line 359, in install_dependencies self._docker_client.images.pull(repo_name, image_name) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/models/images.py", line 465, in pull pull_log = self.client.api.pull( File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/image.py", line 429, in pull self._raise_for_status(response) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 270, in _raise_for_status raise create_api_error_from_http_exception(e) from e File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception raise cls(e, response=response, explanation=explanation) from e docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.41/images/create?tag=build.aws.python.3.7&fromImage=spcleth%2Fserverless-benchmarks: Internal Server Error ("Head "https://registry-1.docker.io/v2/spcleth/serverless-benchmarks/manifests/build.aws.python.3.7": read tcp 10.0.2.15:56812->34.205.13.154:443: read: connection reset by peer")

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 50, in test func = deployment_client.get_function( File "/home/justinchigu/serverless-benchmarks/sebs/faas/system.py", line 157, in getfunction rebuilt, = code_package.build(self.package_code) File "/home/justinchigu/serverless-benchmarks/sebs/benchmark.py", line 499, in build self.install_dependencies(self._output_dir) File "/home/justinchigu/serverless-benchmarks/sebs/benchmark.py", line 361, in install_dependencies raise RuntimeError("Docker pull of image {} failed!".format(image_name)) RuntimeError: Docker pull of image build.aws.python.3.7 failed!

sebs.regression.AWSTestSequence.test_503.graph-bfs: fail sebs.regression.AWSTestSequence.test_503.graph-bfs: Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 268, in _raise_for_status response.raise_for_status() File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/images/spcleth/serverless-benchmarks:build.aws.python.3.7/json

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/sebs/benchmark.py", line 351, in install_dependencies self._docker_client.images.get(repo_name + ":" + image_name) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/models/images.py", line 335, in get return self.prepare_model(self.client.api.inspect_image(name)) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/utils/decorators.py", line 19, in wrapped return f(self, resource_id, *args, **kwargs) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/image.py", line 251, in inspect_image return self._result( File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 274, in _result self._raise_for_status(response) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 270, in _raise_for_status raise create_api_error_from_http_exception(e) from e File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception raise cls(e, response=response, explanation=explanation) from e docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.41/images/spcleth/serverless-benchmarks:build.aws.python.3.7/json: Not Found ("no such image: spcleth/serverless-benchmarks:build.aws.python.3.7: No such image: spcleth/serverless-benchmarks:build.aws.python.3.7")

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 268, in _raise_for_status response.raise_for_status() File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.41/images/create?tag=build.aws.python.3.7&fromImage=spcleth%2Fserverless-benchmarks

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/sebs/benchmark.py", line 359, in install_dependencies self._docker_client.images.pull(repo_name, image_name) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/models/images.py", line 465, in pull pull_log = self.client.api.pull( File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/image.py", line 429, in pull self._raise_for_status(response) File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 270, in _raise_for_status raise create_api_error_from_http_exception(e) from e File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception raise cls(e, response=response, explanation=explanation) from e docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.41/images/create?tag=build.aws.python.3.7&fromImage=spcleth%2Fserverless-benchmarks: Internal Server Error ("Head "https://registry-1.docker.io/v2/spcleth/serverless-benchmarks/manifests/build.aws.python.3.7": read tcp 10.0.2.15:56810->34.205.13.154:443: read: connection reset by peer")

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 50, in test func = deployment_client.get_function( File "/home/justinchigu/serverless-benchmarks/sebs/faas/system.py", line 157, in getfunction rebuilt, = code_package.build(self.package_code) File "/home/justinchigu/serverless-benchmarks/sebs/benchmark.py", line 499, in build self.install_dependencies(self._output_dir) File "/home/justinchigu/serverless-benchmarks/sebs/benchmark.py", line 361, in install_dependencies raise RuntimeError("Docker pull of image {} failed!".format(image_name)) RuntimeError: Docker pull of image build.aws.python.3.7 failed!

OUTPUT 2

OUTPUT 3

jchigu commented 1 year ago

OUTPUT 2 AWS (python-venv) justinchigu@justinchigu-VirtualBox:~/serverless-benchmarks$ ./sebs.py experiment invoke perf-cost --config config/example.json --deployment aws ERROR:root:Docker pull of image build.aws.python.3.7 failed! Traceback (most recent call last): File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 268, in _raise_for_status response.raise_for_status() File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/images/spcleth/serverless-benchmarks:build.aws.python.3.7/json

jchigu commented 1 year ago

OUTPUT 2 AWS

(python-venv) justinchigu@justinchigu-VirtualBox:~/serverless-benchmarks$ ./sebs.py experiment invoke perf-cost --config config/example.json --deployment aws
ERROR:root:Docker pull of image build.aws.python.3.7 failed!
Traceback (most recent call last):
  File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/images/spcleth/serverless-benchmarks:build.aws.python.3.7/json
jchigu commented 1 year ago

OUTPUT 3 - running LOCAL ----these commands no longer give errors, so what is it that I really need to do from now onwards until I can see, analyse and present results.

(python-venv) justinchigu@justinchigu-VirtualBox:~/serverless-benchmarks$ ./sebs.py storage start minio --port 9011 --output-json out_storage.json
16:10:31,193 INFO root: Starting storage Storage.MINIO on port 9011.
16:10:31,744 INFO minio.Minio-8a25: Starting minio instance at 172.17.0.2:9000
16:10:31,744 INFO root: Writing storage configuration to out_storage.json.
(python-venv) justinchigu@justinchigu-VirtualBox:~/serverless-benchmarks$ jq --argfile file1 out_storage.json '.deployment.local.storage = $file1 ' config/example.json > config/local_deployment.json
(python-venv) justinchigu@justinchigu-VirtualBox:~/serverless-benchmarks$ ./sebs.py local start 110.dynamic-html test out_benchmark.json --config config/local_deployment.json --deployments 1
(python-venv) justinchigu@justinchigu-VirtualBox:~/serverless-benchmarks$ curl 172.17.0.3:9000 --request POST --data '{"random_len": 10,"username": "testname"}' --header 'Content-Type: application/json'
{"begin": "1673359935.623813", "end": "1673359935.672718", "request_id": "45c3d9fa-603f-4bd8-b8c7-07cbb0d9362b", "is_cold": false, "result": {"output": {"result": "<!DOCTYPE html>\n<html>\n  <head>\n    <title>Randomly generated data.</title>\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\n    <link href=\"http://netdna.bootstrapcdn.com/bootstrap/3.0.0/css/bootstrap.min.css\" rel=\"stylesheet\" media=\"screen\">\n    <style type=\"text/css\">\n      .container {\n        max-width: 500px;\n        padding-top: 100px;\n      }\n    </style>\n  </head>\n  <body>\n    <div class=\"container\">\n      <p>Welcome testname!</p>\n      <p>Data generated at: 2023-01-10 14:12:15.672802!</p>\n      <p>Requested random numbers:</p>\n      <ul>\n        \n        <li>592917</li>\n        \n        <li>995690</li>\n        \n        <li>191043</li>\n        \n        <li>439770</li>\n        \n        <li>672474</li>\n        \n        <li>56221</li>\n        \n        <li>201959</li>\n        \n        <li>833328</li>\n        \n        <li>741493</li>\n        \n        <li>755390</li>\n        \n      </ul>\n    </div>\n  </body>\n</html>"(python-venv) justinchigu@justinchigu-VirtualBox:~/serverless-benchmarks$ ./sebs.py local stop out_benchmark.jsonson
./sebs.py storage stop out_storage.json
16:13:08,264 INFO root: Stopping deployment from /home/justinchigu/serverless-benchmarks/out_benchmark.json
16:13:08,300 INFO Local.LocalFunction-2b02: Stopping function container a5734b1391903eaa0930a3097d1b9ab759bfc9cf6ed5ae78bfe246b02b22b7bb
16:13:08,514 INFO Local.LocalFunction-2b02: Function container a5734b1391903eaa0930a3097d1b9ab759bfc9cf6ed5ae78bfe246b02b22b7bb stopped succesfully
16:13:08,514 INFO root: Stopped deployment from /home/justinchigu/serverless-benchmarks/out_benchmark.json
16:13:09,523 INFO root: Stopping storage deployment of minio.
16:13:09,544 INFO minio.Minio-94a8: Stopping minio container at 172.17.0.2:9000.
16:13:10,279 INFO minio.Minio-94a8: Stopped minio container at 172.17.0.2:9000.
16:13:10,279 INFO root: Stopped storage deployment of minio.
(python-venv) justinchigu@justinchigu-VirtualBox:~/serverless-benchmarks$
jchigu commented 1 year ago 
(python-venv) justinchigu@justinchigu-VirtualBox:~/serverless-benchmarks$ ./sebs.py benchmark invoke 110.dynamic-html test --config config/example.json --deployment aws --verbose
16:25:59,532 INFO SeBS-ecf8: Created experiment output at /home/justinchigu/serverless-benchmarks
16:26:00,423 INFO AWS.Credentials-1872: Using cached credentials for AWS
16:26:00,424 INFO AWS.Resources-927f: Using cached resources for AWS
16:26:00,424 INFO AWS.Config-9598: Using cached config for AWS
16:26:00,775 INFO Benchmark-8c32: Building benchmark 110.dynamic-html. Reason: no cached code package.
16:26:00,813 INFO Benchmark-8c32: Docker pull of image spcleth/serverless-benchmarks:build.aws.python.3.7
ERROR:root:Docker pull of image build.aws.python.3.7 failed!
Traceback (most recent call last):
  File "/home/justinchigu/serverless-benchmarks/python-venv/lib/python3.10/site-packages/docker/api/client.py", line 268, in _raise_for_status
mcopik commented 1 year ago

@jchigu Our testing and experiments are optimized for cloud deployment; the local deployment has been designed for testing and detailed profiling each function. While the experiments should work with local deployment, it is much more likely that there will be bugs. I should be able to take a closer look by the end of this week.

Regarding your most recent example, I'm not sure why you get the following error:

ERROR:root:Docker pull of image build.aws.python.3.7 failed!

This image is publically available. Can you please post the full error, including the exception description (should be at the end)?

mcopik commented 1 year ago

@jchigu The error "404 Client Error: Not Found for url: http+docker://localhost [...]" seems to be pretty weird but it has been reported before in our dependencies. One of the potential reasons is a full partition that is hosting the Docker image.

Can you succeed running docker pull spcleth/serverless-benchmarks:build.aws.python.3.7? These images are not small and they can create issues in VMs with constrainted disk size. If the regular docker command runs, then it should also work by using our library; otherwise, the issue might be in Docker installation such as permissions, disk space issue, or running behind a proxy (there was another HTTP 500 before).

jchigu commented 1 year ago

Docker pull was successful

(python-venv) justinchigu@justinchigu-VirtualBox:~/serverless-benchmarks$ docker pull spcleth/serverless-benchmarks:build.aws.python.3.7 build.aws.python.3.7: Pulling from spcleth/serverless-benchmarks 77e94c8586bc: Pull complete c1e4458103e7: Pull complete bb7caa430a73: Pull complete 1418415b0a5d: Pull complete 705030a34033: Pull complete 43dfd56739e6: Pull complete 741ca7194c6f: Pull complete c9351969459a: Pull complete a37ed3c00717: Pull complete 98b71cec17a6: Pull complete 39110b27d5dc: Pull complete Digest: sha256:ec5309c41f7ec804ee36e04782456e37732c3ea57610cdc273c8dc15e2882c99 Status: Downloaded newer image for spcleth/serverless-benchmarks:build.aws.python.3.7 docker.io/spcleth/serverless-benchmarks:build.aws.python.3.7 (python-venv) justinchigu@justinchigu-VirtualBox:~/serverless-benchmarks$

adhinneupane commented 1 year ago

@jchigu , not sure if @mcopik remembers but I faced a similar issue last year when using the repository. Solved it by using another public docker image with same versions.

As long as you match the dependencies in the docker image, you should be good.

jchigu commented 1 year ago

The following error appeared on the following benchmarks when I run curl.... 501.graph-pagerank, 502.graph-mst, 503.graph-bfs, and 411.image-recognition. The other benchmarks gave out results.

(python-venv) justinchigu@justinchigu-VirtualBox:~/serverless-benchmarks$ curl 172.17.0.3:9000 --request POST -o graph-pagerankresults.json --data '{"size": 10}' --header 'Content-Type: application/json' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1660 100 1648 100 12 392k 2923 --:--:-- --:--:-- --:--:-- 540k (python-venv) justinchigu@justinchigu-VirtualBox:~/serverless-benchmarks$ cat graph-pagerankresults.json

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html>
    <head>
        <title>Error: 500 Internal Server Error</title>
        <style type="text/css">
          html {background-color: #eee; font-family: sans;}
          body {background-color: #fff; border: 1px solid #ddd;
                padding: 15px; margin: 15px;}
          pre {background-color: #eee; border: 1px solid #ddd; padding: 5px;}
        </style>
    </head>
    <body>
        <h1>Error: 500 Internal Server Error</h1>
        <p>Sorry, the requested URL <tt>&#039;http://172.17.0.3:9000/&#039;</tt>
           caused an error:</p>
        <pre>Internal Server Error</pre>
          <h2>Exception:</h2>
          <pre>ImportError(&#039;libxml2.so.2: cannot open shared object file: No such file or directory&#039;)</pre>
          <h2>Traceback:</h2>
          <pre>Traceback (most recent call last):

File "/usr/local/lib/python3.7/site-packages/bottle.py", line 868, in _handle return route.call(*args) File "/usr/local/lib/python3.7/site-packages/bottle.py", line 1748, in wrapper rv = callback(a, *ka) File "/sebs/server.py", line 14, in flush_log from function import function File "/function/function/function.py", line 2, in <module> import igraph File "/function/.python_packages/lib/site-packages/igraph/init.py", line 34, in <module> from igraph._igraph import ImportError: libxml2.so.2: cannot open shared object file: No such file or directory (python-venv) justinchigu@justinchigu-VirtualBox:~/serverless-benchmarks$

mcopik commented 1 year ago

@adhinneupane Thanks for sharing - indeed, there seems to be an issue with the Docker Hub that I cannot reproduce.

mcopik commented 1 year ago

@jchigu The problem with graph benchmarks has been resolved in issue #111. Unfortunately, I cannot reproduce the problem with benchmark 411.image-registration - it works fine for me for Python versions 3.7 and 3.8.

If you are still having problems, then please post a formatted output with the error visible. Closing as mentioned per email.