spcl / serverless-benchmarks

SeBS: serverless benchmarking suite for automatic performance analysis of FaaS platforms.
https://mcopik.github.io/projects/sebs/
BSD 3-Clause "New" or "Revised" License
142 stars 64 forks source link

Unknown subprocess.CalledProcessError #146

Closed HelloWorldGitHubUser closed 1 year ago

HelloWorldGitHubUser commented 1 year ago

hi, I used to get everything right and can proceed with the benchmark on OpenWhisk successfully, but recently maybe I've add some packages or configure sth else on my server (but not for serverless-benchmarks), then I'm not able to run the benchmark on OpenWhisk again. And the error seems quite ambiguous:

(python-venv) lyan@user-1288H-V6:~/serverless-benchmarks$ ./sebs.py benchmark invoke 110.dynamic-html test --config config/openwhisk.json --deployment openwhisk --verbose [21:52:13.405255] SeBS-93d3 Created experiment output at /home/ lyan/serverless-benchmarks [21:52:13.412049] OpenWhisk.Resources-8b54 Using user-provided Docker registry for OpenWhisk. [21:52:13.412137] OpenWhisk.Resources-8b54 Using user-provided configuration of storage for OpenWhisk. [21:52:13.414205] Benchmark-db18 Using cached benchmark 110.dynamic-html at /home/ lyan/serverless-benchmarks/cache/110.dynamic-html/openwhisk/python/3.7/110.dynamic-html.zip [21:52:13.414617] OpenWhisk-9fe0 Using cached function 110.dynamic-html-python-3.7 in /home/ lyan/serverless-benchmarks/cache/110.dynamic-html/openwhisk/python/3.7/110.dynamic-html.zip [21:52:13.427521] OpenWhisk-9fe0 Updating function configuration due to changed storage configuration. [21:52:13.427675] OpenWhisk-9fe0 Update configuration of an existing OpenWhisk action 110.dynamic-html-python-3.7. ERROR:root:Command '['/usr/local/bin/wsk', '-i', 'action', 'update', '110.dynamic-html-python-3.7', '--memory', '128', '--timeout', '10000', '-p', 'MINIO_STORAGE_SECRET_KEY', '2f17fadd7f79ad1746a217c0f53895c0d64c20098cac99cd3da6c6a22b3156c2', '-p', 'MINIO_STORAGE_ACCESS_KEY', 'yV1kQ6ZkOzqyMqV8g1oddE33z4jHZfcbNDqlPY_MO8M', '-p', 'MINIO_STORAGE_CONNECTION_URL', '192.168.3.71:9011']' returned non-zero exit status 144. Traceback (most recent call last): File "/home/ lyan/serverless-benchmarks/./sebs.py", line 30, in call return self.main(args, kwargs) File "/home/ lyan/serverless-benchmarks/python-venv/lib/python3.10/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/home/ lyan/serverless-benchmarks/python-venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/ lyan/serverless-benchmarks/python-venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/ lyan/serverless-benchmarks/python-venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, ctx.params) File "/home/ lyan/serverless-benchmarks/python-venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke return __callback(args, kwargs) File "/home/ lyan/serverless-benchmarks/./sebs.py", line 72, in wrapper return func(*args, *kwargs) File "/home/ lyan/serverless-benchmarks/./sebs.py", line 97, in wrapper return func(args, kwargs) File "/home/ lyan/serverless-benchmarks/./sebs.py", line 233, in invoke func = deployment_client.get_function( File "/home/ lyan/serverless-benchmarks/sebs/faas/system.py", line 219, in get_function self.update_function_configuration(function, code_package) File "/home/ lyan/serverless-benchmarks/sebs/openwhisk/openwhisk.py", line 359, in update_function_configuration subprocess.run( File "/usr/lib/python3.10/subprocess.py", line 524, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['/usr/local/bin/wsk', '-i', 'action', 'update', '110.dynamic-html-python-3.7', '--memory', '128', '--timeout', '10000', '-p', 'MINIO_STORAGE_SECRET_KEY', '2f17fadd7f79ad1746a217c0f53895c0d64c20098cac99cd3da6c6a22b3156c2', '-p', 'MINIO_STORAGE_ACCESS_KEY', 'yV1kQ6ZkOzqyMqV8g1oddE33z4jHZfcbNDqlPY_MO8M', '-p', 'MINIO_STORAGE_CONNECTION_URL', '192.168.3.71:9011']' returned non-zero exit status 144.

Could you please tell me what could possibly be wrong? Or do I have some ways to print more detailed log info? Thx a lot :)

HelloWorldGitHubUser commented 1 year ago

If I need to provide more information, please tell me :)

HelloWorldGitHubUser commented 1 year ago

I've compared the log info before with this now and I've discovered that the former log info contains:

Cached function 110.dynamic-html-python-3.7 is up to date.

but now it says:

21:52:13.427521] OpenWhisk-9fe0 Updating function configuration due to changed storage configuration. [21:52:13.427675] OpenWhisk-9fe0 Update configuration of an existing OpenWhisk action 110.dynamic-html-python-3.7. and then it comes with the error.

So maybe there's sth wrong with cache/updating?

HelloWorldGitHubUser commented 1 year ago

Solved this by re-cloning the whole repository

mcopik commented 1 year ago

@HelloWorldGitHubUser It is possible that you restarted/relaunched OpenWhisk? In that case, the cache is invalidated, and we try to update a non-existing function. Since the OpenWhisk instance is fresh, there's no function with such a name, but our cache thinks it exists. If by "re-cloning" your mean replacing the sebs repository, then likely you removed the cache which would solve the issue.

HelloWorldGitHubUser commented 1 year ago

@mcopik Yes I did relaunch OpenWhisk in my kind cluster. I think that's why I failed before and I'll delete the cache next time I relaunch, thx :)

mcopik commented 1 year ago

We need to add here a proper error message for the user to explain what might be happening.

mcopik commented 1 year ago

Fixed in dev - now we provide additional message.