AWS 500 Internal Server Error on 110.dynamic-html

[OutisNoMore]

Describe the bug Getting an internal server error, code 500 when running benchmark 110.dynamic-html using aws.

To Reproduce Steps to reproduce the behavior: ./sebs.py benchmark invoke 110.dynamic-html test --config config/example.json --deployment aws --verbose

Expected behavior Successful completion of benchmark with outputs.

Logs

[13:49:45.208423] SeBS-d048 Created experiment output at serverless-benchmarks
[13:49:46.233298] AWS.Resources-2188 Using cached resources for AWS
[13:49:46.233956] AWS.Config-ceb4 Using cached config for AWS
[13:49:46.364648] AWS-4107 Using existing resource name: 222752e8.
[13:49:46.379418] Benchmark-5ec1 Using cached benchmark 110.dynamic-html at serverless-benchmarks/cache/110.dynamic-html/aws/python/3.8/110.dynamic-html.zip
[13:49:46.380589] AWS-4107 Using cached function 110_dynamic_html_python_3_8 in serverless-benchmarks/cache/110.dynamic-html/aws/python/3.8/110.dynamic-html.zip
[13:49:46.381059] AWS-4107 Cached function 110_dynamic_html_python_3_8 is up to date.
[13:49:46.386065] SeBS-d048 Beginning repetition 1/5
[13:49:46.386194] AWS.HTTPTrigger-9b6d Invoke function https://hxjoj9mewl.execute-api.us-east-1.amazonaws.com
[13:49:48.764924] AWS.HTTPTrigger-9b6d Invocation on URL https://hxjoj9mewl.execute-api.us-east-1.amazonaws.com failed!
[13:49:48.765455] AWS.HTTPTrigger-9b6d Output: {'message': 'Internal Server Error'}
ERROR:root:Failed invocation of function! Output: {'message': 'Internal Server Error'}
Traceback (most recent call last):
  File "./sebs.py", line 30, in __call__
    return self.main(*args, **kwargs)
  File "serverless-benchmarks/python-venv/lib/python3.8/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "serverless-benchmarks/python-venv/lib/python3.8/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "serverless-benchmarks/python-venv/lib/python3.8/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "serverless-benchmarks/python-venv/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "serverless-benchmarks/python-venv/lib/python3.8/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "./sebs.py", line 72, in wrapper
    return func(*args, **kwargs)
  File "./sebs.py", line 103, in wrapper
    return func(*args, **kwargs)
  File "./sebs.py", line 263, in invoke
    ret = trigger.sync_invoke(input_config)
  File "serverless-benchmarks/sebs/aws/triggers.py", line 112, in sync_invoke
    return self._http_invoke(payload, self.url)
  File "serverless-benchmarks/sebs/faas/function.py", line 218, in _http_invoke
    raise RuntimeError(f"Failed invocation of function! Output: {output}")
RuntimeError: Failed invocation of function! Output: {'message': 'Internal Server Error'}
[13:49:48.778286] Benchmark-2944 Update cached config serverless-benchmarks/cache/azure.json
[13:49:48.779665] Benchmark-2944 Update cached config serverless-benchmarks/cache/aws.json

System (please complete the following information):

• SeBS version: branch master, commit ID: 655f644
• Python version 3.8
• OS WSL

Additional context All given steps in the documentation were followed, with all appropriate permissions applied. Similar errors were found when running 411.image-recognition, and 220.video-processing benchmarks

[mcopik]

@OutisNoMore Thank you for trying SeBS! Unfortunately, I was not able to reproduce this issue and it looks very strange - this benchmark is very simple and usually works.

(1) Are you able to access the AWS website console and open CloudWatch logs associated with this function? These can provide us with essential information to identify the problem.

(2) Can you please share with me the code package prepared by SeBS? It should be located in cache/110.dynamic-html/aws/python/3.8/110.dynamic-html.zip and be quite small (< 0.5 MB).

Similar errors were found when running 411.image-recognition, and 220.video-processing benchmarks

All other benchmarks work; only these 3 fail?

[OutisNoMore]

Thank you for the help! I included the requested information below. All other benchmarks are working fine, it is just those 3 that are giving me some problems.

(1) 110.dynamic-html: [ERROR] FileNotFoundError: [Errno 2] No such file or directory: '/var/task/function/templates/template.html' Traceback (most recent call last): File "/var/task/handler.py", line 21, in handler ret = function.handler(event) File "/var/task/function/function.py", line 18, in handler template = Template( open(path.join(SCRIPT_DIR, 'templates', 'template.html'), 'r').read())

It looks like it's trying to open template.html when it doesn't exist. Do I need to install this file?

(2) 110.dynamic-html.zip

All other benchmarks run fine, only the 3 fail. Taking a look at 220.video-processing logs it seems that it cannot find ffmpeg: [ERROR] FileNotFoundError: [Errno 2] No such file or directory: '/var/task/function/ffmpeg/ffmpeg' Traceback (most recent call last): File "/var/task/handler.py", line 21, in handler ret = function.handler(event) File "/var/task/function/function.py", line 79, in handler upload_path = operations[op](download_path, duration, event) File "/var/task/function/function.py", line 41, in watermark call_ffmpeg([ File "/var/task/function/function.py", line 15, in call_ffmpeg ret = subprocess.run([os.path.join(SCRIPT_DIR, 'ffmpeg', 'ffmpeg'), '-y'] + args, File "/var/lang/lib/python3.8/subprocess.py", line 493, in run with Popen(*popenargs, **kwargs) as process: File "/var/lang/lib/python3.8/subprocess.py", line 858, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "/var/lang/lib/python3.8/subprocess.py", line 1720, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename)

And 411.image-recognition cannot find an imagenet_class_index.json file.

[ERROR] FileNotFoundError: [Errno 2] No such file or directory: '/var/task/function/imagenet_class_index.json' Traceback (most recent call last): File "/var/task/handler.py", line 20, in handler from function import function File "/var/task/function/function.py", line 22, in <module> class_idx = json.load(open(os.path.join(SCRIPT_DIR, "imagenet_class_index.json"), 'r'))

[mcopik]

@OutisNoMore Thank you for the bug report! These .html and .json files are normally packaged together with the code. For some reason, they are missing in your code package.

We had users on WSL before but we have not seen this before. Perhaps this is an artifact of using the zip: https://github.com/spcl/serverless-benchmarks/blob/master/sebs/aws/aws.py#L152

I will try to find a machine with Windows and reproduce the issue.

[mcopik]

@OutisNoMore I forgot initially that these resources are added by a dedicated Bash script init.sh: https://github.com/spcl/serverless-benchmarks/blob/master/benchmarks/100.webapps/110.dynamic-html/python/init.sh https://github.com/spcl/serverless-benchmarks/blob/master/benchmarks/200.multimedia/220.video-processing/init.sh https://github.com/spcl/serverless-benchmarks/blob/master/benchmarks/400.inference/411.image-recognition/python/init.sh

Which should be invoked here: https://github.com/spcl/serverless-benchmarks/blob/655f6447f9393d716136de7d54220ddb8e9c7b70/sebs/benchmark.py#L264

Can you please remove cache, rerun the 110.dynamic-html and share the output with me? Possibly this step is failing.

[OutisNoMore]

It turns out it was just a formatting error from trying to run the linux scripts on windows. When I tried to manually run the init.sh files I was getting:

benchmarks/100.webapps/110.dynamic-html/python/init.sh: line 2: $'\r': command not found benchmarks/100.webapps/110.dynamic-html/python/init.sh: line 11: syntax error: unexpected end of file

I fixed it by using sed to replace the trailing \r characters, and I was able to get all the tests running. Thank you again for all the help!