search

penkow / llama-lambda

Deploying LLama 2 as AWS Lambda function for scalable serverless inference
22 stars 4 forks source link

Got illegal instruction when running on AWS Lambda #1

Open yoletech opened 11 months ago

yoletech commented 11 months ago

Hi,

Thanks for the good article about this repo , I tried to follow same steps but got this error when invoking the lambda function :

INIT_REPORT Init Duration: 282.61 ms Phase: init Status: error Error Type: Runtime.ExitError INIT_REPORT Init Duration: 96.89 ms Phase: invoke Status: error Error Type: Runtime.ExitError START RequestId: 352aa9b0-767d-4192-b437-165173cf5d92 Version: $LATEST RequestId: 352aa9b0-767d-4192-b437-165173cf5d92 Error: Runtime exited with error: signal: illegal instruction Runtime.ExitError END RequestId: 352aa9b0-767d-4192-b437-165173cf5d92 REPORT RequestId: 352aa9b0-767d-4192-b437-165173cf5d92 Duration: 97.83 ms Billed Duration: 98 ms Memory Size: 10240 MB Max Memory Used: 22 MB

do you have an idea about the possible cause?

Thanks.

penkow commented 11 months ago

Could you provide more information on how you call the lambda and also the full error log (if there is more)?

yoletech commented 11 months ago

I called it using the console interface with "Test" functionality and also with aws cli invoke.

This is full error : 

INIT_REPORT Init Duration: 315.67 ms    Phase: init Status: error   Error Type: Runtime.ExitError
INIT_REPORT Init Duration: 93.57 ms Phase: invoke   Status: error   Error Type: Runtime.ExitError
START RequestId: bdba7e75-d6cf-4f0e-8aac-718b138cac03 Version: $LATEST
RequestId: bdba7e75-d6cf-4f0e-8aac-718b138cac03 Error: Runtime exited with error: signal: illegal instruction
Runtime.ExitError
END RequestId: bdba7e75-d6cf-4f0e-8aac-718b138cac03
REPORT RequestId: bdba7e75-d6cf-4f0e-8aac-718b138cac03  Duration: 95.58 ms  Billed Duration: 96 ms  Memory Size: 10240 MB   Max Memory Used: 21 MB  
XRAY TraceId: 1-65758f7d-1db8a00f429eedac7bb5b6fc   SegmentId: 19986ad91c09ef38 Sampled: true

I think the compiled "llama_cpp" is the origin of this issue , when i remove it from the docker image it does not show this error.

EthanLLL commented 10 months ago

I called it using the console interface with "Test" functionality and also with aws cli invoke.

This is full error :

INIT_REPORT Init Duration: 315.67 ms  Phase: init Status: error   Error Type: Runtime.ExitError
INIT_REPORT Init Duration: 93.57 ms   Phase: invoke   Status: error   Error Type: Runtime.ExitError
START RequestId: bdba7e75-d6cf-4f0e-8aac-718b138cac03 Version: $LATEST
RequestId: bdba7e75-d6cf-4f0e-8aac-718b138cac03 Error: Runtime exited with error: signal: illegal instruction
Runtime.ExitError
END RequestId: bdba7e75-d6cf-4f0e-8aac-718b138cac03
REPORT RequestId: bdba7e75-d6cf-4f0e-8aac-718b138cac03    Duration: 95.58 ms  Billed Duration: 96 ms  Memory Size: 10240 MB   Max Memory Used: 21 MB  
XRAY TraceId: 1-65758f7d-1db8a00f429eedac7bb5b6fc SegmentId: 19986ad91c09ef38 Sampled: true

I think the compiled "llama_cpp" is the origin of this issue , when i remove it from the docker image it does not show this error.

I have ran into same issue. Any solution?

EthanLLL commented 10 months ago

I called it using the console interface with "Test" functionality and also with aws cli invoke.

This is full error :

INIT_REPORT Init Duration: 315.67 ms  Phase: init Status: error   Error Type: Runtime.ExitError
INIT_REPORT Init Duration: 93.57 ms   Phase: invoke   Status: error   Error Type: Runtime.ExitError
START RequestId: bdba7e75-d6cf-4f0e-8aac-718b138cac03 Version: $LATEST
RequestId: bdba7e75-d6cf-4f0e-8aac-718b138cac03 Error: Runtime exited with error: signal: illegal instruction
Runtime.ExitError
END RequestId: bdba7e75-d6cf-4f0e-8aac-718b138cac03
REPORT RequestId: bdba7e75-d6cf-4f0e-8aac-718b138cac03    Duration: 95.58 ms  Billed Duration: 96 ms  Memory Size: 10240 MB   Max Memory Used: 21 MB  
XRAY TraceId: 1-65758f7d-1db8a00f429eedac7bb5b6fc SegmentId: 19986ad91c09ef38 Sampled: true

I think the compiled "llama_cpp" is the origin of this issue , when i remove it from the docker image it does not show this error.

Hi I think it is because avx2 & avx512. Lambda support only avx2 right now, so if we build our docker image with llama-cpp with avx512 supported, it will not run on Lambda, so we have to turn avx512 off when build llama cpp. I started an AWS m4 instance to build the image and it works fine on Lambda.