Getting TestDataSyncFailure some times when running on AWS Lambda

[Panzki]

Version info:

VERSION INFO:

Artillery: 2.0.16
Node.js:   v20.12.0
OS:        darwin

Running this command:

npx artillery run-lambda --config "/test/project/root/artillery-config.yml" --variables '{"NR_LICENSE_KEY":{}}' --environment preview --region eu-west-1 --count 15 scenario.yml

I expected to see this happen:

Artillery executes the given scenario reliably every time.

Instead, this happened:

About 50% of the time Artillery fails with the following error message:

A Lambda function has exited with an error. Reason: TestDataSyncFailure
{
  err: {
    message: 'Number of files in metadata.json (3) does not match number of files (8) in local directory! Something went wrong!',
    stack: 'Error: Number of files in metadata.json (3) does not match number of files (8) in local directory! Something went wrong!\n' +
      '    at syncTestData (/artillery/a9-handler-dependencies.js:36:11)\n' +
      '    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n' +
      '    at async Runtime.handler (/artillery/a9-handler-index.js:70:5)'
  }
}

I could not reproduce the error reliably, sometimes it occurred some time everything is fine. It looks like there are already some files in the test data directory on the local files system of the Lambda function. The problem might be triggered if the AWS Lambda execution context is reused. The AWS docs state the following:

Each execution environment provides between 512 MB and 10,240 MB, in 1-MB increments, of disk space in the /tmp directory. The directory content remains when the execution environment is frozen, providing a transient cache that can be used for multiple invocations. You can add extra code to check if the cache has the data that you stored.

Could you please have a look into this problem? If you need more information I'm happy to provide what's necessary.

[bernardobridge]

Hi @Panzki 👋 .

Thanks for reporting the issue. I'll be taking a look at it.

A couple of questions:

• Is this happening with a specific test?
• If you could share which files are being bundled with the test, and the contents of the test script, that could help us get started.

Thank you!

[longbeom]

I've had the same problem, I solved it by deleting the Lambda function and running it.