Lambda does not exit on errors, keeps running until timeout

[OperationalFallacy]

Describe the bug

I haven't tested it super thoroughly, but I saw Lambda runs till max timeout even thought it couldn't process anything. One example was sending gz files which I realized not supported

Steps to reproduce (if applicable) Steps to reproduce the behavior:

1. deploy indexer lambda
2. send compressed file, test.json.gz

Expected behavior

It should exit immediately and not run until timeout

Configuration:

export class RustFunction extends Function {
  constructor(scope: Construct, id: string, props?: Partial<FunctionProps>) {
    const lambdaAssetPath = path.join(
      __dirname,
      "../../quickwit/quickwit",
      "quickwit-lambda/deploy",
      id
    );
    console.log("lambdaAssetPath", lambdaAssetPath);
    super(scope, id+'_f', {
      ...props,
      code: Code.fromAsset(lambdaAssetPath),
      handler: id, // use id to specify either "indexer" or "searcher"
      runtime: Runtime.PROVIDED_AL2,
      architecture: Architecture.ARM_64,
      logRetention: RetentionDays.ONE_DAY,
      tracing: Tracing.DISABLED,
    });
  }
}

Binaries compiled with this command (I didn't use cross it was too slow on Mac)

LIBZ_SYS_STATIC=1 TARGET_CC=aarch64-linux-musl-gcc RUSTFLAGS="-C linker=aarch64-linux-musl-gcc -C link-arg=-static -C opt-level=z -C lto" cargo build --release --target aarch64-unknown-linux-musl

1. Output of quickwit --version

checked out v0.8.1

2. The index_config.yaml

version: 0.7

index_id: test-index

doc_mapping: field_mappings:

• name: website type: text
• name: name type: text
• name: founded type: i64 indexed: true fast: true
• name: size type: text
• name: locality type: text
• name: region type: text fast: true
• name: country type: text fast: true
• name: industry type: text fast: true
• name: linkedin_url type: text

search_settings: default_search_fields: [name]

indexing_settings: split_num_docs_target: 2000000

[rdettai]

Hi @OperationalFallacy, thanks for reporting this.

It would be a useful improvement indeed, even though it is far from trivial to identify all un-recoverable errors and bubble them up to stop the lambda.

One example was sending gz files which I realized not supported

gz files are supported, if you run Quickwit Lambda with these packages it should work just fine.

In general, we don't guaranty that main or the Quickwit release tags (e.g 0.8.1) builds to a functioning AWS Lambda. The release cycle of Quickwit Lambda is still independent at this stage. You should use the Lambda release tags instead (aws-lambda-beta-xx).

[OperationalFallacy]

Agree, won't be easy. I saw it runs server-side software. Pretty remarkable you made it work in Lambda, and so well.

Is there an option to make all errors unrecoverable? If users prefer Lambda can use own retry mechanism. I recognize, though - this is probably not how quickwit servers designed :)

Thanks for pointing to the tags, I'll try it!

[rdettai]

Is there an option to make all errors unrecoverable?

There is currently a retry mechanism at the indexing pipeline level that could be disabled. That wouldn't be a silver bullet solution but might already avoid hanging in many failure cases.

[OperationalFallacy]

That should help. Is there a configuration option to disable it?