[Feature Request] Support bulk API (NDJSON) request to OpenSearch

[vicary]

I have a pipeline resolver for an AppSync Mutation. It contains two functions, the first one is a Lambda sending updates to RDS, the second one should take the result from $ctx.prev.result and index it via OpenSearch datasource.

In the request resolver mapping template of the second one, I am composing the bulk body in NDJSON similar to the following mannar:

#set($bulk = $util.toJson({ "index": { "_id": "${$ctx.prev.result.id}" } }))
#set($bulk = "${bulk}
${util.toJson($ctx.prev.result)}")

{
  "version": "2017-02-28",
  "operation": "POST",
  "path": "/_bulk",
  "params": {
    "body": $util.toJson($bulk)
  }
}

Lacking proper debugging tools, I have been using $util.error as a logging method to get my $bulk contents. And it looks like the following format, which seems correct.

{"index":{"_id":"A8DEF210-C342-48CB-9A4A-DA7D1E4D6AF1"}}
{"foo":123,"bar":999,"baz":1234567}

But when I actually runs the mutation via AppSync, I got a MappingTemplate error Unable to transform for the body: $[params][body]..

I took a look at [re:Post] Appsync HTTP resolver supported content types, which inspired me to take another look at Resolver Mappping Template for OpenSearch (params). It seems that POST body only accepts a single JSON object.

1. Am I correct that NDJSON is not currently supported yet?
2. If so, is supporting the bulk API in the upcoming plans?
3. What is the currently recommended way to index multiple "normalized" documents from the same resolver?

[mstoyanovv]

I've just encountered the same issue. Is there any progress on the above or at least is it correct that Appsync does not support ndjson?

[benmcp]

I would also love to hear any updates on this.

Another temporary solution may be to create a HTTP Resolver with Opensearch as the endpoint as HTTP Resolvers support ndjson.

However, I cannot create a working request mapping template to prove that this approach can work, debugging is difficult.

[vicary]

@benmcp AppSync API has a method evalucate-mapping-template which you can quickly generates an output with a manually crafted input JSON as its context.

See the Testing section here https://github.com/aws/aws-appsync-community/issues/147#issuecomment-1231754613 for more information, you may call that via aws-sdk programmatically or via AWS CLI.

HTTP resolver comes with the question of security, because AppSync neither uses dedicated IP range for domain level access control in Opensearch, nor a trivial method to do IAM v4 signing inside VTL.

I am still curious about your test results on the HTTP resolver, it's nice to know at least it works.

[benmcp]

Thanks for the heads up on the evalucate-mapping-template method, I did not know this existed.

Also, very good point about security. I was going to say we lock down opensearch via IP and IAM Role conditions but to your point, HTTP Resolvers don't actually have a method of associating a role (I read your comment and thought "what, you can totally associate a role to a HTTP Resolver...", but turns out that is indeed not true) and as you mentioned, neither does it have a trivial method to do IAM v4 signing inside VTL.

But yes I'll report back with my findings when I get back around to it if I find (a non secure) method. I feel like documentation is getting a bit sparse around these areas. Fingers crossed support for bulk API is coming in the near future!