zapier / zapier-platform

The SDK for you to build an integration on Zapier
https://platform.zapier.com
Other
344 stars 188 forks source link

PDE-5309 feat(core): add support for "buffered creates" #832

Closed eliangcs closed 1 month ago

eliangcs commented 2 months ago

Introducing "Buffered Create Actions"...

A Buffered Create allows you to create objects in bulk with a single or fewer API request(s). This is useful when you want to reduce the number of requests made to your server. When enabled, Zapier holds the data until the buffer reaches a size limit or a certain time has passed, then sends the buffered data using the performBuffer function you define.

To implement a Buffered Create, you define a buffer configuration object and a performBuffer function in the operation object. In the buffer config object, you specify how you want to group the buffered data using the groupedBy setting and the maximum number of items to buffer using the limit setting.

Here's an example:

const performBuffer = async (z, bufferedBundle) => {
  // Send a request to create tasks in bulk
  const response = await z.request({
    method: 'POST',
    url: 'https://httpbin.zapier-tooling.com/post',
    body: {
      items: bufferedBundle.buffer,
    },
  });

  const items = response.data.json.items;
  const result = {};

  for (const item of items) {
    const idempotencyId = item.meta.id;
    const itemProps = item.inputData;
    result[idempotencyId] = {
      outputData: {
        id: Date.now(),
        ...itemProps,
      },
    };
  }

  return result;
};

module.exports = {
  key: 'task_buffered',
  noun: 'Task (Buffered)',

  display: {
    label: 'Create Task (Buffered)',
    description: 'Creates a task.',
  },

  operation: {
    performBuffer,
    buffer: {
      groupedBy: ['project_id'],
      limit: 100,
    },

    inputFields: [
      { key: 'project_id', required: true },
      { key: 'title', required: true },
      { key: 'content' },
    ],

    sample: {
      id: 1,
      name: 'Test',
    },

    outputFields: [],
  },
};
standielpls commented 1 month ago

I reviewed bc it was referenced in the backend MR, but noticed this was still in draft, so did a quick pass through, lmk when this is ready!

eliangcs commented 1 month ago

@standielpls thanks for the review! I added some text mentioning the feature is internal-only. Can you re-review?