[FEATURE] Airtable sources should optionally emit all fields for new / updated records, including fields that hold no values

[dylburger]

From Slack:

Re: Airtable (and possibly other Pipedream-built) triggers: Is there a way to force ALL columns/fields -- even those with null values -- to be returned in that Pipedream object? We are syncing new/updated Airtable data to Snowflake as JSON blob versions of that returned object and it seems that the Pipedream's Airtable new/updated trigger is ignoring null fields when creating that returned object - sometimes the object ay be 5 fields, sometimes 7 etc. And the Snowflake team say they'd love to get a consistent set of the full fields every time.

Airtable only includes fields that hold valid values in their list records API response, but it's possible the Airtable source could use the metadata API to list the table's schema, decorating the emitted event with the field names (and null values) for other fields.

[LucBerge]

I'm looking for "updated fields in a table" trigger. Having the list of the updated fields names in the event would do the work.

[michelle0927]

@dylburger It looks like the metadata API requires exposing a client secret. The Airtable actions do retrieve the tables' schemas, but the methods are not available to sources/triggers. Should this be labeled as blocked?

[dylburger]

I think so @michelle0927 . It sounds like we'd need to do some work to expose the metadata to the source runtime in the same way we did the work for actions. CC @dannyroosevelt and @js07 since they worked on the actions more closely.

[dannyroosevelt]

@js07 do you know what would be involved in this?

[js07]

I think this would involve some backend work to get the credentials used to make Airtable Metadata API calls to the source runtime, in a similar way to the runtime for actions.