Sharded Audit Log Data Sources

[3v-dgudaitis]

I set this package up for the first time and it mostly works great, so thank you.

However, my audit logs are going in to date-sharded tables that aren't compatible with this package.

The fix is simple, and I'm happy to submit a PR with the fix, but I wanted to check how you would like it handled.

All I need is to add an identifier line so the src_bigquery_audit_log.yml file.

sources:

  - name: bigquery_audit_log
    database: "{{ var ('leaner_query_database', target.database) }}"
    schema: "{{ var ('leaner_query_source_schema', 'bigquery_audit_logs') }}"
    tables:
      - name: "{{ var ('leaner_query_data_access_table', 'cloudaudit_googleapis_com_data_access') }}"
        identifier: "{{ var ('leaner_query_data_access_table', 'cloudaudit_googleapis_com_data_access') }}_*"
        description: Bigquery data access logging

Do you want at new variable to handle the identifier, like cloudaudit_googleapis_com_data_access_identifier? This ensures maximum compatibility with sharded and non-sharded tables but maybe, if sharded tables are the standard now, then we just hard-code the shard selector as I did above.

How would you like to proceed?

[bobsamuels]

@3v-dgudaitis Good call. We didn't account for sharded tables as we're just using date partitioning. I like the idea of the identifier variable - that's the most flexible for folks that are on either side of this. Please do open a quick PR!