Add Options for the config file

aimementoring / tap-airtable

Singer TAP for Airtbale

8 stars 26 forks source link

Add Options for the config file #6

Open camposcristian opened 4 years ago

camposcristian commented 4 years ago

In the config.json, support the following fields:

[x] tableId so we can export a particular table instead of all the tables
[ ] view so we can only export a particular view from the table
[ ] fields to include a list of fields to export
[x] filterEmojis to remove any emojis from both the table name and the data

bfischetti commented 4 years ago

Hey @camposcristianeze, according to Singer's guidelines, the way to do this is to specify which of the tables and fields you would like to export in the properties.json, that has the schema with all the metadata to export.

The idea is that you first need to run the discovery mode for a particular base, which creates the properties.json, and then you would need to modify the file in order to "select" the tables and fields.

I don't know if we should support extracting views for now, as this is a lot of work and we might need to prioritise.

As for the filterEmojis, I believe that this could easily be included in the config.json but I will need to research how to filter them effectively.

camposcristian commented 4 years ago

Hey @camposcristianeze, according to Singer's guidelines, the way to do this is to specify which of the tables and fields you would like to export in the properties.json, that has the schema with all the metadata to export. The idea is that you first need to run the discovery mode for a particular base, which creates the properties.json, and then you would need to modify the file in order to "select" the tables and fields. I don't know if we should support extracting views for now, as this is a lot of work and we might need to prioritise.

Great! Being able to filter base, table and fields is already enough, so we can get along with the discovery step and properties.json

As for the filterEmojis, I believe that this could easily be included in the config.json but I will need to research how to filter them effectively.

Ok, this will save a lot of time in terms of reporting. Here and here are some good resources to start in terms of finding a range of unicode to filter. Let me know how it goes!

bfischetti commented 4 years ago

Hey @camposcristianeze, already merged #8, which has the changes related to the config.json file and the select-by-default of tables.

And I've just opened #9, which adds the option to remove emojis from the sync. Could you take a quick look @jw-0? Thanks!

jw-0 commented 4 years ago

I may not get a chance this week because of AI @bfischetti but I can have a look next Monday