MeltanoLabs / tap-salesforce

Singer.io tap for the Salesforce API
GNU Affero General Public License v3.0
1 stars 30 forks source link

fix: Exclude location fields when `api_type` is `BULK` #51

Closed dlouseiro closed 5 months ago

dlouseiro commented 10 months ago

In the current implementation of the tap, fields described with type=address are correctly excluded when using api_type=BULK.

Although, this is not the case for geolocation fields (described with type=location).

I'm not a Salesforce specialist per se, but know that the changes I'm applying on this branch solved the issue described here.

In my specific case I had a field like this:

                    "Geolocation__c": {
                        "type": [
                            "number",
                            "object",
                            "null"
                        ],
                        "properties": {
                            "longitude": {
                                "type": [
                                    "null",
                                    "number"
                                ]
                            },
                            "latitude": {
                                "type": [
                                    "null",
                                    "number"
                                ]
                            }
                        }
                    },

As it's a compound field, split over multiple properties (longitude and latitude), the parent field Geolocation__c does not have much value as the relevant information is propagated in the "sub-fields" (latitude and longitude).

So what I'm doing in this PR is to ensure that:

Snippet of the excluded schema after this PR:

                {
                    "breadcrumb": [
                        "properties",
                        "Geolocation__c"
                    ],
                    "metadata": {
                        "inclusion": "unsupported",
                        "unsupported-description": "cannot query compound address fields with bulk API"
                    }
                },

Snippet of the "sub-fields" in the schema:

                    "Geolocation__Latitude__s": {
                        "type": [
                            "null",
                            "number"
                        ]
                    },
                    "Geolocation__Longitude__s": {
                        "type": [
                            "null",
                            "number"
                        ]
                    },
edgarrmondragon commented 5 months ago

Thanks @dlouseiro!