Missing type `Json` in TYPE_MAP

[oscar-latorre-av]

Resource EmployeeAgreement has a newly added field (Config) which has a new type (Json, as reported by the INFO endpoint). This new field type is not accounted for in TYPE_MAP in discover.py, and is throwing an error every time Stitch (or singer) tries to perform an extraction.

https://github.com/DeputyApp/singer-tap-deputy/blob/6126abafdd7e07613d823c61c84c06235782044f/tap_deputy/discover.py#L64-L71

Logs

``` $ tap-deputy -c config.json --dev WARNING Executing Tap in Dev mode DEBUG Starting new HTTPS connection (1): [REDACTED].eu.deputy.com:443 DEBUG https://[REDACTED].eu.deputy.com:443 "GET /api/v1/resource/Address/INFO HTTP/1.1" 200 208 INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.2956879138946533, "tags": {"endpoint": "resource_info", "http_status_code": 200, "status": "succeeded"}} DEBUG https://[REDACTED].eu.deputy.com:443 "GET /api/v1/resource/Category/INFO HTTP/1.1" 200 177 INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.1277756690979004, "tags": {"endpoint": "resource_info", "http_status_code": 200, "status": "succeeded"}} DEBUG https://[REDACTED].eu.deputy.com:443 "GET /api/v1/resource/Comment/INFO HTTP/1.1" 200 175 INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.11403942108154297, "tags": {"endpoint": "resource_info", "http_status_code": 200, "status": "succeeded"}} DEBUG https://[REDACTED].eu.deputy.com:443 "GET /api/v1/resource/Company/INFO HTTP/1.1" 200 254 INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.11162185668945312, "tags": {"endpoint": "resource_info", "http_status_code": 200, "status": "succeeded"}} DEBUG https://276be407021147.eu.deputy.com:443 "GET /api/v1/resource/CompanyPeriod/INFO HTTP/1.1" 200 166 INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.09200048446655273, "tags": {"endpoint": "resource_info", "http_status_code": 200, "status": "succeeded"}} DEBUG https://[REDACTED].eu.deputy.com:443 "GET /api/v1/resource/Contact/INFO HTTP/1.1" 200 215 INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.10479259490966797, "tags": {"endpoint": "resource_info", "http_status_code": 200, "status": "succeeded"}} DEBUG https://[REDACTED].eu.deputy.com:443 "GET /api/v1/resource/Country/INFO HTTP/1.1" 200 184 INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.10700106620788574, "tags": {"endpoint": "resource_info", "http_status_code": 200, "status": "succeeded"}} DEBUG https://[REDACTED].eu.deputy.com:443 "GET /api/v1/resource/CustomAppData/INFO HTTP/1.1" 200 189 INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.16899871826171875, "tags": {"endpoint": "resource_info", "http_status_code": 200, "status": "succeeded"}} DEBUG https://[REDACTED].eu.deputy.com:443 "GET /api/v1/resource/CustomField/INFO HTTP/1.1" 200 277 INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.09703683853149414, "tags": {"endpoint": "resource_info", "http_status_code": 200, "status": "succeeded"}} DEBUG https://[REDACTED].eu.deputy.com:443 "GET /api/v1/resource/CustomFieldData/INFO HTTP/1.1" 200 410 INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.1805715560913086, "tags": {"endpoint": "resource_info", "http_status_code": 200, "status": "succeeded"}} DEBUG https://[REDACTED].eu.deputy.com:443 "GET /api/v1/resource/Employee/INFO HTTP/1.1" 200 427 INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.22699999809265137, "tags": {"endpoint": "resource_info", "http_status_code": 200, "status": "succeeded"}} DEBUG https://[REDACTED].eu.deputy.com:443 "GET /api/v1/resource/EmployeeAgreement/INFO HTTP/1.1" 200 294 INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.22404694557189941, "tags": {"endpoint": "resource_info", "http_status_code": 200, "status": "succeeded"}} CRITICAL 'Json' Traceback (most recent call last): File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "[REDACTED]\.venv\Scripts\tap-deputy.exe\__main__.py", line 7, in File "[REDACTED]\.venv\lib\site-packages\singer\utils.py", line 235, in wrapped return fnc(*args, **kwargs) File "[REDACTED]\.venv\lib\site-packages\tap_deputy\__init__.py", line 50, in main sync(client, File "[REDACTED]\.venv\lib\site-packages\tap_deputy\sync.py", line 88, in sync catalog = discover(client) File "[REDACTED]\.venv\lib\site-packages\tap_deputy\discover.py", line 120, in discover schema_dict, metadata = get_schema(client, resource_name) File "[REDACTED]\.venv\lib\site-packages\tap_deputy\discover.py", line 96, in get_schema 'type': ['null', TYPE_MAP[field_type]] KeyError: 'Json' ```

[P-Arnold]

Facing the same issue here, but the change will need to be made in the repo https://github.com/singer-io/tap-deputy . I believe 'Json': 'object'should be added to the TYPE_MAP referenced above.

[Bell-Richard2380]

Does anyone have a work around or anything for this so we can keep the integration going while this gets sorted? Or should I reach out to Deputy to see if they can revert the change made that introduced the Config type?

Any help would be appreciated as we use this data daily throughout the business.

Or can we adjust the tap_deputy to include this in the mapping ourselves?

[oscar-latorre-av]

Facing the same issue here, but the change will need to be made in the repo https://github.com/singer-io/tap-deputy . I believe 'Json': 'object'should be added to the TYPE_MAP referenced above.

Simply adding 'Json': 'object' doesn't cut it, as the Resource API returns empty/null EmployeeAgreement.Configs as "[]" and singer complains that that does not match type ['object', 'null']. Those empty objects should be somehow handled, sadly I don't know enough about singer to try that.

Example EmployeeAgreement with empty Config

`/api/v1/resource/EmployeeAgreement/14811` ```json { "Id": 14811, "EmployeeId": 10544, "PayPoint": 60, "EmpType": 1, "CompanyName": null, "Active": true, "StartDate": "2023-04-18T00:00:00+02:00", "EndDate": null, "Contract": 757, "SalaryPayRule": 0, "ContractFile": null, "PayrollId": "[REDACTED]", "PayPeriod": 1, "HistoryId": 104199, "Creator": 8450, "Created": "2023-04-17T02:08:54+02:00", "Modified": "2023-04-17T02:08:57+02:00", "BaseRate": null, "Config": "[]", "_DPMetaData": { "System": "EmployeeAgreement", "CreatorInfo": { "Id": 8450, "DisplayName": "[REDACTED]", "EmployeeProfile": 8456, "Employee": 8456, "Photo": "[REDACTED]", "Pronouns": 0, "CustomPronouns": "" } } } ```

Logs

``` $ tap-deputy -c config.json --dev ... INFO employee_agreements - Syncing data since 2023-04-17T00:00:00+00:00 {"type": "SCHEMA", "stream": "employee_agreements", "schema": {"properties": {"Id": {"type": ["null", "integer"]}, "EmployeeId": {"type": ["null", "integer"]}, "PayPoint": {"type": ["null", "integer"]}, "EmpType": {"type": ["null", "integer"]}, "CompanyName": {"type": ["null", "string"]}, "Active": {"type": ["null", "boolean"]}, "StartDate": {"format": "date-time", "type": ["null", "string"]}, "EndDate": {"format": "date-time", "type": ["null", "string"]}, "Contract": {"type": ["null", "integer"]}, "SalaryPayRule": {"type": ["null", "integer"]}, "ContractFile": {"type": ["null", "integer"]}, "PayrollId": {"type": ["null", "string"]}, "PayPeriod": {"type": ["null", "integer"]}, "HistoryId": {"type": ["null", "integer"]}, "Creator": {"type": ["null", "integer"]}, "Created": {"format": "date-time", "type": ["null", "string"]}, "Modified": {"format": "date-time", "type": ["null", "string"]}, "BaseRate": {"type": ["null", "number"]}, "Config": {"type": ["null", "object"]}}, "type": "object", "additionalProperties": false}, "key_properties": ["Id"]} DEBUG https://[REDACTED].eu.deputy.com:443 "POST /api/v1/resource/EmployeeAgreement/QUERY HTTP/1.1" 200 7472 INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.24155116081237793, "tags": {"endpoint": "employee_agreements", "http_status_code": 200, "status": "succeeded"}} DEBUG Removed 1 paths during transforms: _DPMetaData DEBUG Removed paths list: ['_DPMetaData'] INFO METRIC: {"type": "counter", "metric": "record_count", "value": 0, "tags": {"endpoint": "employee_agreements"}} CRITICAL Errors during transform CRITICAL Config: [] does not match {'type': ['object', 'null']} CRITICAL : {'Id': 14811, 'EmployeeId': 10544, 'PayPoint': 60, 'EmpType': 1, 'CompanyName': None, 'Active': True, 'StartDate': '2023-04-18T00:00:00+02:00', 'EndDate': None, 'Contract': 757, 'SalaryPayRule': 0, 'ContractFile': None, 'PayrollId': '[REDACTED]', 'PayPeriod': 1, 'HistoryId': 104199, 'Creator': 8450, 'Created': '2023-04-17T02:08:54+02:00', 'Modified': '2023-04-17T02:08:57+02:00', 'BaseRate': None, 'Config': '[]', '_DPMetaData': {'System': 'EmployeeAgreement', 'CreatorInfo': {'Id': 8450, 'DisplayName': '[REDACTED]', 'EmployeeProfile': 8456, 'Employee': 8456, 'Photo': '[REDACTED]', 'Pronouns': 0, 'CustomPronouns': ''}}} does not match {'properties': {'Id': {'type': ['integer', 'null']}, 'EmployeeId': {'type': ['integer', 'null']}, 'PayPoint': {'type': ['integer', 'null']}, 'EmpType': {'type': ['integer', 'null']}, 'CompanyName': {'type': ['string', 'null']}, 'Active': {'type': ['boolean', 'null']}, 'StartDate': {'format': 'date-time', 'type': ['string', 'null']}, 'EndDate': {'format': 'date-time', 'type': ['string', 'null']}, 'Contract': {'type': ['integer', 'null']}, 'SalaryPayRule': {'type': ['integer', 'null']}, 'ContractFile': {'type': ['integer', 'null']}, 'PayrollId': {'type': ['string', 'null']}, 'PayPeriod': {'type': ['integer', 'null']}, 'HistoryId': {'type': ['integer', 'null']}, 'Creator': {'type': ['integer', 'null']}, 'Created': {'format': 'date-time', 'type': ['string', 'null']}, 'Modified': {'format': 'date-time', 'type': ['string', 'null']}, 'BaseRate': {'type': ['number', 'null']}, 'Config': {'type': ['object', 'null']}}, 'type': 'object', 'additionalProperties': False} CRITICAL CRITICAL CRITICAL Errors during transform: [Config: [] does not match {'type': ['object', 'null']}, : {'Id': 14811, 'EmployeeId': 10544, 'PayPoint': 60, 'EmpType': 1, 'CompanyName': None, 'Active': True, 'StartDate': '2023-04-18T00:00:00+02:00', 'EndDate': None, 'Contract': 757, 'SalaryPayRule': 0, 'ContractFile': None, 'PayrollId': '[REDACTED]', 'PayPeriod': 1, 'HistoryId': 104199, 'Creator': 8450, 'Created': '2023-04-17T02:08:54+02:00', 'Modified': '2023-04-17T02:08:57+02:00', 'BaseRate': None, 'Config': '[]', '_DPMetaData': {'System': 'EmployeeAgreement', 'CreatorInfo': {'Id': 8450, 'DisplayName': '[REDACTED]', 'EmployeeProfile': 8456, 'Employee': 8456, 'Photo': '[REDACTED]', 'Pronouns': 0, 'CustomPronouns': ''}}} does not match {'properties': {'Id': {'type': ['integer', 'null']}, 'EmployeeId': {'type': ['integer', 'null']}, 'PayPoint': {'type': ['integer', 'null']}, 'EmpType': {'type': ['integer', 'null']}, 'CompanyName': {'type': ['string', 'null']}, 'Active': {'type': ['boolean', 'null']}, 'StartDate': {'format': 'date-time', 'type': ['string', 'null']}, 'EndDate': {'format': 'date-time', 'type': ['string', 'null']}, 'Contract': {'type': ['integer', 'null']}, 'SalaryPayRule': {'type': ['integer', 'null']}, 'ContractFile': {'type': ['integer', 'null']}, 'PayrollId': {'type': ['string', 'null']}, 'PayPeriod': {'type': ['integer', 'null']}, 'HistoryId': {'type': ['integer', 'null']}, 'Creator': {'type': ['integer', 'null']}, 'Created': {'format': 'date-time', 'type': ['string', 'null']}, 'Modified': {'format': 'date-time', 'type': ['string', 'null']}, 'BaseRate': {'type': ['number', 'null']}, 'Config': {'type': ['object', 'null']}}, 'type': 'object', 'additionalProperties': False}] Traceback (most recent call last): File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "[REDACTED]\.venv\Scripts\tap-deputy.exe\__main__.py", line 7, in File "[REDACTED]\.venv\lib\site-packages\singer\utils.py", line 235, in wrapped return fnc(*args, **kwargs) File "[REDACTED]\.venv\lib\site-packages\tap_deputy\__init__.py", line 50, in main sync(client, File "[REDACTED]\.venv\lib\site-packages\tap_deputy\sync.py", line 96, in sync sync_stream(client, catalog, state, start_date, stream, mdata) File "[REDACTED]\.venv\lib\site-packages\tap_deputy\sync.py", line 78, in sync_stream max_modified = process_records(stream, mdata, max_modified, records) File "[REDACTED]\.venv\lib\site-packages\tap_deputy\sync.py", line 30, in process_records record = transformer.transform(record, File "[REDACTED]\.venv\lib\site-packages\singer\transform.py", line 153, in transform raise SchemaMismatch(self.errors) singer.transform.SchemaMismatch: Errors during transform Config: [] does not match {'type': ['object', 'null']} : {'Id': 14811, 'EmployeeId': 10544, 'PayPoint': 60, 'EmpType': 1, 'CompanyName': None, 'Active': True, 'StartDate': '2023-04-18T00:00:00+02:00', 'EndDate': None, 'Contract': 757, 'SalaryPayRule': 0, 'ContractFile': None, 'PayrollId': '[REDACTED]', 'PayPeriod': 1, 'HistoryId': 104199, 'Creator': 8450, 'Created': '2023-04-17T02:08:54+02:00', 'Modified': '2023-04-17T02:08:57+02:00', 'BaseRate': None, 'Config': '[]', '_DPMetaData': {'System': 'EmployeeAgreement', 'CreatorInfo': {'Id': 8450, 'DisplayName': '[REDACTED]', 'EmployeeProfile': 8456, 'Employee': 8456, 'Photo': '[REDACTED]', 'Pronouns': 0, 'CustomPronouns': ''}}} does not match {'properties': {'Id': {'type': ['integer', 'null']}, 'EmployeeId': {'type': ['integer', 'null']}, 'PayPoint': {'type': ['integer', 'null']}, 'EmpType': {'type': ['integer', 'null']}, 'CompanyName': {'type': ['string', 'null']}, 'Active': {'type': ['boolean', 'null']}, 'StartDate': {'format': 'date-time', 'type': ['string', 'null']}, 'EndDate': {'format': 'date-time', 'type': ['string', 'null']}, 'Contract': {'type': ['integer', 'null']}, 'SalaryPayRule': {'type': ['integer', 'null']}, 'ContractFile': {'type': ['integer', 'null']}, 'PayrollId': {'type': ['string', 'null']}, 'PayPeriod': {'type': ['integer', 'null']}, 'HistoryId': {'type': ['integer', 'null']}, 'Creator': {'type': ['integer', 'null']}, 'Created': {'format': 'date-time', 'type': ['string', 'null']}, 'Modified': {'format': 'date-time', 'type': ['string', 'null']}, 'BaseRate': {'type': ['number', 'null']}, 'Config': {'type': ['object', 'null']}}, 'type': 'object', 'additionalProperties': False} Errors during transform: [Config: [] does not match {'type': ['object', 'null']}, : {'Id': 14811, 'EmployeeId': 10544, 'PayPoint': 60, 'EmpType': 1, 'CompanyName': None, 'Active': True, 'StartDate': '2023-04-18T00:00:00+02:00', 'EndDate': None, 'Contract': 757, 'SalaryPayRule': 0, 'ContractFile': None, 'PayrollId': '[REDACTED]', 'PayPeriod': 1, 'HistoryId': 104199, 'Creator': 8450, 'Created': '2023-04-17T02:08:54+02:00', 'Modified': '2023-04-17T02:08:57+02:00', 'BaseRate': None, 'Config': '[]', '_DPMetaData': {'System': 'EmployeeAgreement', 'CreatorInfo': {'Id': 8450, 'DisplayName': '[REDACTED]', 'EmployeeProfile': 8456, 'Employee': 8456, 'Photo': '[REDACTED]', 'Pronouns': 0, 'CustomPronouns': ''}}} does not match {'properties': {'Id': {'type': ['integer', 'null']}, 'EmployeeId': {'type': ['integer', 'null']}, 'PayPoint': {'type': ['integer', 'null']}, 'EmpType': {'type': ['integer', 'null']}, 'CompanyName': {'type': ['string', 'null']}, 'Active': {'type': ['boolean', 'null']}, 'StartDate': {'format': 'date-time', 'type': ['string', 'null']}, 'EndDate': {'format': 'date-time', 'type': ['string', 'null']}, 'Contract': {'type': ['integer', 'null']}, 'SalaryPayRule': {'type': ['integer', 'null']}, 'ContractFile': {'type': ['integer', 'null']}, 'PayrollId': {'type': ['string', 'null']}, 'PayPeriod': {'type': ['integer', 'null']}, 'HistoryId': {'type': ['integer', 'null']}, 'Creator': {'type': ['integer', 'null']}, 'Created': {'format': 'date-time', 'type': ['string', 'null']}, 'Modified': {'format': 'date-time', 'type': ['string', 'null']}, 'BaseRate': {'type': ['number', 'null']}, 'Config': {'type': ['object', 'null']}}, 'type': 'object', 'additionalProperties': False}] ```

[P-Arnold]

Thanks for your contribution @oscar-latorre-av . That makes sense to me. I will look in to the code a bit more to see if there's a way to handle those empty array values. Or hopefully someone else can help! I have opened a PR with my initial suggestion here and have linked back to this discussion in the description.

[RhysHutchison]

Appreciate you reaching out @P-Arnold, apologies for not seeing this issue sooner. I'll make sure this channels issues become instantly visible in future!

For now, let me start looking into this issue and see what I can get sorted for you all!