search

singer-io / getting-started

This repository is a getting started guide to Singer.
https://singer.io
1.25k stars 148 forks source link

Intermittent Errors in Salesforce Marketing Cloud Integration #85

Open FionaYiZhao opened 3 years ago

FionaYiZhao commented 3 years ago

Hi Community,

We are using the SFMC integration to sync data from SFMC to BigQuery. There are intermittent errors happening quite often. Sometimes it last for over a day, sometimes last for a couple of hours. And they can recover at sometime later on w/o any work from our side. Could you help us to understand

  1. the root cause of such error
  2. does the integration need to end up omitting some data in order to recover such error ? (or what's changed for the sync to move on w/o errors)

Thanks!

One error log example attached below

` 2021-05-27 11:55:57,858Z target - INFO replicated 6805 records from "data_extension.PROD_Businesses" endpoint 2021-05-27 11:55:58,008Z tap - INFO Getting more results from 'DataExtensionObject' endpoint 2021-05-27 11:55:58,555Z tap - ERROR <suds.sax.document.Document object at 0x7f6e142ad470> 2021-05-27 11:55:58,555Z tap - ERROR :2:62: syntax error 2021-05-27 11:55:58,555Z tap - Traceback (most recent call last): 2021-05-27 11:55:58,555Z tap - File "/root/.pyenv/versions/3.5.2/lib/python3.5/xml/sax/expatreader.py", line 210, in feed 2021-05-27 11:55:58,555Z tap - self._parser.Parse(data, isFinal) 2021-05-27 11:55:58,555Z tap - xml.parsers.expat.ExpatError: syntax error: line 2, column 62 2021-05-27 11:55:58,555Z tap - 2021-05-27 11:55:58,555Z tap - During handling of the above exception, another exception occurred: 2021-05-27 11:55:58,555Z tap - 2021-05-27 11:55:58,556Z tap - Traceback (most recent call last): 2021-05-27 11:55:58,556Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_exacttarget/init.py", line 136, in do_sync 2021-05-27 11:55:58,556Z tap - stream_accessor.sync() 2021-05-27 11:55:58,556Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_exacttarget/dao.py", line 74, in sync 2021-05-27 11:55:58,556Z tap - return self.sync_data() 2021-05-27 11:55:58,556Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_exacttarget/endpoints/data_extensions.py", line 283, in sync_data 2021-05-27 11:55:58,556Z tap - replication_key=replication_key) 2021-05-27 11:55:58,556Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_exacttarget/endpoints/data_extensions.py", line 209, in _replicate 2021-05-27 11:55:58,556Z tap - for row in result: 2021-05-27 11:55:58,556Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_exacttarget/client.py", line 153, in request_from_cursor 2021-05-27 11:55:58,556Z tap - response = tap_exacttargetgetMoreResults(cursor, batch_size=batch_size) 2021-05-27 11:55:58,556Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_exacttarget/fuel_overrides.py", line 32, in tap_exacttargetgetMoreResults 2021-05-27 11:55:58,556Z tap - obj = TapExacttargetET_Continue(cursor.auth_stub, cursor.last_request_id, batch_size) 2021-05-27 11:55:58,556Z target - INFO Serializing batch with 1518 messages for table data_extension.PROD_Businesses 2021-05-27 11:55:58,557Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_exacttarget/fuel_overrides.py", line 26, in init__ 2021-05-27 11:55:58,557Z tap - response = auth_stub.soap_client.service.Retrieve(ws_continueRequest) 2021-05-27 11:55:58,557Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/suds/client.py", line 521, in call 2021-05-27 11:55:58,557Z tap - return client.invoke(args, kwargs) 2021-05-27 11:55:58,557Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/suds/client.py", line 581, in invoke 2021-05-27 11:55:58,557Z tap - result = self.send(soapenv) 2021-05-27 11:55:58,557Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/suds/client.py", line 621, in send 2021-05-27 11:55:58,557Z tap - original_soapenv=original_soapenv) 2021-05-27 11:55:58,557Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/suds/client.py", line 661, in process_reply 2021-05-27 11:55:58,557Z tap - replyroot = _parse(reply) 2021-05-27 11:55:58,557Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/suds/client.py", line 832, in _parse 2021-05-27 11:55:58,557Z tap - return Parser().parse(string=string) 2021-05-27 11:55:58,558Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/suds/sax/parser.py", line 133, in parse 2021-05-27 11:55:58,558Z tap - sax.parse(source) 2021-05-27 11:55:58,558Z tap - File "/root/.pyenv/versions/3.5.2/lib/python3.5/xml/sax/expatreader.py", line 110, in parse 2021-05-27 11:55:58,558Z tap - xmlreader.IncrementalParser.parse(self, source) 2021-05-27 11:55:58,558Z tap - File "/root/.pyenv/versions/3.5.2/lib/python3.5/xml/sax/xmlreader.py", line 125, in parse 2021-05-27 11:55:58,558Z tap - self.feed(buffer) 2021-05-27 11:55:58,558Z tap - File "/root/.pyenv/versions/3.5.2/lib/python3.5/xml/sax/expatreader.py", line 214, in feed 2021-05-27 11:55:58,558Z tap - self._err_handler.fatalError(exc) 2021-05-27 11:55:58,558Z tap - File "/root/.pyenv/versions/3.5.2/lib/python3.5/xml/sax/handler.py", line 38, in fatalError 2021-05-27 11:55:58,558Z tap - raise exception 2021-05-27 11:55:58,558Z tap - xml.sax._exceptions.SAXParseException: :2:62: syntax error 2021-05-27 11:55:58,558Z tap - ERROR Failed to sync endpoint, moving on!

`