ConflictError when running a sync

[ppf2]

Encountered the following critical error while trying to test sync runs ins the S3 connector. This happens between various failed sync runs (where it had trouble connecting with S3). Certainly, we will look into the connection issue, but the following indicates that they might be a race condition/safeguard issue causing it to throw a conflict error on the connector sync tracking document:

[FMWK][14:50:33][INFO] Filtering validation errors: None
[FMWK][14:50:33][INFO] *** Connector baEuzIUB28qdopNRCptS HEARTBEAT
[FMWK][14:50:33][DEBUG] sync_now is true, syncing!
[FMWK][14:50:33][DEBUG] Syncing 's3'
[FMWK][14:50:33][CRITICAL] ConflictError(409, 'version_conflict_engine_exception', '[baEuzIUB28qdopNRCptS]: version conflict, required seqNo [287], primary term [6]. current document has seqNo [288] and primary term [6]')
Traceback (most recent call last):
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-main/connectors/services/sync.py", line 119, in _run
    await self._one_sync(connector, es, sync_now)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-main/connectors/services/sync.py", line 79, in _one_sync
    await connector.sync(es, self.idling, sync_now)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-main/connectors/byoc.py", line 582, in sync
    job = await self._sync_starts()
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-main/connectors/byoc.py", line 436, in _sync_starts
    await self.sync_doc()
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-main/connectors/byoc.py", line 395, in sync_doc
    await self.index.save(self)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-main/connectors/byoc.py", line 104, in save
    return await self.client.update(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-main/lib/python3.10/site-packages/elasticsearch/_async/client/__init__.py", line 4482, in update
    return await self.perform_request(  # type: ignore[return-value]
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-main/lib/python3.10/site-packages/elasticsearch/_async/client/_base.py", line 321, in perform_request
    raise HTTP_EXCEPTIONS.get(meta.status, ApiError)(
elasticsearch.ConflictError: ConflictError(409, 'version_conflict_engine_exception', '[baEuzIUB28qdopNRCptS]: version conflict, required seqNo [287], primary term [6]. current document has seqNo [288] and primary term [6]')

[akanshi-crest]

This issue needs to be taken care at the framework level. Attaching Issue link: #118 cc: @tarekziade

[danajuratoni]

@tarekziade linked issue is done, can this be closed?