Open marcosmarxm opened 2 years ago
Comment made from Zendesk by Marcos Marx on 2022-08-31 at 18:39:
Thanks for reporting this Ryan I opened the issue https://github.com/airbytehq/airbyte/issues/16182 to solve the issue.
I've written a fix for this on my fork, but not sure where to aim it when submitting the PR.
@wirelessrpm please read Airbyte's docs about contributing to the project: https://docs.airbyte.com/contributing-to-airbyte/
FYI: I just also ran into this. I dug a bit in the code, and the OVERWRITE
mode only works when the Upsert Records
checkbox is deselected (default it is on). When it is off, it performs a sync to a temporary index which replaces the original index. I suppose to have some form of atomic replacement to mitigate any downtime.
Not sure if this is the most obvious solution, but it does seem to work 👍
TLDR: can be fixed by disabling the Upsert Records
config.
FYI: I just also ran into this. I dug a bit in the code, and the
OVERWRITE
mode only works when theUpsert Records
checkbox is deselected (default it is on). When it is off, it performs a sync to a temporary index which replaces the original index. I suppose to have some form of atomic replacement to mitigate any downtime.
- ElasticsearchWriteConfig.java#L86-L88 determines whether to use temporary index.
- ElasticsearchAirbyteMessageConsumerFactory.java#L125-L131 selects which index to write to
- ElasticsearchAirbyteMessageConsumerFactory.java#L68-L70 finally replaces the index
Not sure if this is the most obvious solution, but it does seem to work 👍
TLDR: can be fixed by disabling the
Upsert Records
config.
Hi sir, how to disable Upsert Records to use Overwrite mode with ElasticSearch? Where can I can config in Airbyte source code or in ElasticSearch config?
’Upsert Records’ is just a configuration setting in Airbyte when setting up the connection. You don't need to change the code.
Hi Sir, I don't see that config during the setup of my connections. Plz show me where can I find that configuration? Many thanks!
This Github issue is synchronized with Zendesk:
Ticket ID: #2084 Priority: normal Group: Community Assistance Engineer Assignee: Marcos Marx
Original ticket description: