Closed AM-I-Human closed 1 year ago
Hi @AM-I-Human could you please share the entire sync attempts log as an attached file? Even if there's a failure do you consider the replication as complete?
I can't retrieve it because airbyte hangs while showing the logs. I can see them using potrainer, but I cannot download them.
I downloaded the log, it is 300 mb, I can't upload it.
The job seems complete. I found that the data on snowflake (the source), contained the double of the records (the load loaded the same file twice). So in my views that needed to be transferred, there was 2 identical row for each row. I don't think that this caused the issue but the problem happen only with this data that had this issue.
logs-82-0.txt One log of a single table upload that is killed after competition. This table have not the problem that I sad before. So that is not the issue, as expected.
2022-02-24 10:12:06 [43mdestination [0m > 2022-02-24 10:12:06 [32mINFO [m i.a.i.d.m.MongodbRecordConsumer(close):88 - Migration finished with no explicit errors. Copying data from tmp tables to permanent
2022-02-24 10:12:16 destination > /airbyte/javabase.sh: line 26: 10 Killed /airbyte/bin/"$APPLICATION" "$@"
The problem happens even in local with
We have a similar behavior.
Source: Salesforce Destination: MongoDB (DocDB AWS cluster) Logs: logs-71.txt
First, line 2246:
2022-08-04 01:41:05 [43mdestination[0m > 2022-08-04 01:41:05 [32mINFO[m i.a.i.d.m.MongodbRecordConsumer(close):89 - Migration finished with no explicit errors. Copying data from tmp tables to permanent
2022-08-04 01:46:51 [43mdestination[0m > /airbyte/javabase.sh: line 26: 9 Killed /airbyte/bin/"$APPLICATION" "$@"
2022-08-04 01:46:51 [1;31mERROR[m i.a.w.g.DefaultReplicationWorker(run):181 - Sync worker failed.
2nd l. 3427:
2022-08-04 05:01:58 [32mINFO[m i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):325 - Records read: 875000 (9 GB)
2022-08-04 05:01:59 [32mINFO[m i.a.w.t.TemporalAttemptExecution(lambda$getCancellationChecker$3):191 - Running sync worker cancellation...
2022-08-04 05:01:59 [32mINFO[m i.a.w.g.DefaultReplicationWorker(cancel):444 - Cancelling replication worker...
2022-08-04 05:02:06 [32mINFO[m i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):325 - Records read: 876000 (9 GB)
2022-08-04 05:02:09 [32mINFO[m i.a.w.g.DefaultReplicationWorker(cancel):452 - Cancelling destination...
2022-08-04 05:02:09 [32mINFO[m i.a.w.i.DefaultAirbyteDestination(cancel):125 - Attempting to cancel destination process...
2022-08-04 05:02:09 [32mINFO[m i.a.w.i.DefaultAirbyteDestination(cancel):130 - Destination process exists, cancelling...
2022-08-04 05:02:09 [32mINFO[m i.a.w.g.DefaultReplicationWorker(run):175 - One of source or destination thread complete. Waiting on the other.
2022-08-04 05:02:09 [32mINFO[m i.a.w.g.DefaultReplicationWorker(run):177 - Source and destination threads complete.
2022-08-04 05:02:09 [32mINFO[m i.a.w.i.DefaultAirbyteDestination(cancel):132 - Cancelled destination process!
2022-08-04 05:02:09 [32mINFO[m i.a.w.g.DefaultReplicationWorker(cancel):459 - Cancelling source...
2022-08-04 05:02:09 [32mINFO[m i.a.w.i.DefaultAirbyteSource(cancel):142 - Attempting to cancel source process...
2022-08-04 05:02:09 [32mINFO[m i.a.w.i.DefaultAirbyteSource(cancel):147 - Source process exists, cancelling...
2022-08-04 05:02:09 [33mWARN[m i.a.c.i.LineGobbler(voidCall):86 - airbyte-source gobbler IOException: Stream closed. Typically happens when cancelling a job.
2022-08-04 05:02:09 [1;31mERROR[m i.a.w.g.DefaultReplicationWorker(run):181 - Sync worker failed.
Is cancellation due to the 1st. error log?
EDIT: We identified that this interrupt is encountered when memory limit is reached. We tried to add swap, nothing changes.
What is causing this sync to hit 8Gb RAM ? (we're currently investigating with ctop
to see what container is consuming the RAM)
Answer: The destination writer (mongodb) is taking all the RAM, could we do the same 'batch payload' you did with other destinations?
I discover that machine was used by another service. I think it was out of memory issue, and docker killed the container. Now the problem seems to be resolved on the last version I tried > 0.50.12. I close the issue because seems obsolete to me.
Environment
Current Behavior
After a sync from a Snowflake Database to a MongoDB database. After the copy was completed the process was killed and there are different attempts.
Expected Behavior
The process will continue and complete the copy between temporary collections to permanents one
Logs
LOG
``` source - 2022-01-16 19:06:23 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-01-16 19:06:17 INFO i.a.i.b.IntegrationRunner(run):133 - Completed integration: io.airbyte.integrations.source.snowflake.SnowflakeSource source - 2022-01-16 19:06:23 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-01-16 19:06:17 INFO i.a.i.s.s.SnowflakeSource(main):31 - completed source: class io.airbyte.integrations.source.snowflake.SnowflakeSource 2022-01-16 19:06:23 INFO () DefaultReplicationWorker(run):138 - Source thread complete. 2022-01-16 19:06:23 INFO () DefaultReplicationWorker(run):139 - Waiting for destination thread to join. destination - 2022-01-16 19:06:25 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-01-16 19:06:25 INFO i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):60 - Airbyte message consumer: succeeded. destination - 2022-01-16 19:06:25 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-01-16 19:06:25 INFO i.a.i.d.m.MongodbRecordConsumer(close):88 - Migration finished with no explicit errors. Copying data from tmp tables to permanent destination - 2022-01-16 19:09:23 ERROR () LineGobbler(voidCall):82 - /airbyte/javabase.sh: line 26: 8 Killed /airbyte/bin/"$APPLICATION" "$@" 2022-01-16 19:09:24 INFO () DefaultReplicationWorker(run):141 - Destination thread complete. 2022-01-16 19:09:24 ERROR () DefaultReplicationWorker(run):145 - Sync worker failed. io.airbyte.workers.WorkerException: Destination process exit with code 137. This warning is normal if the job was cancelled. at io.airbyte.workers.protocols.airbyte.DefaultAirbyteDestination.close(DefaultAirbyteDestination.java:114) ~[io.airbyte-airbyte-workers-0.33.9-alpha.jar:?] at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:143) ~[io.airbyte-airbyte-workers-0.33.9-alpha.jar:?] at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:49) ~[io.airbyte-airbyte-workers-0.33.9-alpha.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:167) ~[io.airbyte-airbyte-workers-0.33.9-alpha.jar:?] at java.lang.Thread.run(Thread.java:832) [?:?] 2022-01-16 19:09:24 INFO () DefaultReplicationWorker(run):169 - sync summary: io.airbyte.config.ReplicationAttemptSummary@5ad5811c[status=failed,recordsSynced=6413690,bytesSynced=2414834373,startTime=1642351465642,endTime=1642360164103] 2022-01-16 19:09:24 INFO () DefaultReplicationWorker(run):178 - Source did not output any state messages 2022-01-16 19:09:24 WARN () DefaultReplicationWorker(run):186 - State capture: No new state, falling back on input state: io.airbyte.config.State@258511ff[state={}] 2022-01-16 19:09:24 INFO () TemporalAttemptExecution(get):137 - Stopping cancellation check scheduling... ```Steps to Reproduce
Are you willing to submit a PR?
Yes, but I don't know if my employer let me (in this case I don't know where is the problem)