search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.19k stars 4.14k forks source link

Destination Snowflake: unsafe memory access exception #2107

Closed sherifnada closed 3 years ago

sherifnada commented 3 years ago

Description Behavior

A user reported the following exception when trying to sync a 40gb table from MySQL:

2021-02-16 13:27:22 INFO (/tmp/workspace/100/0) LineGobbler(voidCall):69 - 2021-02-16 13:27:22 [32mINFO[m i.a.i.d.s.SnowflakeSqlOperations(insertRecords):60 - {} - actual size of batch: 933
2021-02-16 13:27:30 INFO (/tmp/workspace/100/0) LineGobbler(voidCall):69 - 2021-02-16 13:27:30 [32mINFO[m i.a.i.d.b.BufferedStreamConsumer(writeStreamsWithNRecords):194 - {} - Writing stream tripactions.booking_flight. Max batch size: 10000, Actual batch size: 502, Remaining buffered records: 0
2021-02-16 13:27:30 INFO (/tmp/workspace/100/0) LineGobbler(voidCall):69 - 2021-02-16 13:27:30 [32mINFO[m i.a.i.d.s.SnowflakeSqlOperations(insertRecords):60 - {} - actual size of batch: 502
2021-02-16 13:27:45 INFO (/tmp/workspace/100/0) LineGobbler(voidCall):69 - 2021-02-16 13:27:45 [32mINFO[m i.a.i.d.b.BufferedStreamConsumer(writeStreamsWithNRecords):194 - {} - Writing stream tripactions.booking_flight. Max batch size: 10000, Actual batch size: 1001, Remaining buffered records: 0
2021-02-16 13:27:45 INFO (/tmp/workspace/100/0) LineGobbler(voidCall):69 - 2021-02-16 13:27:45 [32mINFO[m i.a.i.d.s.SnowflakeSqlOperations(insertRecords):60 - {} - actual size of batch: 1001
2021-02-16 13:27:53 INFO (/tmp/workspace/100/0) LineGobbler(voidCall):69 - 2021-02-16 13:27:53 [32mINFO[m i.a.i.d.b.BufferedStreamConsumer(writeStreamsWithNRecords):194 - {} - Writing stream tripactions.booking_flight. Max batch size: 10000, Actual batch size: 798, Remaining buffered records: 0
2021-02-16 13:27:53 INFO (/tmp/workspace/100/0) LineGobbler(voidCall):69 - 2021-02-16 13:27:53 [32mINFO[m i.a.i.d.s.SnowflakeSqlOperations(insertRecords):60 - {} - actual size of batch: 798
2021-02-16 13:28:17 INFO (/tmp/workspace/100/0) LineGobbler(voidCall):69 - 2021-02-16 13:28:17 [32mINFO[m i.a.i.b.FailureTrackingConsumer(close):64 - {} - hasFailed: false.
2021-02-16 13:28:17 INFO (/tmp/workspace/100/0) LineGobbler(voidCall):69 - 2021-02-16 13:28:17 [32mINFO[m i.a.i.d.b.BufferedStreamConsumer(close):164 - {} - executing on success close procedure.
2021-02-16 13:28:17 ERROR (/tmp/workspace/100/0) LineGobbler(voidCall):69 - Exception in thread "main" java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
2021-02-16 13:28:17 ERROR (/tmp/workspace/100/0) LineGobbler(voidCall):69 - at io.airbyte.integrations.base.IntegrationRunner.consumeWriteStream(IntegrationRunner.java:126)
2021-02-16 13:28:17 ERROR (/tmp/workspace/100/0) LineGobbler(voidCall):69 - at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:112)
2021-02-16 13:28:17 ERROR (/tmp/workspace/100/0) LineGobbler(voidCall):69 - at io.airbyte.integrations.destination.snowflake.SnowflakeDestination.main(SnowflakeDestination.java:59)
2021-02-16 13:28:17 ERROR (/tmp/workspace/100/0) LineGobbler(voidCall):69 - Suppressed: java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
2021-02-16 13:28:17 ERROR (/tmp/workspace/100/0) LineGobbler(voidCall):69 - at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._parsePosNumber(ReaderBasedJsonParser.java:1288)
2021-02-16 13:28:17 ERROR (/tmp/workspace/100/0) LineGobbler(voidCall):69 - at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextFieldName(ReaderBasedJsonParser.java:962)
2021-02-16 13:28:17 ERROR (/tmp/workspace/100/0) LineGobbler(voidCall):69 - at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:250)
2021-02-16 13:28:17 ERROR (/tmp/workspace/100/0) LineGobbler(voidCall):69 - at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:68)
2021-02-16 13:28:17 ERROR (/tmp/workspace/100/0) LineGobbler(voidCall):69 - at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15)
2021-02-16 13:28:17 ERROR (/tmp/workspace/100/0) LineGobbler(voidCall):69 - at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
2021-02-16 13:28:17 ERROR (/tmp/workspace/100/0) LineGobbler(voidCall):69 - at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288)
2021-02-16 13:28:17 ERROR (/tmp/workspace/100/0) LineGobbler(voidCall):69 - at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
2021-02-16 13:28:17 ERROR (/tmp/workspace/100/0) LineGobbler(voidCall):69 - at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4218)
2021-02-16 13:28:17 ERROR (/tmp/workspace/100/0) LineGobbler(voidCall):69 - at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3214)
2021-02-16 13:28:17 ERROR (/tmp/workspace/100/0) LineGobbler(voidCall):69 - at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3182)
2021-02-16 13:28:17 ERROR (/tmp/workspace/100/0) LineGobbler(voidCall):69 - at io.airbyte.commons.json.Jsons.deserialize(Jsons.java:61)
2021-02-16 13:28:17 ERROR (/tmp/workspace/100/0) LineGobbler(voidCall):69 - at io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer.lambda$writeStreamsWithNRecords$1(BufferedStreamConsumer.java:191)
2021-02-16 13:28:17 ERROR (/tmp/workspace/100/0) LineGobbler(voidCall):69 - at java.base/java.util.stream.Referenc2021-02-16 13:28:17 DEBUG (/tmp/workspace/100/0) DefaultAirbyteSource(close):109 - Closing tap process

Severity of the bug for you

High - blocking user from using Airbyte

Airbyte Version

MySQL 0.1.10, Snowflake 0.1.17, Airbyte v0.14.0-alpha

Additional context

slack conversation

┆Issue is synchronized with this Asana task by Unito

sherifnada commented 3 years ago

the user reported that the issue is because disk space filled up. Need to investigate why disk space is filling up due to this sync. Airbyte is supposed to be streaming data.

cgardens commented 3 years ago

the user reported that the issue is because disk space filled up.

why would the memory issue be due to lack of disk space?

69mb commented 3 years ago

Have the same issue with the Snowflake destination but the root exception is a JsonParseException:

2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 - Exception in thread "main" java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -       at java.base/java.io.Reader.read(Reader.java:188)                                                                                                       
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -       at java.base/java.util.Scanner.readInput(Scanner.java:882)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -       at java.base/java.util.Scanner.findWithinHorizon(Scanner.java:1796)                                                                                           2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -       at java.base/java.util.Scanner.hasNextLine(Scanner.java:1610)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -       at io.airbyte.integrations.base.IntegrationRunner.consumeWriteStream(IntegrationRunner.java:128)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -       at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:115)     
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -       at io.airbyte.integrations.destination.snowflake.SnowflakeDestination.main(SnowflakeDestination.java:59)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -       Suppressed: java.lang.RuntimeException: java.lang.RuntimeException: com.fasterxml.jackson.core.JsonParseException: Illegal character ((CTRL-CHAR, code 0)): on
ly regular white space (\r, \n, \t) is allowed between tokens                                                                                                                                                                                 
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -  at [Source: (String)""; line: 1, column: 2]                                                                                                                        
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer.writeStreamsWithNRecords(BufferedStreamConsumer.java:198)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer.close(BufferedStreamConsumer.java:171)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at io.airbyte.integrations.base.FailureTrackingConsumer.close(FailureTrackingConsumer.java:77)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at io.airbyte.integrations.base.IntegrationRunner.consumeWriteStream(IntegrationRunner.java:126)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               ... 2 more                                                                                                                                            
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -       Caused by: java.lang.RuntimeException: com.fasterxml.jackson.core.JsonParseException: Illegal character ((CTRL-CHAR, code 0)): only regular white space (\r, \
n, \t) is allowed between tokens                                                                                                                                                                                                              
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -  at [Source: (String)""; line: 1, column: 2]
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at io.airbyte.commons.json.Jsons.deserialize(Jsons.java:70)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer.lambda$writeStreamsWithNRecords$1(BufferedStreamConsumer.java:1
91)                                                        
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at java.base/java.util.stream.SliceOps$1$1.accept(SliceOps.java:199)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at io.airbyte.commons.lang.Queues$1.tryAdvance(Queues.java:52)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:127)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:502)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:488)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer.writeStreamsWithNRecords(BufferedStreamConsumer.java:192)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               ... 5 more
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -       Caused by: com.fasterxml.jackson.core.JsonParseException: Illegal character ((CTRL-CHAR, code 0)): only regular white space (\r, \n, \t) is allowed between to
kens                                                       
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -  at [Source: (String)""; line: 1, column: 2]
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1840)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:712)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at com.fasterxml.jackson.core.base.ParserMinimalBase._throwInvalidSpace(ParserMinimalBase.java:690)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._skipWSOrEnd(ReaderBasedJsonParser.java:2372)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:671)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4356)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4205)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3214)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3182)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               at io.airbyte.commons.json.Jsons.deserialize(Jsons.java:68)
2021-04-05 06:50:05 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 -               ... 17 more
2021-04-05 06:50:39 ERROR (/tmp/workspace/1/0) LineGobbler(voidCall):69 - read unix @->/var/run/docker.sock: read: connection reset by peer
2021-04-05 06:50:39 DEBUG (/tmp/workspace/1/0) DefaultAirbyteSource(close):109 - Closing tap process
69mb commented 3 years ago

Turns out I was running out of disk space as well. I'm doing the initial sync of a large table from Postgres to Snowflake and there are a bunch of page files created in /tmp by the destination-snowflake connector that were filling up the disk.

sherifnada commented 3 years ago

@cgardens didn't see your initial comment. My guess is bigqueue but I haven't checked. @69mb would you be able to do an ls of the file names piling up on disk to give us an idea of what they are?

69mb commented 3 years ago

@sherifnada this is what I've got(replaced schema name and table name with schema.table):

# ls -lR /tmp/
/tmp/:
total 0
drwxr-xr-x 1 root root 16 Apr  6 16:09 hsperfdata_root
drwx------ 3 root root 30 Apr  6 16:10 queues14306134417334400042

/tmp/hsperfdata_root:
total 32
-rw------- 1 root root 32768 Apr  6 16:39 12

/tmp/queues14306134417334400042:
total 0
drwxr-xr-x 3 root root 30 Apr  6 16:10 schema.table

/tmp/queues14306134417334400042/schema.table:
total 0
drwxr-xr-x 6 root root 67 Apr  6 16:10 schema.table

/tmp/queues14306134417334400042/schema.table/schema.table:
total 8
drwxr-xr-x 2 root root 4096 Apr  6 16:39 data
drwxr-xr-x 2 root root   24 Apr  6 16:10 front_index
drwxr-xr-x 2 root root 4096 Apr  6 16:39 index
drwxr-xr-x 2 root root   24 Apr  6 16:10 meta_data

/tmp/queues14306134417334400042/schema.table/schema.table/data:
total 5928868
-rw-r--r-- 1 root root 134217728 Apr  6 16:10 page-0.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:11 page-1.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:17 page-10.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:17 page-11.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:18 page-12.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:19 page-13.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:19 page-14.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:20 page-15.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:21 page-16.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:21 page-17.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:22 page-18.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:23 page-19.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:12 page-2.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:23 page-20.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:24 page-21.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:25 page-22.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:25 page-23.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:26 page-24.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:27 page-25.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:27 page-26.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:28 page-27.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:29 page-28.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:29 page-29.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:12 page-3.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:30 page-30.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:31 page-31.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:31 page-32.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:32 page-33.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:33 page-34.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:33 page-35.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:34 page-36.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:35 page-37.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:35 page-38.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:36 page-39.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:13 page-4.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:37 page-40.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:37 page-41.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:38 page-42.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:38 page-43.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:39 page-44.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:39 page-45.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:14 page-5.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:14 page-6.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:15 page-7.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:16 page-8.dat
-rw-r--r-- 1 root root 134217728 Apr  6 16:16 page-9.dat

/tmp/queues14306134417334400042/schema.table/schema.table/front_index:
total 4
-rw-r--r-- 1 root root 8 Apr  6 16:39 page-0.dat

/tmp/queues14306134417334400042/schema.table/schema.table/index:
total 424844
-rw-r--r-- 1 root root 4194304 Apr  6 16:10 page-0.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:10 page-1.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:13 page-10.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:39 page-100.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:39 page-101.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:39 page-102.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:39 page-103.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:13 page-11.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:14 page-12.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:14 page-13.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:14 page-14.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:15 page-15.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:15 page-16.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:15 page-17.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:15 page-18.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:16 page-19.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:11 page-2.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:16 page-20.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:16 page-21.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:17 page-22.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:17 page-23.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:17 page-24.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:17 page-25.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:18 page-26.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:18 page-27.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:18 page-28.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:19 page-29.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:11 page-3.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:19 page-30.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:19 page-31.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:19 page-32.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:20 page-33.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:20 page-34.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:20 page-35.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:21 page-36.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:21 page-37.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:21 page-38.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:21 page-39.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:11 page-4.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:22 page-40.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:22 page-41.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:22 page-42.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:23 page-43.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:23 page-44.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:23 page-45.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:23 page-46.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:24 page-47.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:24 page-48.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:24 page-49.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:12 page-5.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:25 page-50.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:25 page-51.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:25 page-52.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:25 page-53.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:26 page-54.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:26 page-55.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:26 page-56.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:27 page-57.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:27 page-58.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:27 page-59.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:12 page-6.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:27 page-60.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:28 page-61.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:28 page-62.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:28 page-63.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:28 page-64.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:29 page-65.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:29 page-66.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:29 page-67.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:30 page-68.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:30 page-69.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:12 page-7.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:30 page-70.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:30 page-71.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:31 page-72.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:31 page-73.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:31 page-74.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:32 page-75.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:32 page-76.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:32 page-77.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:32 page-78.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:33 page-79.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:12 page-8.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:33 page-80.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:33 page-81.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:34 page-82.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:34 page-83.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:34 page-84.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:34 page-85.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:35 page-86.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:35 page-87.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:35 page-88.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:35 page-89.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:13 page-9.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:36 page-90.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:36 page-91.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:36 page-92.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:37 page-93.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:37 page-94.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:37 page-95.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:37 page-96.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:38 page-97.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:38 page-98.dat
-rw-r--r-- 1 root root 4194304 Apr  6 16:38 page-99.dat

/tmp/queues14306134417334400042/schema.table/schema.table/meta_data:
total 4
-rw-r--r-- 1 root root 16 Apr  6 16:39 page-0.dat
sherifnada commented 3 years ago

@69mb thanks!

@cgardens this is potentially solved by the bulk upload mechanism

davinchia commented 3 years ago

@69mb is this still happening at later versions? I think we should have gotten this along the way.

davinchia commented 3 years ago

This should be fixed. I'm closing this. Please reopen if you still see this.