airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.51k stars 4k forks source link

Basic normalization doesn't extract nested objects using `$ref`/`$defs` #8241

Closed monai closed 2 years ago

monai commented 2 years ago

Enviroment

Current Behavior

Basic normalization doesn't extract nested objects when they're referenced using $ref and defined using $defs in the same schema. The current workaround is to inline all, usually duplicated, object definitions.

Example schema that uses $ref/$defs: https://github.com/airbytehq/airbyte/pull/8177/commits/5303b0b848732f3c79643e465bee8d3c79808b0b

Expected Behavior

I expect to deduplicate schemas and have all the benefits of basic normalization nested object extraction.

Logs

Failed nested object extraction:

normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - WARN: Unknown type for column Address at processed_order_details/CustomerInfo
normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - WARN: Unknown type for column BillingAddress at processed_order_details/CustomerInfo
LOG ``` 2021-11-22 13:40:15 INFO () WorkerRun(call):49 - Executing worker wrapper. Airbyte version: 0.32.0-alpha 2021-11-22 13:40:16 INFO () TemporalAttemptExecution(get):116 - Executing worker wrapper. Airbyte version: 0.32.0-alpha 2021-11-22 13:40:16 WARN () Databases(createPostgresDatabaseWithRetry):41 - Waiting for database to become available... 2021-11-22 13:40:16 INFO () JobsDatabaseInstance(lambda$static$2):25 - Testing if jobs database is ready... 2021-11-22 13:40:16 INFO () Databases(createPostgresDatabaseWithRetry):58 - Database available! 2021-11-22 13:40:16 WARN () JsonMetaSchema(newValidator):338 - Unknown keyword example - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2021-11-22 13:40:16 INFO () DefaultReplicationWorker(run):82 - start sync worker. job id: 232 attempt id: 0 2021-11-22 13:40:16 INFO () DefaultReplicationWorker(run):91 - configured sync modes: {null.stock_items=full_refresh - append, null.processed_order_details=full_refresh - append, null.processed_orders=full_refresh - append, null.stock_locations=full_refresh - append} 2021-11-22 13:40:16 INFO () DefaultAirbyteDestination(start):64 - Running destination... 2021-11-22 13:40:16 INFO () LineGobbler(voidCall):82 - Checking if airbyte/destination-postgres:0.3.12 exists... 2021-11-22 13:40:16 INFO () LineGobbler(voidCall):82 - airbyte/destination-postgres:0.3.12 not found locally. Attempting to pull the image... 2021-11-22 13:40:36 INFO () LineGobbler(voidCall):82 - Pulled airbyte/destination-postgres:0.3.12 from remote. 2021-11-22 13:40:36 INFO () DockerProcessFactory(create):127 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/232/0 --network host --log-driver none airbyte/destination-postgres:0.3.12 write --config destination_config.json --catalog destination_catalog.json 2021-11-22 13:40:36 INFO () LineGobbler(voidCall):82 - Checking if airbyte/source-linnworks:dev exists... 2021-11-22 13:40:36 INFO () LineGobbler(voidCall):82 - airbyte/source-linnworks:dev was found locally. 2021-11-22 13:40:36 INFO () DockerProcessFactory(create):127 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/232/0 --network host --log-driver none airbyte/source-linnworks:dev read --config source_config.json --catalog source_catalog.json 2021-11-22 13:40:36 INFO () DefaultReplicationWorker(lambda$getDestinationOutputRunnable$3):226 - Destination output thread started. 2021-11-22 13:40:36 INFO () DefaultReplicationWorker(run):119 - Waiting for source thread to join. 2021-11-22 13:40:36 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):190 - Replication thread started. destination - 2021-11-22 13:40:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:42 INFO i.a.i.d.p.PostgresDestination(main):69 - {} - starting destination: class io.airbyte.integrations.destination.postgres.PostgresDestination destination - 2021-11-22 13:40:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:42 INFO i.a.i.b.IntegrationRunner(run):76 - {} - Running integration: io.airbyte.integrations.base.ssh.SshWrappedDestination destination - 2021-11-22 13:40:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:42 INFO i.a.i.b.IntegrationCliParser(parseOptions):118 - {} - integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json} destination - 2021-11-22 13:40:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:42 INFO i.a.i.b.IntegrationRunner(run):80 - {} - Command: WRITE destination - 2021-11-22 13:40:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:42 INFO i.a.i.b.IntegrationRunner(run):81 - {} - Integration config: IntegrationConfig{command=WRITE, configPath='destination_config.json', catalogPath='destination_catalog.json', statePath='null'} destination - 2021-11-22 13:40:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:42 WARN c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword destination - 2021-11-22 13:40:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:42 WARN c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword destination - 2021-11-22 13:40:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:42 WARN c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword destination - 2021-11-22 13:40:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:42 WARN c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword multiline - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword destination - 2021-11-22 13:40:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:43 INFO i.a.i.b.s.SshTunnel(getInstance):170 - {} - Starting connection with method: NO_TUNNEL destination - 2021-11-22 13:40:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:43 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$toWriteConfig$0):96 - {} - Write config: WriteConfig{streamName=processed_order_details, namespace=null, outputSchemaName=linnworks, tmpTableName=_airbyte_tmp_ctb_processed_order_details, outputTableName=_airbyte_raw_processed_order_details, syncMode=append} destination - 2021-11-22 13:40:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:43 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$toWriteConfig$0):96 - {} - Write config: WriteConfig{streamName=processed_orders, namespace=null, outputSchemaName=linnworks, tmpTableName=_airbyte_tmp_dok_processed_orders, outputTableName=_airbyte_raw_processed_orders, syncMode=append} destination - 2021-11-22 13:40:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:43 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$toWriteConfig$0):96 - {} - Write config: WriteConfig{streamName=stock_items, namespace=null, outputSchemaName=linnworks, tmpTableName=_airbyte_tmp_yan_stock_items, outputTableName=_airbyte_raw_stock_items, syncMode=append} destination - 2021-11-22 13:40:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:43 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$toWriteConfig$0):96 - {} - Write config: WriteConfig{streamName=stock_locations, namespace=null, outputSchemaName=linnworks, tmpTableName=_airbyte_tmp_trv_stock_locations, outputTableName=_airbyte_raw_stock_locations, syncMode=append} destination - 2021-11-22 13:40:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:43 INFO i.a.i.d.b.BufferedStreamConsumer(startTracked):124 - {} - class io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer started. destination - 2021-11-22 13:40:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:43 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):121 - {} - Preparing tmp tables in destination started for 4 streams destination - 2021-11-22 13:40:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:43 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):125 - {} - Preparing tmp table in destination started for stream processed_order_details. schema: linnworks, tmp table name: _airbyte_tmp_ctb_processed_order_details destination - 2021-11-22 13:40:44 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:44 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):125 - {} - Preparing tmp table in destination started for stream processed_orders. schema: linnworks, tmp table name: _airbyte_tmp_dok_processed_orders destination - 2021-11-22 13:40:44 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:44 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):125 - {} - Preparing tmp table in destination started for stream stock_items. schema: linnworks, tmp table name: _airbyte_tmp_yan_stock_items destination - 2021-11-22 13:40:44 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:44 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):125 - {} - Preparing tmp table in destination started for stream stock_locations. schema: linnworks, tmp table name: _airbyte_tmp_trv_stock_locations destination - 2021-11-22 13:40:44 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:44 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):131 - {} - Preparing tables in destination completed. 2021-11-22 13:40:51 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 1000 2021-11-22 13:41:03 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 2000 2021-11-22 13:41:18 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 3000 2021-11-22 13:41:30 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 4000 2021-11-22 13:41:53 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 5000 2021-11-22 13:42:08 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 6000 2021-11-22 13:42:32 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 7000 2021-11-22 13:42:57 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 8000 2021-11-22 13:43:27 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 9000 2021-11-22 13:43:49 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 10000 2021-11-22 13:44:23 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 11000 2021-11-22 13:45:08 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 12000 2021-11-22 13:45:45 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 13000 2021-11-22 13:46:14 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 14000 2021-11-22 13:46:26 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 15000 2021-11-22 13:46:48 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 16000 2021-11-22 13:47:10 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 17000 2021-11-22 13:47:23 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 18000 2021-11-22 13:47:41 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 19000 2021-11-22 13:48:01 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 20000 2021-11-22 13:48:20 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 21000 2021-11-22 13:48:36 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 22000 2021-11-22 13:48:47 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 23000 2021-11-22 13:49:04 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 24000 2021-11-22 13:49:23 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 25000 2021-11-22 13:49:47 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 26000 2021-11-22 13:50:06 INFO () DefaultReplicationWorker(run):121 - Source thread complete. 2021-11-22 13:50:06 INFO () DefaultReplicationWorker(run):122 - Waiting for destination thread to join. destination - 2021-11-22 13:50:06 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:06 INFO i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):60 - {} - Airbyte message consumer: succeeded. destination - 2021-11-22 13:50:06 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:06 INFO i.a.i.d.b.BufferedStreamConsumer(close):199 - {} - executing on success close procedure. destination - 2021-11-22 13:50:23 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:23 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):160 - {} - Finalizing tables in destination started for 4 streams destination - 2021-11-22 13:50:23 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:23 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):165 - {} - Finalizing stream processed_order_details. schema linnworks, tmp table _airbyte_tmp_ctb_processed_order_details, final table _airbyte_raw_processed_order_details destination - 2021-11-22 13:50:23 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:23 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):165 - {} - Finalizing stream processed_orders. schema linnworks, tmp table _airbyte_tmp_dok_processed_orders, final table _airbyte_raw_processed_orders destination - 2021-11-22 13:50:23 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:23 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):165 - {} - Finalizing stream stock_items. schema linnworks, tmp table _airbyte_tmp_yan_stock_items, final table _airbyte_raw_stock_items destination - 2021-11-22 13:50:23 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:23 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):165 - {} - Finalizing stream stock_locations. schema linnworks, tmp table _airbyte_tmp_trv_stock_locations, final table _airbyte_raw_stock_locations destination - 2021-11-22 13:50:23 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:23 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):178 - {} - Executing finalization of tables. destination - 2021-11-22 13:50:27 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:27 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):180 - {} - Finalizing tables in destination completed. destination - 2021-11-22 13:50:27 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:27 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):183 - {} - Cleaning tmp tables in destination started for 4 streams destination - 2021-11-22 13:50:27 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:27 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):187 - {} - Cleaning tmp table in destination started for stream processed_order_details. schema linnworks, tmp table name: _airbyte_tmp_ctb_processed_order_details destination - 2021-11-22 13:50:27 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:27 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):187 - {} - Cleaning tmp table in destination started for stream processed_orders. schema linnworks, tmp table name: _airbyte_tmp_dok_processed_orders destination - 2021-11-22 13:50:27 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:27 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):187 - {} - Cleaning tmp table in destination started for stream stock_items. schema linnworks, tmp table name: _airbyte_tmp_yan_stock_items destination - 2021-11-22 13:50:27 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:27 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):187 - {} - Cleaning tmp table in destination started for stream stock_locations. schema linnworks, tmp table name: _airbyte_tmp_trv_stock_locations destination - 2021-11-22 13:50:27 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:27 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):192 - {} - Cleaning tmp tables in destination completed. destination - 2021-11-22 13:50:27 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:27 INFO i.a.i.b.IntegrationRunner(run):133 - {} - Completed integration: io.airbyte.integrations.base.ssh.SshWrappedDestination destination - 2021-11-22 13:50:27 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:27 INFO i.a.i.d.p.PostgresDestination(main):71 - {} - completed destination: class io.airbyte.integrations.destination.postgres.PostgresDestination 2021-11-22 13:50:28 INFO () DefaultReplicationWorker(run):124 - Destination thread complete. 2021-11-22 13:50:28 INFO () DefaultReplicationWorker(run):152 - sync summary: io.airbyte.config.ReplicationAttemptSummary@52540356[status=completed,recordsSynced=26782,bytesSynced=66088631,startTime=1637588416788,endTime=1637589028283] 2021-11-22 13:50:28 INFO () DefaultReplicationWorker(run):161 - Source did not output any state messages 2021-11-22 13:50:28 WARN () DefaultReplicationWorker(run):172 - State capture: No state retained. 2021-11-22 13:50:28 INFO () TemporalAttemptExecution(get):137 - Stopping cancellation check scheduling... 2021-11-22 13:50:28 INFO () ReplicationActivityImpl(replicate):121 - sync summary: io.airbyte.config.StandardSyncOutput@59e5da9f[standardSyncSummary=io.airbyte.config.StandardSyncSummary@15220a54[status=completed,recordsSynced=26782,bytesSynced=66088631,startTime=1637588416788,endTime=1637589028283],state=,outputCatalog=io.airbyte.protocol.models.ConfiguredAirbyteCatalog@6d3f01ea[streams=[io.airbyte.protocol.models.ConfiguredAirbyteStream@3d2c4068[stream=io.airbyte.protocol.models.AirbyteStream@4a6a5470[name=processed_order_details,jsonSchema={"type":"object","$defs":{"customer_address":{"type":"object","properties":{"Town":{"type":"string","description":"Customer's town."},"Region":{"type":"string","description":"Customer's region."},"Company":{"type":"string","description":"Customer's company name."},"Country":{"type":"string","description":"Customer's country."},"Address1":{"type":"string","description":"First line of customer address."},"Address2":{"type":"string","description":"Second line of customer address."},"Address3":{"type":"string","description":"Third line of customer address."},"FullName":{"type":"string","description":"Customer's first and second name."},"PostCode":{"type":"string","description":"Customer's postcode."},"Continent":{"type":"string","description":"Customer's continent"},"CountryId":{"type":"string"},"PhoneNumber":{"type":"string","description":"Customer's telephone number."},"EmailAddress":{"type":"string","description":"Customer's email address."}},"additionalProperties":false}},"$schema":"http://json-schema.org/draft-07/schema#","properties":{"Items":{"type":"array","items":{"type":"object","properties":{"SKU":{"type":"string","description":"Product SKU"},"Tax":{"type":"number","description":"Actual tax value on an item"},"Cost":{"type":"number","description":"Total item cost (exc tax)"},"Boxes":{"type":"array","items":{"type":"object","properties":{"BoxId":{"type":"integer","description":"Unique box id."},"Width":{"type":"number","description":"Width of the box"},"Height":{"type":"number","description":"Height of the box"},"Length":{"type":"number","description":"Depth of the box"},"Weight":{"type":"number","description":"Total weight of the box."},"Barcode":{"type":"string","description":"Box barcode, max 64 characters."},"BoxName":{"type":"string","description":"Box name max 16 characters"},"LogicalDelete":{"type":"boolean","description":"IsDeleted flag."},"StockItemIntId":{"type":"integer"},"PackagingTypeId":{"type":"string","description":"Packaging type id"},"ValuePercentage":{"type":"number","description":"Value break down percentage"}},"additionalProperties":false}},"Level":{"type":"integer","description":"Current stock level"},"RowId":{"type":"string","description":"Record row ID"},"Title":{"type":"string","description":"Item title"},"ItemId":{"type":"string","description":"Stock Item ID"},"Market":{"type":"integer","description":"Market"},"Weight":{"type":"number","description":"Order weight"},"BinRack":{"type":"string","description":"Binrack location"},"ImageId":{"type":["null","string"],"description":"Image ID"},"OnOrder":{"type":"integer","description":"Level due in purchase orders"},"OrderId":{"type":"string","description":"Order ID (pkOrderID)"},"StockId":{"type":"string"},"TaxRate":{"type":"number","description":"Tax rate"},"BinRacks":{"type":"array","items":{"type":"object","properties":{"BatchId":{"type":["null","integer"],"description":"If the item is batched, identifies the batch number"},"BinRack":{"type":"string","description":"BinRack"},"Location":{"type":"string","description":"LocationId of the BinRack"},"Quantity":{"type":"integer","description":"Quantity for BinRack per Location"},"OrderItemBatchId":{"type":["null","integer"],"description":"If the item is batched, identifies the unique order item batch row"}},"additionalProperties":false},"description":"List of BinRacks used for OrderItem"},"Discount":{"type":"number","description":"Percentage (0%, 10%, 20%, etc...)"},"HasImage":{"type":"boolean","description":"If item got an image"},"ItemName":{"type":"string","description":"PartShippedQty"},"Quantity":{"type":"integer","description":"Quantity"},"SalesTax":{"type":"number","description":"Sales Tax"},"UnitCost":{"type":"number","description":"Unit cost"},"IsService":{"type":"boolean","description":"if item is a service"},"CategoryId":{"type":"string"},"ChannelSKU":{"type":"string","description":"Channel product SKU"},"CostIncTax":{"type":"number","description":"Total item cost (inc tax)"},"IsUnlinked":{"type":"boolean"},"ItemNumber":{"type":"string","description":"Item number as on channel"},"ItemSource":{"type":"string","description":"Item source / channel name"},"InOrderBook":{"type":["null","integer"],"description":"Quantity currently in open orders"},"PartShipped":{"type":"boolean","description":"If order is partly shipped"},"StockItemId":{"type":"string","description":"Stock Item ID"},"CategoryName":{"type":"string","description":"Product category"},"ChannelTitle":{"type":"string","description":"Channel product title"},"MinimumLevel":{"type":["null","integer"],"description":"Minimum level"},"ParentItemId":{"type":"string"},"PricePerUnit":{"type":"number","description":"Unit price"},"ShippingCost":{"type":"number","description":"If batch number scan required"},"BarcodeNumber":{"type":"string","description":"Product barcode"},"DiscountValue":{"type":"number"},"AdditionalInfo":{"type":"array","items":{"type":"object","properties":{"Value":{"type":"string","description":"Value of the option"},"Property":{"type":"string","description":"Option property"},"pkOptionId":{"type":"string","description":"Option ID"}},"additionalProperties":false},"description":"List of order item options"},"AvailableStock":{"type":"integer","description":"Currently available stock level (Level-InOrderBook)"},"PartShippedQty":{"type":"integer","description":"ShippingCost"},"StockItemIntId":{"type":"integer"},"OnPurchaseOrder":{"type":"object","properties":{"Rowid":{"type":"string"},"SupplierName":{"type":"string"},"fkLocationId":{"type":"string"},"fkSupplierId":{"type":"string"},"pkPurchaseId":{"type":"string"},"DateOfDelivery":{"type":"string","format":"date-time"},"pkPurchaseItemId":{"type":"string","description":"Primary key of the bound"},"QuotedDeliveryDate":{"type":"string","format":"date-time"},"ExternalInvoiceNumber":{"type":"string"}},"description":"Purchase order bound to this item","additionalProperties":false},"TaxCostInclusive":{"type":"boolean","description":"If tax is included in a cost"},"CompositeSubItems":{"type":"array","items":{"type":"object"},"$comment":"It should be \"$ref\": \"#/properties/Items\" but Airbyte doesn't support recursive $refs."},"IsWarehouseManaged":{"type":"boolean"},"isBatchedStockItem":{"type":"boolean","description":"If item has batches"},"StockLevelIndicator":{"type":"integer","description":"Stock level indicator"},"CompositeAvailablity":{"type":["null","integer"],"description":"Composite availability"},"StockLevelsSpecified":{"type":"boolean","description":"If stock level specified"},"DespatchStockUnitCost":{"type":"number","description":"Despatch stock unit cost"},"InventoryTrackingType":{"type":"integer","description":"Identifies whether the item has a sell by date or other defined order in which inventory is to be sold"},"BatchNumberScanRequired":{"type":"boolean","description":"ItemName"},"SerialNumberScanRequired":{"type":"boolean","description":"If serial number scan required"}},"additionalProperties":false},"description":"List of order items"},"Notes":{"type":"array","items":{"type":"object","properties":{"Note":{"type":"string","description":"Note's text"},"OrderId":{"type":"string","description":"Order Id"},"Internal":{"type":"boolean","description":"order note type (Internal or External)"},"NoteDate":{"type":"string","format":"date-time","description":"Date and time when note was added"},"CreatedBy":{"type":"string","description":"User that created note"},"NoteTypeId":{"type":["null","string"]},"OrderNoteId":{"type":"string","description":"Order note ID"}},"additionalProperties":false},"description":"List of order notes"},"TaxId":{"type":"string","description":"Buyer's tax number."},"OrderId":{"type":"string","description":"Order ID (pkOrderId)"},"Processed":{"type":"boolean","description":"If order is processed"},"FolderName":{"type":"array","items":{"type":"string"},"description":"Folder names assigned to an order"},"NumOrderId":{"type":"integer","description":"Linnworks order number"},"TotalsInfo":{"type":"object","properties":{"Tax":{"type":"number","description":"Tax"},"Currency":{"type":"string","description":"Order currency"},"Subtotal":{"type":"number","description":"Order subtotal"},"pkOrderId":{"type":"string","description":"Order Id"},"PostageCost":{"type":"number","description":"Order postage cost"},"TotalCharge":{"type":"number","description":"Total charge"},"ProfitMargin":{"type":"number","description":"Profit margin"},"PaymentMethod":{"type":"string","description":"Payment method"},"TotalDiscount":{"type":"number","description":"Total discount applied to the order"},"ConversionRate":{"type":"number","description":"Currency conversion rate. Set at point of save by the currency"},"CountryTaxRate":{"type":"number","description":"Country tax rate"},"PaymentMethodId":{"type":"string","description":"Payment method ID"},"PostageCostExTax":{"type":"number","description":"Order postage cost ex. tax"}},"description":"Order totals information","additionalProperties":false},"GeneralInfo":{"type":"object","properties":{"Notes":{"type":"integer","description":"Quantity of order notes"},"Marker":{"type":["null","integer"],"description":"Order marker (0 = NOT TAG, 1 = Tag 1, 2 = Tag 2, 3 = Tag 3, 4 = Tag 4, 5 = Tag 5, 6 = Tag 6, 7 = Parked)"},"Source":{"type":"string","description":"Order ChannelName/Source (e.g. EBAY)"},"Status":{"type":"integer","description":"Order Status (0 = UNPAID, 1 = PAID, 2 = RETURN, 3 = PENDING, 4 = RESEND)"},"IsParked":{"type":"boolean","description":"Is the order parked?"},"Location":{"type":"string","description":"Order location ID"},"NumItems":{"type":"integer","description":"Quantity of order items"},"SiteCode":{"type":"string","description":"SiteCode used to differentiate between different sites from a single channel (eg. Amazon UK, Amazon US, Amazon FR...)"},"IsRuleRun":{"type":"boolean","description":"If rules engine rule ran on an order"},"SubSource":{"type":"string","description":"Order Subsource (e.g. EBAY1)"},"LabelError":{"type":"string","description":"Is there a label error"},"Identifiers":{"type":"array","items":{"type":"object","properties":{"Tag":{"type":"string","description":"Internal tag for identification purposes"},"Name":{"type":"string","description":"Name displayed where the tag is used"},"ImageId":{"type":"string"},"ImageUrl":{"type":"string"},"IsCustom":{"type":"boolean","description":"Is the tag user or system defined?"},"IdentifierId":{"type":"integer","description":"Internal identifier id. Use to update image and name."}},"additionalProperties":false},"description":"Order identifiers. [Prime | Scheduled]"},"PartShipped":{"type":"boolean","description":"If order partly shipped"},"PickwaveIds":{"type":"array","items":{"type":"integer"},"description":"All related Pickwave Ids"},"HoldOrCancel":{"type":"boolean","description":"This shows whether the order has been marked as on hold, for processed orders if the order has been cancelled OnHold = 1"},"LabelPrinted":{"type":"boolean","description":"Is label printed"},"ReceivedDate":{"type":"string","format":"date-time","description":"The date and time at which the order was placed on the sales channel"},"ReferenceNum":{"type":"string","description":"Order reference number (Channel defined)"},"DespatchByDate":{"type":"string","format":"date-time","description":"Despatch by Date"},"InvoicePrinted":{"type":"boolean","description":"Is invoice printed"},"PickListPrinted":{"type":"boolean","description":"Is pick list printed"},"ScheduledDelivery":{"type":"object","properties":{"To":{"type":"string","format":"date-time"},"From":{"type":"string","format":"date-time"}},"description":"Scheduled delivery dates. Take priority over despatch by date","additionalProperties":false},"SecondaryReference":{"type":"string","description":"An additional reference number for the orderr (Used by some channels)"},"StockAllocationType":{"type":["null","string"]},"ExternalReferenceNum":{"type":"string","description":"This is an additional reference number from the sales channel, typically used by eBay"},"HasScheduledDelivery":{"type":"boolean"}},"description":"General information about order","additionalProperties":false},"CustomerInfo":{"type":"object","properties":{"Address":{"$ref":"#/definitions/#/$defs/customer_address"},"BillingAddress":{"$ref":"#/definitions/#/$defs/customer_address"},"ChannelBuyerName":{"type":"string","description":"Username of customer (Comes from channel)"}},"description":"Order Customer information (Name, email etc)","additionalProperties":false},"PaidDateTime":{"type":["null","string"],"format":"date-time","description":"Date and time when the order was marked as paid"},"ShippingInfo":{"type":"object","properties":{"Vendor":{"type":"string","description":"Courier name (e.g. Royal Mail)"},"ItemWeight":{"type":"number","description":"If order is processed"},"PackageType":{"type":"string","description":"Package type name"},"PostageCost":{"type":"number","description":"Order postage cost"},"TotalWeight":{"type":"number","description":"Order total weight"},"ManualAdjust":{"type":"boolean","description":"If there is an adjustment to shipping cost was made"},"PackageTypeId":{"type":["null","string"],"description":"Package type ID"},"TrackingNumber":{"type":"string","description":"Order tracking number provided by courier"},"PackageCategory":{"type":"string","description":"Package category name"},"PostalServiceId":{"type":"string","description":"Postal service ID"},"PostageCostExTax":{"type":"number","description":"Order postage cost excluding tax"},"PackageCategoryId":{"type":"string","description":"Package category ID"},"PostalServiceName":{"type":"string","description":"Postal service name (e.g. Next day delivery)"}},"description":"Order shipping information","additionalProperties":false},"ProcessedDateTime":{"type":["null","string"],"format":"date-time","description":"Date and time when order was processed"},"ExtendedProperties":{"type":"array","items":{"type":"object","properties":{"Name":{"type":"string","description":"Extended property name"},"Type":{"type":"string","description":"Extended property type"},"RowId":{"type":"string","description":"Record row ID"},"Value":{"type":"string","description":"Extended property value"}},"additionalProperties":false},"description":"Extended properties of an order"},"FulfilmentLocationId":{"type":"string","description":"Location ID"}},"definitions":{"#/$defs/customer_address":{"type":"object","properties":{"Town":{"type":"string","description":"Customer's town."},"Region":{"type":"string","description":"Customer's region."},"Company":{"type":"string","description":"Customer's company name."},"Country":{"type":"string","description":"Customer's country."},"Address1":{"type":"string","description":"First line of customer address."},"Address2":{"type":"string","description":"Second line of customer address."},"Address3":{"type":"string","description":"Third line of customer address."},"FullName":{"type":"string","description":"Customer's first and second name."},"PostCode":{"type":"string","description":"Customer's postcode."},"Continent":{"type":"string","description":"Customer's continent"},"CountryId":{"type":"string"},"PhoneNumber":{"type":"string","description":"Customer's telephone number."},"EmailAddress":{"type":"string","description":"Customer's email address."}},"additionalProperties":false}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[NumOrderId]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=append,primaryKey=[[NumOrderId]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@520b2f31[stream=io.airbyte.protocol.models.AirbyteStream@3ab55142[name=processed_orders,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"Town":{"type":"string","description":"Town"},"fTax":{"type":"number","description":"Order tax"},"TaxId":{"type":"string","description":"Order tax id"},"Region":{"type":"string","description":"Region, county, area"},"Source":{"type":"string","description":"ChannelName/Source (e.g. EBAY)"},"Vendor":{"type":"string","description":"Courier name (e.g. DPD)"},"Company":{"type":"string","description":"Company"},"dPaidOn":{"type":"string","format":"date-time","description":"When order was marked as PAID"},"nStatus":{"type":"integer","description":"Order status"},"Address1":{"type":"string","description":"Order first line of address"},"Address2":{"type":"string","description":"Order second line of address"},"Address3":{"type":"string","description":"Order third line of address"},"IsResend":{"type":"boolean","description":"If order was created from a resend"},"Subtotal":{"type":"number","description":"Order subtotal"},"cCountry":{"type":"string","description":"Country"},"nOrderId":{"type":"integer","description":"Linnworks order ID"},"timeDiff":{"type":"number","description":"Days elapsed between order received and order processed"},"SubSource":{"type":"string","description":"Order subsource (e.g. EBAY1)"},"cCurrency":{"type":"string","description":"Order currency"},"cFullName":{"type":"string","description":"Customer full name"},"cPostCode":{"type":"string","description":"Post Code"},"pkOrderID":{"type":"string","description":"Order ID"},"IsExchange":{"type":"boolean","description":"If order was created from an exchange"},"ItemWeight":{"type":"number","description":"Items weight"},"AccountName":{"type":"string","description":"Customer channel account name"},"BillingName":{"type":"string","description":"Customer billing name"},"BillingTown":{"type":"string","description":"Billing town"},"TotalWeight":{"type":"number","description":"Total order weight"},"HoldOrCancel":{"type":"boolean","description":"If order on hold or cancelled."},"PackageTitle":{"type":"string","description":"Package name"},"ProfitMargin":{"type":"number","description":"Profit margin"},"ReferenceNum":{"type":"string","description":"Order reference number"},"dCancelledOn":{"type":"string","format":"date-time","description":"When order was cancelled"},"dProcessedOn":{"type":"string","format":"date-time","description":"Date when order was processed"},"fPostageCost":{"type":"number","description":"Order postage cost"},"fTotalCharge":{"type":"number","description":"Order total charge"},"BillingRegion":{"type":"string","description":"Billing region, area, county"},"TotalDiscount":{"type":"number","description":"Total discount"},"cEmailAddress":{"type":"string","description":"Customer email address"},"dReceivedDate":{"type":"string","format":"date-time","description":"Date when order was received on a channel"},"BillingCompany":{"type":"string","description":"Customer billing company"},"CountryTaxRate":{"type":"number","description":"Country specific tax rate"},"BillingAddress1":{"type":"string","description":"Billing address line one"},"BillingAddress2":{"type":"string","description":"Billing address line two"},"BillingAddress3":{"type":"string","description":"Billing address line three"},"BillingPostCode":{"type":"string","description":"Billing postcode"},"PackageCategory":{"type":"string","description":"Package category"},"cBillingAddress":{"type":"string","description":"Customer billing address"},"BuyerPhoneNumber":{"type":"string","description":"Buyer phone number"},"ChannelBuyerName":{"type":"string","description":"Channel specific name used to identify the buyer, such as a username, normally used for eBay"},"FolderCollection":{"type":"string","description":"Folder name of an order"},"PostageCostExTax":{"type":"number","description":"Postage cost excluding tax"},"cShippingAddress":{"type":"string","description":"Customer's shipping address"},"ExternalReference":{"type":"string","description":"This is an additional reference number from the sales channel, typically used by eBay"},"PostalServiceCode":{"type":"string","description":"Postal service code"},"PostalServiceName":{"type":"string","description":"Postal service name (e.g. Next day delivery)"},"BillingCountryName":{"type":"string","description":"Billing country"},"BillingPhoneNumber":{"type":"string","description":"Billing phone number"},"SecondaryReference":{"type":"string","description":"An additional reference number for the order"},"BillingEmailAddress":{"type":"string"},"PostalTrackingNumber":{"type":"string","description":"Postal tracking number"},"FulfilmentLocationName":{"type":"string","description":"Order fulfilment location"}},"additionalProperties":false},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[dReceivedDate],sourceDefinedPrimaryKey=[[nOrderId]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[dReceivedDate],destinationSyncMode=append,primaryKey=[[nOrderId]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@47a30581[stream=io.airbyte.protocol.models.AirbyteStream@630a1fa2[name=stock_items,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"Depth":{"type":"number","description":"Item depth"},"Width":{"type":"number","description":"Item width"},"Height":{"type":"number","description":"Item height"},"Images":{"type":"array","items":{"type":"object","properties":{"IsMain":{"type":"boolean","description":"Is the image the main image"},"Source":{"type":"string","description":"URL to thumnail image"},"pkRowId":{"type":"string","description":"Unique id of image"},"SortOrder":{"type":"integer","description":"Sort order for the image"},"FullSource":{"type":"string","description":"Url to full size image"},"RawChecksum":{"type":"string","description":"Raw file checksum (Used for UI to determine if the image file is the same before submitting for upload)"},"StockItemId":{"type":"string","description":"Stock Item Id"},"CheckSumValue":{"type":"string","description":"Image check sum"},"ChecksumValue":{"type":"string","description":"Internal checksum value"},"StockItemIntId":{"type":"integer","description":"Stock Item interger Id"}},"additionalProperties":false},"description":"Image urls"},"Weight":{"type":"number","description":"Item weight"},"TaxRate":{"type":"number","description":"Default item tax rate. Set -1 to use country tax rate"},"MetaData":{"type":"string","description":"Item description"},"ItemTitle":{"type":"string","description":"Item title"},"Suppliers":{"type":"array","items":{"type":"object","properties":{"Code":{"type":"string","description":"Supplier code"},"LeadTime":{"type":"integer","description":"Supplier lead time"},"MaxPrice":{"type":"number","description":"Maximum price"},"MinPrice":{"type":"number","description":"Minimum price"},"Supplier":{"type":"string","description":"Supplier name"},"IsDefault":{"type":"boolean","description":"If supplier information is default"},"SupplierID":{"type":"string","description":"Supplier ID"},"StockItemId":{"type":"string","description":"Stock Item Id"},"AveragePrice":{"type":"number","description":"Average price"},"PurchasePrice":{"type":"number","description":"Supplier purchase price"},"StockItemIntId":{"type":"integer","description":"Stock Item integer Id"},"AverageLeadTime":{"type":"number","description":"Average lead time"},"SupplierBarcode":{"type":"string","description":"Supplier barcode"},"SupplierCurrency":{"type":"string","description":"Supplier default currency"},"SupplierPackSize":{"type":"integer","description":"Supplier pack size"},"SupplierMinOrderQty":{"type":"integer","description":"Minimum order quantity from this supplier"}},"additionalProperties":false},"description":"Suppliers"},"CategoryId":{"type":"string","description":"Default category id"},"ItemNumber":{"type":"string","description":"SKU"},"RetailPrice":{"type":["null","number"],"description":"Default item retail price"},"StockItemId":{"type":"string","description":"Stock Item Id"},"StockLevels":{"type":"array","items":{"type":"object","properties":{"Due":{"type":"integer","description":"Due to come in purchase orders"},"JIT":{"type":"boolean","description":"Stock Item Just In Time (JIT) status"},"SKU":{"type":"string","description":"Product SKU"},"rowid":{"type":"string","description":"dbo.StockLevel.rowid"},"InOrders":{"type":"integer","description":"Currently in open orders"},"Location":{"type":"object","properties":{"BinRack":{"type":"string","description":"Bin rack"},"LocationTag":{"type":"string","description":"Location tag"},"LocationName":{"type":"string","description":"Location name"},"StockLocationId":{"type":"string","description":"Location ID"},"IsWarehouseManaged":{"type":["null","boolean"],"description":"If the location is warehouse managed"},"StockLocationIntId":{"type":"integer","description":"Stock location integer id"},"IsFulfillmentCenter":{"type":"boolean","description":"If location is a fulfillment center"}},"description":"Location ID","additionalProperties":false},"UnitCost":{"type":"number","description":"if( Quantity == 0 ) dbo.StockItem.PurchasePrice Else CurrentStockValue / Quantity"},"Available":{"type":"integer","description":"StockLevel - InOrders"},"AutoAdjust":{"type":"boolean","description":"If level is auto adjusted"},"StockLevel":{"type":"integer","description":"Stock level"},"StockValue":{"type":"number","description":"Stock value"},"InOrderBook":{"type":"integer","description":"Currently in open orders"},"StockItemId":{"type":"string","description":"Stock Item Id"},"MinimumLevel":{"type":"integer","description":"Minimum level"},"PendingUpdate":{"type":"boolean","description":"dbo.StockLevel.PendingUpdate"},"LastUpdateDate":{"type":"string","format":"date-time","description":"Last time stock level was adjusted"},"StockItemIntId":{"type":"integer","description":"Stock Item interger Id"},"LastUpdateOperation":{"type":"string","description":"Name of last update operation"},"StockItemPurchasePrice":{"type":"number","description":"Stock item purchase price. It's used to calculate UnitCost"}},"additionalProperties":false},"description":"Stock Levels"},"CategoryName":{"type":"string","description":"Default category name"},"CreationDate":{"type":["null","string"],"format":"date-time","description":"Stock item creation date"},"BarcodeNumber":{"type":"string","description":"Barcode number"},"PurchasePrice":{"type":"number","description":"Default item purchase price"},"PackageGroupId":{"type":"string","description":"Default package group id"},"StockItemIntId":{"type":"integer","description":"Stock Item interger Id"},"PostalServiceId":{"type":"string","description":"Default postal service id"},"PackageGroupName":{"type":"string","description":"Default package group name"},"ItemChannelPrices":{"type":"array","items":{"type":"object","properties":{"Tag":{"type":"string","description":"Product price tag"},"Price":{"type":"number","description":"Channel price"},"Rules":{"type":"array","items":{"type":"object","properties":{"Type":{"type":"string","description":"Type"},"Value":{"type":"number","description":"Value/Price level"},"pkRowId":{"type":["null","integer"],"description":"Record row ID (optional)"},"LowerBound":{"type":"integer","description":"Lower level"},"fkStockPricingId":{"type":"string","description":"Stock pricing ID"}},"additionalProperties":false},"description":"Pricing rule"},"Source":{"type":"string","description":"ChannelName/Source (e.g. EBAY)"},"pkRowId":{"type":"string","description":"Record row ID (generate random)"},"SubSource":{"type":"string","description":"SubSource (e.g. EBAY1)"},"StockItemId":{"type":"string","description":"Stock Item Id"},"UpdateStatus":{"type":"string"},"StockItemIntId":{"type":"integer","description":"Stock Item interger Id"}},"additionalProperties":false},"description":"List of item prices"},"ItemChannelTitles":{"type":"array","items":{"type":"object","properties":{"Title":{"type":"string","description":"Item title"},"Source":{"type":"string","description":"ChannelName/Source (e.g. EBAY)"},"pkRowId":{"type":"string","description":"Record row id (generate random)"},"SubSource":{"type":"string","description":"SubSource (EBAY1)"},"StockItemId":{"type":"string","description":"Stock Item Id"},"StockItemIntId":{"type":"integer","description":"Stock Item integer Id"}},"additionalProperties":false},"description":"List item titles"},"PostalServiceName":{"type":"string","description":"Default postal service name"},"isBatchedStockType":{"type":"boolean","description":"Returns true is the stock item is tracked by batch"},"InventoryTrackingType":{"type":"integer","description":"Stock item tracking type. 0 = none. 1 = Ordered by Sell by Date. 2 = Ordered by Priority Sequence"},"ItemExtendedProperties":{"type":"array","items":{"type":"object","properties":{"pkRowId":{"type":"string","description":"Record row ID (generate random)"},"ProperyName":{"type":"string","description":"Property name"},"PropertyType":{"type":"string","description":"Property type"},"PropertyValue":{"type":"string","description":"Property value"},"fkStockItemId":{"type":"string","description":"Stock Item ID"}},"additionalProperties":false},"description":"List of extended properties"},"BatchNumberScanRequired":{"type":"boolean","description":"User must scan batch number when procesing orders"},"ItemChannelDescriptions":{"type":"array","items":{"type":"object","properties":{"Source":{"type":"string","description":"ChannelName/Source (e.g. EBAY)"},"pkRowId":{"type":"string","description":"Record row ID (generate random GUID)"},"SubSource":{"type":"string","description":"Channel subsource (e.g EBAY1)"},"Description":{"type":"string","description":"Product description"},"StockItemId":{"type":"string","description":"Stock Item Id"},"StockItemIntId":{"type":"integer","description":"Stock Item interger Id"}},"additionalProperties":false},"description":"List of item descriptions"},"SerialNumberScanRequired":{"type":"boolean","description":"User must scan item serial number when processing ordesr"}},"additionalProperties":false},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[StockItemIntId]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=append,primaryKey=[[StockItemIntId]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@68f3c655[stream=io.airbyte.protocol.models.AirbyteStream@785ca548[name=stock_locations,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"BinRack":{"type":"string","description":"Bin rack"},"location":{"type":"object","properties":{"City":{"type":"string","description":"City"},"County":{"type":"string","description":"County / Region"},"Country":{"type":"string","description":"Country"},"ZipCode":{"type":"string","description":"Postal code"},"Address1":{"type":"string","description":"1st line of address"},"Address2":{"type":"string","description":"2nd line of address"},"LocationTag":{"type":"string","description":"Location tag"},"LocationName":{"type":"string","description":"Location name"},"IsNotTrackable":{"type":"boolean","description":"Is the location trackable"},"StockLocationId":{"type":"string","description":"Location ID"},"IsWarehouseManaged":{"type":"boolean","description":"Indicates if the location is warehouse managed"},"StockLocationIntId":{"type":"integer","description":"Stock location integer id."},"IsFulfillmentCenter":{"type":"boolean","description":"If location is a fulfillment center"},"CountInOrderUntilAcknowledgement":{"type":"boolean","description":"Count in order"},"FulfilmentCenterDeductStockWhenProcessed":{"type":"boolean","description":"Fulfilment center and stock will be deducted when order processed"}},"additionalProperties":false},"LocationTag":{"type":"string","description":"Location tag"},"LocationName":{"type":"string","description":"Location name"},"StockLocationId":{"type":"string","description":"Location ID"},"IsWarehouseManaged":{"type":["null","boolean"],"description":"If the location is warehouse managed."},"StockLocationIntId":{"type":"integer","description":"Stock location integer id"},"IsFulfillmentCenter":{"type":"boolean","description":"If location is a fulfillment center"}},"additionalProperties":false},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[StockLocationIntId]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=append,primaryKey=[[StockLocationIntId]],additionalProperties={}]],additionalProperties={}]] 2021-11-22 13:50:28 INFO () TemporalAttemptExecution(get):116 - Executing worker wrapper. Airbyte version: 0.32.0-alpha 2021-11-22 13:50:28 WARN () Databases(createPostgresDatabaseWithRetry):41 - Waiting for database to become available... 2021-11-22 13:50:28 INFO () JobsDatabaseInstance(lambda$static$2):25 - Testing if jobs database is ready... 2021-11-22 13:50:28 INFO () Databases(createPostgresDatabaseWithRetry):58 - Database available! 2021-11-22 13:50:28 INFO () DefaultNormalizationWorker(run):46 - Running normalization. 2021-11-22 13:50:28 INFO () DefaultNormalizationRunner(runProcess):114 - Running with normalization version: airbyte/normalization:0.1.59 2021-11-22 13:50:28 INFO () LineGobbler(voidCall):82 - Checking if airbyte/normalization:0.1.59 exists... 2021-11-22 13:50:28 INFO () LineGobbler(voidCall):82 - airbyte/normalization:0.1.59 not found locally. Attempting to pull the image... 2021-11-22 13:51:25 INFO () LineGobbler(voidCall):82 - Pulled airbyte/normalization:0.1.59 from remote. 2021-11-22 13:51:25 INFO () DockerProcessFactory(create):127 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/232/0/normalize --network host --log-driver none airbyte/normalization:0.1.59 run --integration-type postgres --config destination_config.json --catalog destination_catalog.json normalization - 2021-11-22 13:51:38 INFO () LineGobbler(voidCall):82 - Running: transform-config --config destination_config.json --integration-type postgres --out /data/232/0/normalize normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Namespace(config='destination_config.json', integration_type=, out='/data/232/0/normalize') normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - transform_postgres normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Running: transform-catalog --integration-type postgres --profile-config-dir /data/232/0/normalize --catalog destination_catalog.json --out /data/232/0/normalize/models/generated/ --json-column _airbyte_data normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Processing destination_catalog.json... normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Truncating processed_order_details_Items_AdditionalInfo (#44) to processed_order_deta__Items_AdditionalInfo (#43) normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Truncating processed_order_details_Items_OnPurchaseOrder (#45) to processed_order_deta_Items_OnPurchaseOrder (#43) normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Truncating processed_order_details_GeneralInfo_Identifiers (#47) to processed_order_deta_neralInfo_Identifiers (#43) normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Truncating processed_order_details_GeneralInfo_PickwaveIds (#47) to processed_order_deta_neralInfo_PickwaveIds (#43) normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Truncating processed_order_details_GeneralInfo_ScheduledDelivery (#53) to processed_order_deta_nfo_ScheduledDelivery (#43) normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_ab1.sql from processed_order_details normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_ab2.sql from processed_order_details normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_ab3.sql from processed_order_details normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/processed_order_details.sql from processed_order_details normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_orders_ab1.sql from processed_orders normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_orders_ab2.sql from processed_orders normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_orders_ab3.sql from processed_orders normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/processed_orders.sql from processed_orders normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_ab1.sql from stock_items normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_ab2.sql from stock_items normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_ab3.sql from stock_items normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/stock_items.sql from stock_items normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_locations_ab1.sql from stock_locations normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_locations_ab2.sql from stock_locations normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_locations_ab3.sql from stock_locations normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/stock_locations.sql from stock_locations normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_items_ab1.sql from processed_order_details/Items normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_items_ab2.sql from processed_order_details/Items normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_items_ab3.sql from processed_order_details/Items normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/processed_order_details_items.sql from processed_order_details/Items normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_notes_ab1.sql from processed_order_details/Notes normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_notes_ab2.sql from processed_order_details/Notes normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_notes_ab3.sql from processed_order_details/Notes normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/processed_order_details_notes.sql from processed_order_details/Notes normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Ignoring stream 'FolderName' from processed_order_details/FolderName because properties list is empty normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_totalsinfo_ab1.sql from processed_order_details/TotalsInfo normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_totalsinfo_ab2.sql from processed_order_details/TotalsInfo normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_totalsinfo_ab3.sql from processed_order_details/TotalsInfo normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/processed_order_details_totalsinfo.sql from processed_order_details/TotalsInfo normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_generalinfo_ab1.sql from processed_order_details/GeneralInfo normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_generalinfo_ab2.sql from processed_order_details/GeneralInfo normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_generalinfo_ab3.sql from processed_order_details/GeneralInfo normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/processed_order_details_generalinfo.sql from processed_order_details/GeneralInfo normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_customerinfo_ab1.sql from processed_order_details/CustomerInfo normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - WARN: Unknown type for column Address at processed_order_details/CustomerInfo normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - WARN: Unknown type for column BillingAddress at processed_order_details/CustomerInfo normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_customerinfo_ab2.sql from processed_order_details/CustomerInfo normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_customerinfo_ab3.sql from processed_order_details/CustomerInfo normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/processed_order_details_customerinfo.sql from processed_order_details/CustomerInfo normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_shippinginfo_ab1.sql from processed_order_details/ShippingInfo normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_shippinginfo_ab2.sql from processed_order_details/ShippingInfo normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_shippinginfo_ab3.sql from processed_order_details/ShippingInfo normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/processed_order_details_shippinginfo.sql from processed_order_details/ShippingInfo normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_extendedproperties_ab1.sql from processed_order_details/ExtendedProperties normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_extendedproperties_ab2.sql from processed_order_details/ExtendedProperties normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_extendedproperties_ab3.sql from processed_order_details/ExtendedProperties normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/processed_order_details_extendedproperties.sql from processed_order_details/ExtendedProperties normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_images_ab1.sql from stock_items/Images normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_images_ab2.sql from stock_items/Images normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_images_ab3.sql from stock_items/Images normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/stock_items_images.sql from stock_items/Images normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_suppliers_ab1.sql from stock_items/Suppliers normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_suppliers_ab2.sql from stock_items/Suppliers normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_suppliers_ab3.sql from stock_items/Suppliers normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/stock_items_suppliers.sql from stock_items/Suppliers normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_stocklevels_ab1.sql from stock_items/StockLevels normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_stocklevels_ab2.sql from stock_items/StockLevels normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_stocklevels_ab3.sql from stock_items/StockLevels normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/stock_items_stocklevels.sql from stock_items/StockLevels normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_itemchannelprices_ab1.sql from stock_items/ItemChannelPrices normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_itemchannelprices_ab2.sql from stock_items/ItemChannelPrices normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_itemchannelprices_ab3.sql from stock_items/ItemChannelPrices normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/stock_items_itemchannelprices.sql from stock_items/ItemChannelPrices normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_itemchanneltitles_ab1.sql from stock_items/ItemChannelTitles normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_itemchanneltitles_ab2.sql from stock_items/ItemChannelTitles normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_itemchanneltitles_ab3.sql from stock_items/ItemChannelTitles normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/stock_items_itemchanneltitles.sql from stock_items/ItemChannelTitles normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_itemextendedproperties_ab1.sql from stock_items/ItemExtendedProperties normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_itemextendedproperties_ab2.sql from stock_items/ItemExtendedProperties normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_itemextendedproperties_ab3.sql from stock_items/ItemExtendedProperties normalization - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/stock_items_itemextendedproperties.sql from stock_items/ItemExtendedProperties normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_itemchanneldescriptions_ab1.sql from stock_items/ItemChannelDescriptions normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_itemchanneldescriptions_ab2.sql from stock_items/ItemChannelDescriptions normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_itemchanneldescriptions_ab3.sql from stock_items/ItemChannelDescriptions normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/stock_items_itemchanneldescriptions.sql from stock_items/ItemChannelDescriptions normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_locations_location_ab1.sql from stock_locations/location normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_locations_location_ab2.sql from stock_locations/location normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_locations_location_ab3.sql from stock_locations/location normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/stock_locations_location.sql from stock_locations/location normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_items_boxes_ab1.sql from processed_order_details/Items/Boxes normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_items_boxes_ab2.sql from processed_order_details/Items/Boxes normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_items_boxes_ab3.sql from processed_order_details/Items/Boxes normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/processed_order_details_items_boxes.sql from processed_order_details/Items/Boxes normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_items_binracks_ab1.sql from processed_order_details/Items/BinRacks normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_items_binracks_ab2.sql from processed_order_details/Items/BinRacks normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_details_items_binracks_ab3.sql from processed_order_details/Items/BinRacks normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/processed_order_details_items_binracks.sql from processed_order_details/Items/BinRacks normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_deta___items_additionalinfo_ab1.sql from processed_order_details/Items/AdditionalInfo normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_deta___items_additionalinfo_ab2.sql from processed_order_details/Items/AdditionalInfo normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_deta___items_additionalinfo_ab3.sql from processed_order_details/Items/AdditionalInfo normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/processed_order_deta___items_additionalinfo.sql from processed_order_details/Items/AdditionalInfo normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_deta__items_onpurchaseorder_ab1.sql from processed_order_details/Items/OnPurchaseOrder normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_deta__items_onpurchaseorder_ab2.sql from processed_order_details/Items/OnPurchaseOrder normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_deta__items_onpurchaseorder_ab3.sql from processed_order_details/Items/OnPurchaseOrder normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/processed_order_deta__items_onpurchaseorder.sql from processed_order_details/Items/OnPurchaseOrder normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_deta__neralinfo_identifiers_ab1.sql from processed_order_details/GeneralInfo/Identifiers normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_deta__neralinfo_identifiers_ab2.sql from processed_order_details/GeneralInfo/Identifiers normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_deta__neralinfo_identifiers_ab3.sql from processed_order_details/GeneralInfo/Identifiers normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/processed_order_deta__neralinfo_identifiers.sql from processed_order_details/GeneralInfo/Identifiers normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Ignoring stream 'PickwaveIds' from processed_order_details/GeneralInfo/PickwaveIds because properties list is empty normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_deta__nfo_scheduleddelivery_ab1.sql from processed_order_details/GeneralInfo/ScheduledDelivery normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_deta__nfo_scheduleddelivery_ab2.sql from processed_order_details/GeneralInfo/ScheduledDelivery normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/processed_order_deta__nfo_scheduleddelivery_ab3.sql from processed_order_details/GeneralInfo/ScheduledDelivery normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/processed_order_deta__nfo_scheduleddelivery.sql from processed_order_details/GeneralInfo/ScheduledDelivery normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_stocklevels_location_ab1.sql from stock_items/StockLevels/Location normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_stocklevels_location_ab2.sql from stock_items/StockLevels/Location normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_stocklevels_location_ab3.sql from stock_items/StockLevels/Location normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/stock_items_stocklevels_location.sql from stock_items/StockLevels/Location normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_itemchannelprices_rules_ab1.sql from stock_items/ItemChannelPrices/Rules normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_itemchannelprices_rules_ab2.sql from stock_items/ItemChannelPrices/Rules normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_ctes/linnworks/stock_items_itemchannelprices_rules_ab3.sql from stock_items/ItemChannelPrices/Rules normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - Generating airbyte_incremental/linnworks/stock_items_itemchannelprices_rules.sql from stock_items/ItemChannelPrices/Rules normalization - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - detected no config file for ssh, assuming ssh is off. normalization - 2021-11-22 13:51:43 INFO () LineGobbler(voidCall):82 - Running with dbt=0.21.0 normalization - 2021-11-22 13:51:44 INFO () LineGobbler(voidCall):82 - Unable to do partial parsing because ../build/partial_parse.msgpack not found normalization - 2021-11-22 13:51:48 INFO () LineGobbler(voidCall):82 - [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources. normalization - 2021-11-22 13:51:48 INFO () LineGobbler(voidCall):82 - There are 2 unused configuration paths: normalization - 2021-11-22 13:51:48 INFO () LineGobbler(voidCall):82 - - models.airbyte_utils.generated.airbyte_tables normalization - 2021-11-22 13:51:48 INFO () LineGobbler(voidCall):82 - - models.airbyte_utils.generated.airbyte_views normalization - 2021-11-22 13:51:48 INFO () LineGobbler(voidCall):82 - normalization - 2021-11-22 13:51:48 INFO () LineGobbler(voidCall):82 - Found 108 models, 0 tests, 0 snapshots, 0 analyses, 478 macros, 0 operations, 0 seed files, 4 sources, 0 exposures normalization - 2021-11-22 13:51:48 INFO () LineGobbler(voidCall):82 - normalization - 2021-11-22 13:51:49 INFO () LineGobbler(voidCall):82 - 13:51:49 | Concurrency: 32 threads (target='prod') normalization - 2021-11-22 13:51:49 INFO () LineGobbler(voidCall):82 - 13:51:49 | normalization - 2021-11-22 13:51:49 INFO () LineGobbler(voidCall):82 - 13:51:49 | 1 of 27 START incremental model linnworks.stock_locations.................................................... [RUN] normalization - 2021-11-22 13:51:50 INFO () LineGobbler(voidCall):82 - 13:51:50 | 2 of 27 START incremental model linnworks.processed_order_details............................................ [RUN] normalization - 2021-11-22 13:51:50 INFO () LineGobbler(voidCall):82 - 13:51:50 | 3 of 27 START incremental model linnworks.stock_items........................................................ [RUN] normalization - 2021-11-22 13:51:50 INFO () LineGobbler(voidCall):82 - 13:51:50 | 4 of 27 START incremental model linnworks.processed_orders................................................... [RUN] normalization - 2021-11-22 13:51:51 INFO () LineGobbler(voidCall):82 - 13:51:51 | 1 of 27 OK created incremental model linnworks.stock_locations............................................... [INSERT 0 0 in 1.31s] normalization - 2021-11-22 13:51:51 INFO () LineGobbler(voidCall):82 - 13:51:51 | 2 of 27 OK created incremental model linnworks.processed_order_details....................................... [INSERT 0 0 in 0.90s] normalization - 2021-11-22 13:51:51 INFO () LineGobbler(voidCall):82 - 13:51:51 | 3 of 27 OK created incremental model linnworks.stock_items................................................... [INSERT 0 0 in 1.16s] normalization - 2021-11-22 13:51:52 INFO () LineGobbler(voidCall):82 - 13:51:52 | 4 of 27 OK created incremental model linnworks.processed_orders.............................................. [INSERT 0 0 in 1.32s] normalization - 2021-11-22 13:51:53 INFO () LineGobbler(voidCall):82 - 13:51:53 | 5 of 27 START incremental model linnworks.stock_locations_location........................................... [RUN] normalization - 2021-11-22 13:51:53 INFO () LineGobbler(voidCall):82 - 13:51:53 | 6 of 27 START incremental model linnworks.processed_order_details_extendedproperties......................... [RUN] normalization - 2021-11-22 13:51:54 INFO () LineGobbler(voidCall):82 - 13:51:54 | 7 of 27 START incremental model linnworks.processed_order_details_notes...................................... [RUN] normalization - 2021-11-22 13:51:54 INFO () LineGobbler(voidCall):82 - 13:51:54 | 8 of 27 START incremental model linnworks.processed_order_details_customerinfo............................... [RUN] normalization - 2021-11-22 13:51:54 INFO () LineGobbler(voidCall):82 - 13:51:54 | 9 of 27 START incremental model linnworks.processed_order_details_shippinginfo............................... [RUN] normalization - 2021-11-22 13:51:54 INFO () LineGobbler(voidCall):82 - 13:51:54 | 10 of 27 START incremental model linnworks.processed_order_details_generalinfo............................... [RUN] normalization - 2021-11-22 13:51:54 INFO () LineGobbler(voidCall):82 - 13:51:54 | 11 of 27 START incremental model linnworks.stock_items_itemextendedproperties................................ [RUN] normalization - 2021-11-22 13:51:54 INFO () LineGobbler(voidCall):82 - 13:51:54 | 12 of 27 START incremental model linnworks.processed_order_details_totalsinfo................................ [RUN] normalization - 2021-11-22 13:51:54 INFO () LineGobbler(voidCall):82 - 13:51:54 | 14 of 27 START incremental model linnworks.stock_items_itemchanneldescriptions............................... [RUN] normalization - 2021-11-22 13:51:54 INFO () LineGobbler(voidCall):82 - 13:51:54 | 13 of 27 START incremental model linnworks.stock_items_itemchanneltitles..................................... [RUN] normalization - 2021-11-22 13:51:55 INFO () LineGobbler(voidCall):82 - 13:51:54 | 15 of 27 START incremental model linnworks.stock_items_images................................................ [RUN] normalization - 2021-11-22 13:51:55 INFO () LineGobbler(voidCall):82 - 13:51:54 | 16 of 27 START incremental model linnworks.processed_order_details_items..................................... [RUN] normalization - 2021-11-22 13:51:55 INFO () LineGobbler(voidCall):82 - 13:51:54 | 17 of 27 START incremental model linnworks.stock_items_suppliers............................................. [RUN] normalization - 2021-11-22 13:51:55 INFO () LineGobbler(voidCall):82 - 13:51:55 | 18 of 27 START incremental model linnworks.stock_items_itemchannelprices..................................... [RUN] normalization - 2021-11-22 13:51:55 INFO () LineGobbler(voidCall):82 - 13:51:55 | 19 of 27 START incremental model linnworks.stock_items_stocklevels........................................... [RUN] normalization - 2021-11-22 13:51:56 INFO () LineGobbler(voidCall):82 - 13:51:56 | 5 of 27 OK created incremental model linnworks.stock_locations_location...................................... [INSERT 0 0 in 3.11s] normalization - 2021-11-22 13:51:56 INFO () LineGobbler(voidCall):82 - 13:51:56 | 6 of 27 OK created incremental model linnworks.processed_order_details_extendedproperties.................... [INSERT 0 0 in 2.72s] normalization - 2021-11-22 13:51:56 INFO () LineGobbler(voidCall):82 - 13:51:56 | 8 of 27 OK created incremental model linnworks.processed_order_details_customerinfo.......................... [INSERT 0 0 in 2.58s] normalization - 2021-11-22 13:51:56 INFO () LineGobbler(voidCall):82 - 13:51:56 | 10 of 27 OK created incremental model linnworks.processed_order_details_generalinfo.......................... [INSERT 0 0 in 2.44s] normalization - 2021-11-22 13:51:56 INFO () LineGobbler(voidCall):82 - 13:51:56 | 11 of 27 OK created incremental model linnworks.stock_items_itemextendedproperties........................... [INSERT 0 0 in 2.14s] normalization - 2021-11-22 13:51:56 INFO () LineGobbler(voidCall):82 - 13:51:56 | 7 of 27 OK created incremental model linnworks.processed_order_details_notes................................. [INSERT 0 0 in 2.62s] normalization - 2021-11-22 13:51:56 INFO () LineGobbler(voidCall):82 - 13:51:56 | 9 of 27 OK created incremental model linnworks.processed_order_details_shippinginfo.......................... [INSERT 0 0 in 2.52s] normalization - 2021-11-22 13:51:56 INFO () LineGobbler(voidCall):82 - 13:51:56 | 12 of 27 OK created incremental model linnworks.processed_order_details_totalsinfo........................... [INSERT 0 0 in 2.19s] normalization - 2021-11-22 13:51:56 INFO () LineGobbler(voidCall):82 - 13:51:56 | 14 of 27 OK created incremental model linnworks.stock_items_itemchanneldescriptions.......................... [INSERT 0 0 in 2.11s] normalization - 2021-11-22 13:51:57 INFO () LineGobbler(voidCall):82 - 13:51:57 | 16 of 27 OK created incremental model linnworks.processed_order_details_items................................ [INSERT 0 0 in 1.93s] normalization - 2021-11-22 13:51:57 INFO () LineGobbler(voidCall):82 - 13:51:57 | 13 of 27 OK created incremental model linnworks.stock_items_itemchanneltitles................................ [INSERT 0 0 in 2.13s] normalization - 2021-11-22 13:51:57 INFO () LineGobbler(voidCall):82 - 13:51:57 | 15 of 27 OK created incremental model linnworks.stock_items_images........................................... [INSERT 0 0 in 1.98s] normalization - 2021-11-22 13:51:57 INFO () LineGobbler(voidCall):82 - 13:51:57 | 17 of 27 OK created incremental model linnworks.stock_items_suppliers........................................ [INSERT 0 0 in 2.06s] normalization - 2021-11-22 13:51:57 INFO () LineGobbler(voidCall):82 - 13:51:57 | 18 of 27 OK created incremental model linnworks.stock_items_itemchannelprices................................ [INSERT 0 0 in 1.93s] normalization - 2021-11-22 13:51:57 INFO () LineGobbler(voidCall):82 - 13:51:57 | 19 of 27 OK created incremental model linnworks.stock_items_stocklevels...................................... [INSERT 0 0 in 1.84s] normalization - 2021-11-22 13:51:57 INFO () LineGobbler(voidCall):82 - 13:51:57 | 20 of 27 START incremental model linnworks.processed_order_deta__neralinfo_identifiers....................... [RUN] normalization - 2021-11-22 13:51:57 INFO () LineGobbler(voidCall):82 - 13:51:57 | 21 of 27 START incremental model linnworks.processed_order_deta__nfo_scheduleddelivery....................... [RUN] normalization - 2021-11-22 13:51:58 INFO () LineGobbler(voidCall):82 - 13:51:58 | 22 of 27 START incremental model linnworks.processed_order_details_items_binracks............................ [RUN] normalization - 2021-11-22 13:51:58 INFO () LineGobbler(voidCall):82 - 13:51:58 | 23 of 27 START incremental model linnworks.processed_order_details_items_boxes............................... [RUN] normalization - 2021-11-22 13:51:58 INFO () LineGobbler(voidCall):82 - 13:51:58 | 25 of 27 START incremental model linnworks.processed_order_deta__items_onpurchaseorder....................... [RUN] normalization - 2021-11-22 13:51:58 INFO () LineGobbler(voidCall):82 - 13:51:58 | 24 of 27 START incremental model linnworks.processed_order_deta___items_additionalinfo....................... [RUN] normalization - 2021-11-22 13:51:58 INFO () LineGobbler(voidCall):82 - 13:51:58 | 26 of 27 START incremental model linnworks.stock_items_itemchannelprices_rules............................... [RUN] normalization - 2021-11-22 13:51:58 INFO () LineGobbler(voidCall):82 - 13:51:58 | 27 of 27 START incremental model linnworks.stock_items_stocklevels_location.................................. [RUN] normalization - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - 13:51:59 | 20 of 27 OK created incremental model linnworks.processed_order_deta__neralinfo_identifiers.................. [INSERT 0 0 in 1.26s] normalization - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - 13:51:59 | 21 of 27 OK created incremental model linnworks.processed_order_deta__nfo_scheduleddelivery.................. [INSERT 0 0 in 1.22s] normalization - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - 13:51:59 | 25 of 27 OK created incremental model linnworks.processed_order_deta__items_onpurchaseorder.................. [INSERT 0 0 in 0.86s] normalization - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - 13:51:59 | 22 of 27 OK created incremental model linnworks.processed_order_details_items_binracks....................... [INSERT 0 0 in 1.23s] normalization - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - 13:51:59 | 27 of 27 OK created incremental model linnworks.stock_items_stocklevels_location............................. [INSERT 0 0 in 0.68s] normalization - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - 13:51:59 | 24 of 27 OK created incremental model linnworks.processed_order_deta___items_additionalinfo.................. [INSERT 0 0 in 0.90s] normalization - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - 13:51:59 | 23 of 27 OK created incremental model linnworks.processed_order_details_items_boxes.......................... [INSERT 0 0 in 0.93s] normalization - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - 13:51:59 | 26 of 27 OK created incremental model linnworks.stock_items_itemchannelprices_rules.......................... [INSERT 0 0 in 0.81s] normalization - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - 13:51:59 | normalization - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - 13:51:59 | Finished running 27 incremental models in 10.45s. normalization - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - normalization - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - Completed successfully normalization - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - normalization - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - Done. PASS=27 WARN=0 ERROR=0 SKIP=0 TOTAL=27 2021-11-22 13:51:59 INFO () DefaultNormalizationWorker(run):69 - Normalization executed in 1 minute 31 seconds. 2021-11-22 13:51:59 INFO () TemporalAttemptExecution(get):137 - Stopping cancellation check scheduling... ```

Steps to Reproduce

  1. Setup any source with schema using $ref/$defs, e.g. https://github.com/airbytehq/airbyte/pull/8177/commits/5303b0b848732f3c79643e465bee8d3c79808b0b
  2. Setup destination Postgres
  3. Run sync

Are you willing to submit a PR?

Maybe.

harshithmullapudi commented 2 years ago

@alafanechere do you think this https://github.com/airbytehq/airbyte/issues/8330 is also same?

ChristopheDuong commented 2 years ago

we don't support $ref / $defs in airbyte json schema, so normalization does not support that either

The source should resolve and inline the full json into the catalog

monai commented 2 years ago

we don't support $ref / $defs in airbyte json schema, so normalization does not support that either

The source should resolve and inline the full json into the catalog

Well, actually, Airbyte does support $ref's to other files. Referenced schemas are inlined under and $refs point to #/definitions/{filename}.

See the documentation: https://docs.airbyte.io/connector-development/cdk-python/schemas#static-schemas

Important note: any objects referenced via $ref should be placed in the shared/ directory in their own .json files.

ChristopheDuong commented 2 years ago

Sorry for the misunderstanding!

That's the documentation page for the CDK and how to ease implementation of source connectors in airbyte.

But once the source connector produces the catalog.json in Airbyte's protocol, there should be no more $ref afterward as the CDK/source connector should have inlined all the $ref into a full JSON schema object.

The separated YAML/JSON files (used in the refs) are not transmitted through the protocol (only a full json object called catalog), so normalization would not be able to resolve and inline the different JSON object back together. (that's why I said that it is "not supported")

see this issue https://github.com/airbytehq/airbyte/issues/7966 and its associated PRs where we are adding "Source Acceptance Tests" (SAT) to make sure all $ref are resolved by the source connector before "materializing" the catalog.json (that is used downstream by normalization)