search

kromerm / adflab

Azure Data Factory hands-on lab, self-paced. Learn how to lift & shift SSIS packages to the Cloud with ADF. Build new ETL pipelines in ADF, transform data at scale, load Azure Data Warehouse data marts. Also walks through operationalizing ADF pipelines with scheduling and monitoring modules.
http://aka.ms/adflab2
137 stars 129 forks source link

Lab 3 - FAAMaster Path step 21 #7

Closed patpicos closed 6 years ago

patpicos commented 6 years ago

The path should read input/FAAMaster (to allow pulling the schema)

patpicos commented 6 years ago

LAB03 - Rebuilding the Extract and Load with ADF.docx

patpicos commented 6 years ago

if that is not the case, when i try to click "Get Schema" the schema from the source does not carry over. (step 20-23)

error: Loading Error "Error when processing request: The required Blob is missing. ContainerName: https://adflabstorage7nuut.blob.core.windows.net/input, ContainerExist: True, BlobPrefix: FAAmaster.txt, BlobCount: 0. activityId: 731581cb-60a1-4c8e-bd20-7854d1f0061b"

patpicos commented 6 years ago

Doing a test run, the following error: { "errorCode": "2200", "message": "ErrorCode=UserErrorSourceDataContainsMoreColumnsThanDefined,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error found when processing 'Csv/Tsv Format Text' source 'MASTER.txt' with row number 401059: found more columns than expected column count: 34.,Source=Microsoft.DataTransfer.Common,'", "failureType": "UserError", "target": "S3 to Azure blob" }

joshuha commented 6 years ago

Fixed the instructions to mention drilling down into the folder. Technically when copying to a file you don't need a schema on the sink side but we put that in there to make it more consistent. You may not always have a file on the sink in a lot of cases.

Looks like a step was messed for the quote character on the source side. Some rows do have commas in the text and those use " as a quote character. Added that in the documentation as well.