Unable to Run Ingestion Job when using ADLS Gen2 as a source and destination

[Akash-Nair]

My objective is to import a CSV file into Pinot. The file is currently stored in an ADLS Gen2 container. I have implemented the necessary modifications to the relevant components, as advised in the documentation provided at https://docs.pinot.apache.org/basics/data-import/pinot-file-system/import-from-adls-azure.

However, despite these adjustments, the ingestion process is encountering a failure and producing the subsequent error message:

[Jackie-Jiang]

cc @snleee @swaminathanmanish

[Akash-Nair]

Hi, any updates on this issue? i think the directory 41e9835f-fa3f-4e83 .... is not being created and hence getting no such file or directory error.

[snleee]

@Akash-Nair We will need a bit more information (full logs and your ingestion spec configuration). Can you join to Pinot community slack if possible?

[amirjalali1]

i'm facing the same issue @Akash-Nair any resolution? @snleee Here is my spec I'm using the recipe for importing CSV files the only change here is reading from ADLS

` executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner' jobType: SegmentCreationAndTarPush inputDirURI: 'adl2://xxxxx/pin_in' outputDirURI: 'adl2://xxxxx/pin_out' overwriteOutput: true includeFileNamePattern: 'glob:**/import.csv' pinotFSSpecs:

• scheme: adl2 className: org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS configs: accountName: 'xxxx' accessKey: 'xxxxx' fileSystemName: 'xxxxx' recordReaderSpec: dataFormat: 'csv' className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader' configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig' tableSpec: tableName: 'crimes' pinotClusterSpecs:
  • controllerURI: 'http://pinot-controller-csv:9000/' pushJobSpec: pushAttempts: 2 pushRetryIntervalMillis: 1000

`