Closed Okanmercan99 closed 2 months ago
You should add the contentType
field to all sinkSettings
that are FileSystemSinkSettings
.
I believe that contentType
should be optional. In most cases, such as the example below, specifying contentType
is redundant because the file extension (e.g., .csv
) already indicates the content type:
"sourceBinding": {
"source": {
"jsonClass": "FileSystemSource",
"path": "patients.csv",
"contentType": "csv",
"sourceRef": "pilot1-source"
}
}
The use of contentType
should be limited to scenarios where a file extension can represent multiple content types. For instance, a .txt
file could contain either CSV or NDJSON data, so in that case, contentType
would clarify the format (e.g., csv
or ndjson
) while reading the file.
WDYT @YemreGurses, shall we make it optional ?
For this update, you should change the
fileFormat
field tocontentType
. Additionally, you should add thecontentType
field to all sources that areFileSystemSource
.