data-integrations / http

Sink plugin to send the messages from the pipeline to an external http endpoint
http://docs.cask.co/cdap
Other
1 stars 29 forks source link

[PLUGIN-1737] Added Schema Detection for CSV/TSV format #152

Closed psainics closed 9 months ago

psainics commented 9 months ago

Added Schema Detection for CSV/TSV format

Jira : Plugin-1737

Description

User will be able to get Schema for delimited files (CSV/TSV)

UI Fields changes

image

Docs

Get Schema: Auto-detects schema from file. Supported formats are: csv, tsv. Sample Size: The maximum number of rows in a file that will get investigated for automatic data type detection. Default is 100. CSV/TSV Skip First Row: Whether to skip the first row of the HTTP response. This is usually set if the first row is a header row.


Unit Tests

image

image

image

psainics commented 9 months ago

@albertshau The duplicated code is remove. I was not aware the format-X are published deps that can use outside the hydrator repo.

Can you help me with which cdap.plugin.version is appropriate for use here currently i have added the latest =2.13.0-SNAPSHOT.

albertshau commented 9 months ago

minor comments, please fix and squash commits, then we can merge