singer-io / tap-s3-csv

GNU Affero General Public License v3.0

27 stars 52 forks source link

TDL-14068:fixed key-error exception #38

Closed savan-chovatiya closed 3 years ago

savan-chovatiya commented 3 years ago

Description of change

TDL-14068:Fix key-error exception being thrown.

When all 5 CSV files used in sampling are without any records, the catalog will be generated without properties field. It's the same behavior as old tap. We are using these properties in the new tap version for duplicate and no headers handling.

Solution:

Emit all records in sync mode for other files with duplicate and no header handling if the catalog is generated without properties.
If a catalog contains properties, records with only fields mentioned in the catalog will be provided as output.

Manual QA steps

Verified that if all 5 CSV files used in sampling are without records, all the records will be emitted with duplicates and no headers handling.

Risks

No risk

Rollback steps

revert this branch

cosimon commented 3 years ago

@savan-chovatiya Can we get a test for this scenario?

savan-chovatiya commented 3 years ago

@savan-chovatiya Can we get a test for this scenario?

@cosimon Added unit tests and integration test for the same.