search

datamill-co / target-redshift

A Singer.io Target for Redshift
MIT License
23 stars 17 forks source link

Add arguments #20

Closed mirelagrigoras closed 5 years ago

mirelagrigoras commented 5 years ago

I have been experimenting with Redshift in general and especially with target-redhshift and I found the possibility to escape and accept invalid characters (of course, for varchar columns) when importing into Redshift tables particularly useful in order to prevent anomalies (because of odd characters) or in cases in which we do not mind replacing some invalid UTF-8 characters.

mirelagrigoras commented 5 years ago

Maybe this approach (to add more arguments to the target) is not the best one, but I think it would be useful to extend the target with some other features provided by the COPY command. I look forward to hear your opinion on this and if you think another approach (maybe in the config file) would be better, I would be more than happy to help with implementing those changes.

awm33 commented 5 years ago

I may be open to allowing extra copy parameters to be passed in, but I think it should be done via the configuration file instead of the CLI.

However, I think the root fix to your particular issue lies in fixing/ensuring proper UTF-8 text encoding in tap-hubspot.

mirelagrigoras commented 5 years ago

I agree that it started when using tap-hubspot (its a tap issue here), but, as far as Im concerned adding some copy parameters would be an improvement of how the target works right now. Do you mind if I try to do this in the configuration file and come back with a new pull request?