search

meltano / hub

The single source of truth for all Meltano plugins, including all available Singer Taps and Targets: https://hub.meltano.com
https://hub.meltano.com
50 stars 68 forks source link

Add target-clickhouse #652

Open DouweM opened 2 years ago

DouweM commented 2 years ago

https://github.com/Biron-BI/singer-target-clickhouse is written in Node and has a Docker image at ghcr.io/biron-bi/target-clickhouse, so we may be able to support it using container_spec.

DouweM commented 2 years ago

@pandemicsyn Do you know if meltano run could support container_spec-based Singer target? In the end it's just an executable so it should be able to work, but I don't know if our codebase is prepared for that yet.

pandemicsyn commented 2 years ago

@pandemicsyn Do you know if meltano run could support container_spec-based Singer target? In the end it's just an executable so it should be able to work, but I don't know if our codebase is prepared for that yet.

I haven't looked/thought much about container_spec-based taps/targets, but in the README the example npm installed package where target-clickhouse is being invoked directly, should "just work" out of the box.

I think it's just the variation where docker is invoked directly that might not currently be supported in meltano run. Although you could maybe work around that by wrapping the docker call in a script.

DouweM commented 2 years ago

@pandemicsyn Yeah if Meltano can find the executable we should be good, the problem is that it can't currently install npm packages, only pip and docker.

container_spec is currently only used by meltano invoke (https://github.com/meltano/meltano/blob/d6e54dc25425fb6cd5884b974aeda92f1bfef8f0/src/meltano/cli/invoke.py#L167), not when meltano run calls the PluginInvoker, so perhaps this could be solved by moving that logic down and making sure it works with stdin/stdout as expected. Then it should be transparent to meltano run whether it ends up calling an executable directly, or calling it through Docker.

pandemicsyn commented 2 years ago

Yea , afaik, one other hiccup long term might be that container_spec is also only docker api compatible (leverages the actual docker api). Which might present problems for folks when they go to deploy. Which we've got logged here: