add support for sftp source and sink

[clojurians-org]

we have no access right to install vector and mount disk for remote data source. so we can only pull data from remote by sftp.

kafka connect already has a sftp connect.

https://docs.confluent.io/current/connect/kafka-connect-sftp/index.html

[gaby]

@binarylogic @jszwedko Any updates on this? SFTP via a single or pool of ssh connections would be very useful!

[spencergilbert]

This isn't on our near-term roadmap at the moment. We've recently started trying out integrations through OpenDAL, and while not supported yet sftp is on their roadmap.

[gaby]

@spencergilbert Thanks for the info. Looking forward to all this in the future!

[gaby]

@spencergilbert Although not released yet, sftp support was recently merged to OpenDAL :-)

https://github.com/apache/incubator-opendal/pull/2186

[gaby]

SFTP support has been released by OpenDAL: https://github.com/apache/incubator-opendal/releases/tag/v0.33.3

[Xuanwo]

Maybe we can start by adding sftp as a sink.

[gaby]

@Xuanwo That would be awesome!

[gaby]

Has there been any updates on this? 🤔

[jszwedko]

Has there been any updates on this? 🤔

Not that I'm aware of. I think a sink leveraging OpenDAL would be a good start though. If anyone is so motivated you can see the WebHDFS sink for an example of what this might look like (it also uses OpenDAL).

[Xuanwo]

Hi, I start a draft at https://github.com/vectordotdev/vector/pull/18076. Welcome to have a look!

Adding new opendal-based sinks is quite simple, as it mostly involves configuration and documentation tasks.

[gaby]

Hi, I start a draft at #18076. Welcome to have a look!

Adding new opendal-based sinks is quite simple, as it mostly involves configuration and documentation tasks.

Looking forward to this!

[gaby]

@Xuanwo Any updates on your PR? It has been stale for a few months now