search

keedio / flume-ftp-source

FTP network server is source of events for Apache-flume
80 stars 61 forks source link

FTPS using flume #32

Open sangeesivakumar opened 6 years ago

sangeesivakumar commented 6 years ago

I am trying to using ftps connector using flume to fetch secured data.

My property file `# Naming the components on the current agent. FtpAgent.sources = ftps1 FtpAgent.channels = FileChannel FtpAgent.sinks = HDFS

Describing/Configuring the source FtpAgent.sources.ftps1.type = org.keedio.flume.source.ftp.source.Source FtpAgent.sources.ftps1.client.source = ftps FtpAgent.sources.ftps1.name.server = 192.168.1.10 FtpAgent.sources.ftps1.user = xxx FtpAgent.sources.ftps1.password = xxx FtpAgent.sources.ftps1.port = 21

FtpAgent.sources.ftps1.security.enabled = true FtpAgent.sources.ftps1.security.cipher = SSL FtpAgent.sources.ftps1.security.certificate.enabled = true FtpAgent.sources.ftps1.path.keystore = /usr/hdp/current/flume-server/keystore.jks

FtpAgent.sources.ftps1.path.keystore=/etc/ssl/certs/ca-certificates.crt

FtpAgent.sources.ftps1.store.pass = xxx

FtpAgent.sources.ftps1.flushlines = false FtpAgent.sources.ftps1.chunk.size = 1024 FtpAgent.sources.ftps1.run.discover.delay=5000 FtpAgent.sources.ftps1.working.directory = /home/volumata/Desktop/ftp_sample

FtpAgent.sources.ftps1.filter.pattern = pig_.

FtpAgent.sources.ftps1.filter.pattern = .+.csv

FtpAgent.sources.ftps1.search.recursive = false

FtpAgent.sources.ftps1.folder = /home/volumata/Desktop/ftp_sample

Describing/Configuring the sink FtpAgent.sinks.HDFS.type = hdfs FtpAgent.sinks.HDFS.hdfs.path = hdfs://xxx:8000/topics/flume_ftp_source FtpAgent.sinks.HDFS.hdfs.fileType = DataStream FtpAgent.sinks.HDFS.hdfs.writeFormat = Text FtpAgent.sinks.HDFS.hdfs.batchSize = 1000 FtpAgent.sinks.HDFS.hdfs.rollSize = 100 FtpAgent.sinks.HDFS.hdfs.rollCount = 100000 FtpAgent.sinks.hdfs.serializer=Text

Describing/Configuring the channel FtpAgent.channels.FileChannel.type = file FtpAgent.channels.FileChannel.capacity = 100000`

I am running the agent using this command,

bin/flume-ng agent --conf ./conf/ -f conf/ftps_flume_source.conf Dflume.root.logger=DEBUG,console -n FtpAgent

I am getting this error while running the agent

ERROR source.Source: Exception thrown in process, try to reconnect 0 javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: Unsupported record version Unknown-48.48

I am not sure where i did mistakes,please anyone help me resolve this issue.

lazaromedina commented 6 years ago

Hi sangeesivakumar, 

FtpAgent.sources.ftps1.security.cipher = SSL

Try with 'TLS'. The ftp client seems to be trying to connect to a socket that is not expecting a SSL handshake.

best, Luis