search

keedio / flume-ftp-source

FTP network server is source of events for Apache-flume
80 stars 61 forks source link

Filename in the Event Header #15

Closed deepakbs closed 9 years ago

deepakbs commented 9 years ago

Hi,

I require the name of the file being streamed from the FTPSource in my custom interceptor. Is it a good idea to add the file name being processed in the event header in FTPSource so that we get that in the custom interceptor.

Thanks Deepak

danieltardon commented 9 years ago

Hi @deepakbs,

I've noticed that you have been opening this "issue" several times despite of @lazaromedina closed it thinking that is not a bug/problem, so I assume you are very interested in this new feature or you need it soon.

It would be a pleasure to find a way to collaborate and discuss the real need of include these features that you request in this "issue", and in others you have opened, in the mainstream of the project.

Could you provide us an email to talk about it?

Thanks Daniel

deepakbs commented 9 years ago

Hi @danieltardon Thanks for the support.

We need the filename of the file being consumed for our project needs. Along with this we also need to be able to fetch only certain format of files from the FTPSource. So i would like to propose the following 2 additions in FTPSource

1) headers.put("filename", filename); in processMessage method where the filename is a class variable and will be assigned the file name inside the loop for (FTPFile aFile : subFiles) in method discoverElements

2) Introduce filepattern configuration element and
add a FTPFileFilter in discoverElements() using that file pattern,

Can you please add these features,

Thanks Deepak (deepakbs@gmail.com)