search

Castaglia / proftpd-mod_kafka

ProFTPD module for logging via Kafka
1 stars 0 forks source link

Send log messages to Kafka brokers using the librdkafka client library failed! #6

Open leocoder0 opened 2 years ago

leocoder0 commented 2 years ago

HI, I configured it according to the mod_kafka documentation mod_kafka.html , but I still cannot send logs to kafka broker. no error message! transfer log is ok!

What is the reason?

Here are the steps: First, I installed version 1.9.2 of librdkafka

image image

Second, copy the mod_kafka files into: proftpd-dir/contrib/

image

Then，I did the following ./configure \ --prefix=/usr/local/proftpd \ --sysconfdir=/etc/ --enable-nls --enable-openssl --enable-shadow \ --with-modules=mod_kafka \ --enable-dso --with-shared=mod_kafka \ --with-includes=/usr/local/include/librdkafka \ --with-libraries=/usr/local/lib

make 
make install

The proftpd.conf : 

ServerName                     "ProFTPD Default Installation"
ServerType                       standalone
DefaultServer                   on
Port                                   21
UseIPv6                            off
Umask                              022
MaxInstances                  30
User                                  proftp
Group                               proftp
DefaultRoot                     /var/ftp
SystemLog                      /var/log/ftpd/proftpd.log
TransferLog                    /var/log/ftpd/transfer.log
AllowOverwrite              on
RequireValidShell          off
AuthUserFile                  /usr/local/proftpd/ftpd.passwd

<Directory "/var/ftp/*" >
<Limit ALL>
AllowAll
</Limit>
</Directory>

<IfModule mod_kafka.c>
  KafkaEngine on
  KafkaLog /var/log/ftpd/kafka.log
  KafkaBroker 192.168.118.71:9092

  LogFormat kafka "%h %l %u %t \"%r\" %s %b"
  KafkaLogOnEvent ALL kafka first-topic
</IfModule>
Castaglia commented 2 years ago

Can you provide the contents of your configured KafkaLog file, so that we can see what clues may be in those log messages?

leocoder0 commented 2 years ago

Can you provide the contents of your configured KafkaLog file, so that we can see what clues may be in those log messages?

No log files are generated according to my configuration.

So is it that the mod_kafka module does not take effect, or is there any problem with the librdkafka configuration?

Castaglia commented 2 years ago

You compiled and install mod_kafka as a DSO/shared module; this means that in your configuration, you need to explicitly load the module first, before it can be used.

Thus your configuration should look something like:

<IfModule mod_dso.c>
  LoadModule mod_kafka.c
</IfModule>

<IfModule mod_kafka.c>
  KafkaEngine on
  KafkaLog /var/log/ftpd/kafka.log
  KafkaBroker 192.168.118.71:9092

  LogFormat kafka "%h %l %u %t \"%r\" %s %b"
  KafkaLogOnEvent ALL kafka first-topic
</IfModule>
leocoder0 commented 2 years ago
LoadModule mod_kafka.c
image

After adding this configuration, proftpd fails to start, prompting that mod_kafka.c cannot be found

[root@proftpd-74 proftpd]# sbin/proftpd 2022-08-10 10:55:05,420 proftpd-74 proftpd[8233]: fatal: LoadModule: error loading module 'mod_kafka.c': 无效的参数 on line 25 of '/etc/proftpd.conf'

Castaglia commented 2 years ago

Hmm. What does proftpd -V show?

leocoder0 commented 2 years ago

Hmm. What does proftpd -V show?

[root@proftpd-74 proftpd]# sbin/proftpd -V
Compile-time Settings:
  Version: 1.3.7e (maint)
  Platform: LINUX [Linux 5.11.12-300.el7.aarch64 aarch64]
  Built: 三 8月 10 2022 10:26:46 EDT
  Built With:
    configure  '--prefix=/usr/local/proftpd' '--sysconfdir=/etc/' '--enable-nls' '--enable-openssl' '--enable-shadow' '--with-modules=mod_kafka' '--enable-dso' '--with-shared=mod_kafka' '--with-includes=/usr/local/include/librdkafka' '--with-libraries=/usr/local/lib'

  CFLAGS: -g2 -O2 -Wall -fno-omit-frame-pointer -fno-strict-aliasing -Werror=implicit-function-declaration
  LDFLAGS: -L$(top_srcdir)/lib -L$(top_builddir)/lib  -rdynamic -L/usr/local/lib
  LIBS:  -lsupp -lnsl -lresolv -lresolv -lcrypt -ldl 

  Files:
    Configuration File:
      /etc/proftpd.conf
    Pid File:
      /usr/local/proftpd/var/proftpd.pid
    Scoreboard File:
      /usr/local/proftpd/var/proftpd.scoreboard
    Header Directory:
      /usr/local/proftpd/include/proftpd
    Shared Module Directory:
      /usr/local/proftpd/libexec

  Info:
    + Max supported UID: 4294967295
    + Max supported GID: 4294967295

  Features:
    - Autoshadow support
    - Controls support
    - curses support
    - Developer support
    + DSO support
    + IPv6 support
    + Largefile support
    - Lastlog support
    - Memcache support
    - ncurses support
    + NLS support
    - OpenSSL support
    - PCRE support
    - POSIX ACL support
    - Redis support
    + Sendfile support
    + Shadow file support
    - Sodium support
    + Trace support
    + xattr support

  Tunable Options:
    PR_TUNABLE_BUFFER_SIZE = 1024
    PR_TUNABLE_DEFAULT_RCVBUFSZ = 8192
    PR_TUNABLE_DEFAULT_SNDBUFSZ = 8192
    PR_TUNABLE_ENV_MAX = 2048
    PR_TUNABLE_GLOBBING_MAX_MATCHES = 100000
    PR_TUNABLE_GLOBBING_MAX_RECURSION = 8
    PR_TUNABLE_HASH_TABLE_SIZE = 40
    PR_TUNABLE_LOGIN_MAX = 256
    PR_TUNABLE_NEW_POOL_SIZE = 512
    PR_TUNABLE_PATH_MAX = 4096
    PR_TUNABLE_SCOREBOARD_BUFFER_SIZE = 80
    PR_TUNABLE_SCOREBOARD_SCRUB_TIMER = 30
    PR_TUNABLE_SELECT_TIMEOUT = 30
    PR_TUNABLE_TIMEOUTIDENT = 10
    PR_TUNABLE_TIMEOUTIDLE = 600
    PR_TUNABLE_TIMEOUTLINGER = 10
    PR_TUNABLE_TIMEOUTLOGIN = 300
    PR_TUNABLE_TIMEOUTNOXFER = 300
    PR_TUNABLE_TIMEOUTSTALLED = 3600
    PR_TUNABLE_XFER_SCOREBOARD_UPDATES = 10
image image

What files does the mod_kafka files say in the installation documentation? I just copied mod_kafka.c and mod_kafka.h.in, and renamed mod_kafka.h.in to mod_kafka.h,is this correct? If I don't rename it, it will prompt that mod_kafka.h cannot be found when make install.

my nstallation documentation

Castaglia commented 2 years ago

You shouldn't have both --with-modules=mod_kafka and --with-shared=mod_kafka in your configure command; you're trying to tell the ProFTPD build system that you want the same module to be built both as a static and as a shared module (which really doesn't make sense).

I recommend you use just this:

cd proftpd-1.3.7e/
git clone https://github.com/Castaglia/proftpd-mod_kafka.git contrib/mod_kafka/
make clean
./configure  --prefix=/usr/local/proftpd --sysconfdir=/etc/ --enable-nls --enable-openssl --enable-shadow --with-modules=mod_kafka --with-includes=/usr/local/include/librdkafka --with-libraries=/usr/local/lib
make
make install

You should not do anything with the .in files; the build system handles all of that automatically.

After doing the above install, please provide the output from running proftpd -t -d10 -c /path/to/your/proftpd.conf.

leocoder0 commented 2 years ago

According to the above configuration, it still does not work.

[root@proftpd-74 proftpd]# sbin/proftpd -t -d10 -c /etc/proftpd.conf 
sbin/proftpd: error while loading shared libraries: librdkafka.so.1: cannot open shared object file: No such file or directory
[root@proftpd-74 proftpd]# ll /usr/local/lib
-rwxr-xr-x. 1 root root 25033518 8月  10 10:15 librdkafka.a
-rwxr-xr-x. 1 root root  4661476 8月  10 10:15 librdkafka++.a
lrwxrwxrwx. 1 root root       15 8月  10 10:15 librdkafka.so -> librdkafka.so.1
lrwxrwxrwx. 1 root root       17 8月  10 10:15 librdkafka++.so -> librdkafka++.so.1
-rwxr-xr-x. 1 root root  9723888 8月  10 10:15 librdkafka.so.1
-rwxr-xr-x. 1 root root  1707544 8月  10 10:15 librdkafka++.so.1
-rwxr-xr-x. 1 root root 25033518 8月  10 10:15 librdkafka-static.a
drwxr-xr-x. 2 root root       96 8月  10 10:15 pkgconfig
Castaglia commented 2 years ago

Sounds like your dynamic linker cache needs to be updated. You may need to ensure that the /usr/local/lib directory is in your /etc/ld.so.conf (or /etc/ld.so.conf.d/ files), then, as root, run /sbin/ldconfig -V to update its cache.

After doing so, you might try the proftpd -t -d10 -c /path/to/proftpd.conf command again.

leocoder0 commented 2 years ago

Sounds like your dynamic linker cache needs to be updated. You may need to ensure that the /usr/local/lib directory is in your /etc/ld.so.conf (or /etc/ld.so.conf.d/ files), then, as root, run /sbin/ldconfig -V to update its cache.

After doing so, you might try the proftpd -t -d10 -c /path/to/proftpd.conf command again.

Same error, restart doesn't work.

[root@proftpd-74 ~]# ll /etc/ld.so.conf.d/
-rw-r--r--. 1 root root 26 10月 13 2020 bind-export-aarch64.conf
-rw-r--r--. 1 root root 17 10月  1 2020 mariadb-aarch64.conf
-rw-r--r--. 1 root root 15 8月  15 13:55 proftp.conf
-rw-r--r--. 1 root root 21 5月   8 2015 xulrunner-64.conf

[root@proftpd-74 ~]# cat /etc/ld.so.conf.d/proftp.conf 
/usr/local/lib

[root@proftpd-74 ~]# /sbin/ldconfig -V
ldconfig (GNU libc) 2.17
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[root@proftpd-74 ~]# /usr/local/proftpd/sbin/proftpd -t -d10 -c /etc/proftpd.conf 
/usr/local/proftpd/sbin/proftpd: error while loading shared libraries: librdkafka.so.1: cannot open shared object file: No such file or directory
Castaglia commented 2 years ago

Sorry, that should be /sbin/ldconfig -v (lowercase v). Can you run that (and provide its output), then try again?

leocoder0 commented 2 years ago

Sorry, that should be /sbin/ldconfig -v (lowercase v). Can you run that (and provide its output), then try again?

with the first-topic :

 23 <IfModule mod_kafka.c>
 24   KafkaEngine on
 25   KafkaLog /var/log/ftpd/kafka.log
 26   KafkaBroker 192.168.118.71:9092
 27 
 28   LogFormat kafka "%h %l %u %t \"%r\" %s %b"
 29   KafkaLogOnEvent ALL kafka first-topic
 30 </IfModule>
[root@proftpd-74 ~]# /usr/local/proftpd/sbin/proftpd -t -d10 -c /etc/proftpd.conf 
Checking syntax of configuration file
2022-08-15 14:24:48,339 proftpd-74 proftpd[18767]: using TCP receive buffer size of 131072 bytes
2022-08-15 14:24:48,339 proftpd-74 proftpd[18767]: using TCP send buffer size of 16384 bytes
2022-08-15 14:24:48,340 proftpd-74 proftpd[18767]: mod_kafka/0.1: using librdkafka 1.9.2
2022-08-15 14:24:48,340 proftpd-74 proftpd[18767]: using 'UTF-8' as local charset for UTF-8 conversion
2022-08-15 14:24:48,340 proftpd-74 proftpd[18767]: disabling runtime support for IPv6 connections
2022-08-15 14:24:48,340 proftpd-74 proftpd[18767]: retrieved UID 1000 for user 'proftp'
2022-08-15 14:24:48,340 proftpd-74 proftpd[18767]: retrieved GID 1000 for group 'proftp'
2022-08-15 14:24:48,340 proftpd-74 proftpd[18767]: ROOT PRIVS at mod_auth_file.c:1621
2022-08-15 14:24:48,341 proftpd-74 proftpd[18767]: RELINQUISH PRIVS at mod_auth_file.c:1624
2022-08-15 14:24:48,341 proftpd-74 proftpd[18767]: <Directory /var/ftp/*>: adding section for resolved path '/var/ftp/*'
2022-08-15 14:24:48,341 proftpd-74 proftpd[18767]: <IfModule>: using 'mod_kafka.c' section at line 23
2022-08-15 14:24:48,341 proftpd-74 proftpd[18767]: fatal: KafkaLogOnEvent: wrong number of parameters on line 29 of '/etc/proftpd.conf'

If you remove the first-topic, it starts normally, and a kafka topic is created, but when I upload the file, kafka doesn't receive any message. How should I configure it to push all proftpd logs (the most important file upload logs) to the specified topic of kafka?

This is all the logs for uploading a file: logs for upload a file

Castaglia commented 2 years ago

Ah, I see that I need to fix the documentation to be clearer, for specifying custom topics. For your configuration, you would want to use:

KafkaLogOnEvent ALL kafka topic first-topic

The topic keyword helps the config parser know that the tokens/parameters after topic are all custom topic names.

As for the information in each message that is published, here is what I use in the module tests, which uses most of the LogFormat variables to demonstrate:

LogFormat kafka "%A %a %b %c %D %d %E %{epoch} %F %f %{gid} %g %H %h %I %{iso8601} %J %L %l %m %O %P %p %{protocol} %R %r %{remote-port} %S %s %T %t %U %u %{uid} %V %v %{version}"
KafkaLogOnEvent ALL kafka topic first-topic
leocoder0 commented 2 years ago

Thank you very much for your support, so far, it seems that proftpd is normal to connect to kafka, because if I don't specify a topic, it will automatically create one. But this topic still cannot receive any proftpd operation messages (such as remote connection, upload file events). And, the configured kafka log( KafkaLog /var/log/ftpd/kafka.log) file is always empty.

Castaglia commented 2 years ago

OK. To get more information, then, you might try enabling trace logging for the module, using:

TraceLog /var/log/ftpd/kafka.log
Trace kafka:20

Hopefully that will generate log messages for the kafka trace channel to that log file, which might provide more details.

leocoder0 commented 2 years ago

OK. To get more information, then, you might try enabling trace logging for the module, using:

TraceLog /var/log/ftpd/kafka.log
Trace kafka:20

Hopefully that will generate log messages for the kafka trace channel to that log file, which might provide more details.

Here is my final configuration:

ServerName                     "ProFTPD Default Installation"
ServerType                       standalone
DefaultServer                   on
Port                                   21
UseIPv6                            off
Umask                              022
MaxInstances                  30
User                                  proftp
Group                               proftp
DefaultRoot                     /var/ftp
SystemLog                      /var/log/ftpd/proftpd.log
TransferLog                    /var/log/ftpd/transfer.log
AllowOverwrite              on
RequireValidShell          off
AuthUserFile                  /usr/local/proftpd/ftpd.passwd

#TraceLog /var/log/ftpd/kafka.log
#Trace kafka:20

<Directory "/var/ftp/*" >
<Limit ALL>
AllowAll
</Limit>
</Directory>

<IfModule mod_kafka.c>
  KafkaEngine on
  #KafkaLog /var/log/ftpd/kafka.log
  KafkaBroker kafka71:9092,kafka72:9092,kafka73:9092
  LogFormat kafka "%F %f %A %a %b %c %D %d %E %{epoch} %F %f %{gid} %g %H %h %I %{iso8601} %J %L %l %m %O %P %p %{protocol} %R %r %{remote-port} %S %s %T %t %U %u %{uid} %V %v %{version}"
  KafkaLogOnEvent ALL kafka topic first-topic
</IfModule>

file upload message:

{"remote_ip":"192.168.118.1","dir_name":"/","epoch":1660814862,"gid":1001,"group":"leo","server_ip":"proftpd-74","remote_dns":"192.168.118.1","session_bytes_rcvd":48,"timestamp":"2022-08-18 05:27:42,066","command_params":"192,168,118,1,237,107","local_ip":"192.168.118.74","command":"PORT","session_bytes_sent":119,"pid":13079,"local_port":21,"protocol":"ftp","response_millis":0,"raw_command":"PORT 192,168,118,1,237,107","remote_port":60644,"response_msg":"PORT命令成功执行","response_code":200,"local_time":"2022-08-18 09:27:42 +0000","original_user":"leo","user":"leo","uid":1001,"server_dns":"proftpd-74","server_name":"ProFTPD Default Installation","server_version":"1.3.7e"}

I found that even if KafkaLog and TraceLog are not configured, some events can be pushed to the specified topic, but I can't get transfer_path and file by configuring %F and %f. I am very confused, how can I get all the event messages and be able to get all the variables for each event?

Castaglia commented 2 years ago

The %f and %F variables only make sense for specific FTP commands, not all of them. Your example shows a PORT FTP command, which is used to set up the direction/ports used for the data transfer TCP connection; but a PORT command does not involve that actual file/path being transferred.

%f/%F usually are filled in for:

and perhaps some others.

To have mod_kafka publish messages only for those commands, you might try e.g.:

KafkaLogOnEvent RETR,STOR kafka topic first-topic
leocoder0 commented 2 years ago

The %f and %F variables only make sense for specific FTP commands, not all of them. Your example shows a PORT FTP command, which is used to set up the direction/ports used for the data transfer TCP connection; but a PORT command does not involve that actual file/path being transferred.

%f/%F usually are filled in for:

  • DELE
  • MKD
  • RETR
  • RMD
  • STOR
  • STOU

and perhaps some others.

To have mod_kafka publish messages only for those commands, you might try e.g.:

KafkaLogOnEvent RETR,STOR kafka topic first-topic
image

This is the system log of the uploaded file, it looks like there is a STOR command dispatched to the mod_kafka module, but no message is generated for this command and sent to kafka.

Castaglia commented 2 years ago

Hmm. This is unexpected. Does the kafka trace logging show anything?

leocoder0 commented 2 years ago
image

There is no log of the STOR command.

Castaglia commented 2 years ago

Hmm. I'm not able to reproduce this behavior (of no messages published for STOR commands/uploads) locally.

Can you provide the proftpd -V output again, so that I can see what might be different between your installation, and my local setup?

leocoder0 commented 2 years ago

I'm not able to reproduce this behavior (of no messages published for STOR commands/uploads) locally.

[root@proftpd-74 proftpd]# sbin/proftpd -V
Compile-time Settings:
  Version: 1.3.7e (maint)
  Platform: LINUX [Linux 5.11.12-300.el7.aarch64 aarch64]
  Built: 一 8月 15 2022 13:34:26 EDT
  Built With:
    configure  '--prefix=/usr/local/proftpd' '--sysconfdir=/etc/' '--enable-nls' '--enable-openssl' '--enable-shadow' '--with-modules=mod_kafka' '--with-includes=/usr/local/include/librdkafka' '--with-libraries=/usr/local/lib'

  CFLAGS: -g2 -O2 -Wall -fno-omit-frame-pointer -fno-strict-aliasing
  LDFLAGS: -L$(top_srcdir)/lib -L$(top_builddir)/lib  -rdynamic -L/usr/local/lib
  LIBS:  -lrdkafka -lsupp -lcrypt -ldl 

  Files:
    Configuration File:
      /etc/proftpd.conf
    Pid File:
      /usr/local/proftpd/var/proftpd.pid
    Scoreboard File:
      /usr/local/proftpd/var/proftpd.scoreboard

  Info:
    + Max supported UID: 4294967295
    + Max supported GID: 4294967295

  Features:
    - Autoshadow support
    - Controls support
    - curses support
    - Developer support
    - DSO support
    + IPv6 support
    + Largefile support
    - Lastlog support
    - Memcache support
    - ncurses support
    + NLS support
    - OpenSSL support
    - PCRE support
    - POSIX ACL support
    - Redis support
    + Sendfile support
    + Shadow file support
    - Sodium support
    + Trace support
    + xattr support

  Tunable Options:
    PR_TUNABLE_BUFFER_SIZE = 1024
    PR_TUNABLE_DEFAULT_RCVBUFSZ = 8192
    PR_TUNABLE_DEFAULT_SNDBUFSZ = 8192
    PR_TUNABLE_ENV_MAX = 2048
    PR_TUNABLE_GLOBBING_MAX_MATCHES = 100000
    PR_TUNABLE_GLOBBING_MAX_RECURSION = 8
    PR_TUNABLE_HASH_TABLE_SIZE = 40
    PR_TUNABLE_LOGIN_MAX = 256
    PR_TUNABLE_NEW_POOL_SIZE = 512
    PR_TUNABLE_PATH_MAX = 4096
    PR_TUNABLE_SCOREBOARD_BUFFER_SIZE = 80
    PR_TUNABLE_SCOREBOARD_SCRUB_TIMER = 30
    PR_TUNABLE_SELECT_TIMEOUT = 30
    PR_TUNABLE_TIMEOUTIDENT = 10
    PR_TUNABLE_TIMEOUTIDLE = 600
    PR_TUNABLE_TIMEOUTLINGER = 10
    PR_TUNABLE_TIMEOUTLOGIN = 300
    PR_TUNABLE_TIMEOUTNOXFER = 300
    PR_TUNABLE_TIMEOUTSTALLED = 3600
    PR_TUNABLE_XFER_SCOREBOARD_UPDATES = 10
leocoder0 commented 2 years ago

Hi, is there something wrong with my configuration and installation?

Castaglia commented 2 years ago

Nothing looks out of place to me at the moment, so I'm wondering how I could reproduce your environment locally.

What version of the librdkafka library (or package) do you have installed? What Docker image should I use, for the OS (and version) that you are using?

leocoder0 commented 2 years ago

Nothing looks out of place to me at the moment, so I'm wondering how I could reproduce your environment locally.

What version of the librdkafka library (or package) do you have installed? What Docker image should I use, for the OS (and version) that you are using?

The following is my environment information and installation steps

0.environment

Linux OS:

Linux version 5.11.12-300.el7.aarch64 (root@centos7.9) (gcc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), GNU ld version 2.30-55.el7.2) #1 SMP Thu Aug 19 09:02:08 UTC 2021

Software version:

librdkafka.1.9.2.tar.gz

proftpd-1.3.7e.tar.gz

1.librdkafka installation steps

Configure and install

tar -zxvf librdkafka.1.9.2.tar.gz -C /usr/src/

cd /usr/src/librdkafka-1.9.2

./configure

make 

make install

Default installation directory:

[root@proftpd-74 proftpd]# ll /usr/local/lib
-rwxr-xr-x. 1 root root 25033518 8月  10 10:15 librdkafka.a
-rwxr-xr-x. 1 root root  4661476 8月  10 10:15 librdkafka++.a
lrwxrwxrwx. 1 root root       15 8月  10 10:15 librdkafka.so -> librdkafka.so.1
lrwxrwxrwx. 1 root root       17 8月  10 10:15 librdkafka++.so -> librdkafka++.so.1
-rwxr-xr-x. 1 root root  9723888 8月  10 10:15 librdkafka.so.1
-rwxr-xr-x. 1 root root  1707544 8月  10 10:15 librdkafka++.so.1
-rwxr-xr-x. 1 root root 25033518 8月  10 10:15 librdkafka-static.a
drwxr-xr-x. 2 root root       96 8月  10 10:15 pkgconfig

[root@proftpd-74 proftpd]# ll /usr/local/include/librdkafka/
-rwxr-xr-x. 1 root root 124770 8月  10 10:15 rdkafkacpp.h
-rwxr-xr-x. 1 root root 307127 8月  10 10:15 rdkafka.h
-rwxr-xr-x. 1 root root  11665 8月  10 10:15 rdkafka_mock.h

2.proftpd installation steps

Configure and Install

tar -zxvf proftpd-1.3.7e.tar.gz -C /usr/src/

mkdir /usr/local/proftpd

cd /usr/src/proftpd-1.3.7e

git clone https://github.com/Castaglia/proftpd-mod_kafka.git contrib/mod_kafka/

./configure  --prefix=/usr/local/proftpd --sysconfdir=/etc/ --enable-nls --enable-openssl --enable-shadow --with-modules=mod_kafka --with-includes=/usr/local/include/librdkafka --with-libraries=/usr/local/lib

make

make install

Create working directory and user

mkdir /var/ftp
chmod 777 /var/ftp

mkdri /var/log/ftpd
chmod 755 /var/log/ftpd

useradd proftp -s /sbin/nologin -M
useradd leo -s /sbin/nologin -d /home/leo

ftpasswd  --passwd --file=/usr/local/proftpd/ftpd.passwd --name=leo  --uid=1001  --home=/home/leo  --shell=/sbin/nologin

proftpd configuration file

vim /etc/proftpd.conf

ServerName                     "ProFTPD Default Installation"
ServerType                       standalone
DefaultServer                   on
Port                                   21
UseIPv6                            off
Umask                              022
MaxInstances                  30
User                                  proftp
Group                               proftp
DefaultRoot                     /var/ftp
SystemLog                      /var/log/ftpd/proftpd.log
TransferLog                    /var/log/ftpd/transfer.log
AllowOverwrite              on
RequireValidShell          off
AuthUserFile                  /usr/local/proftpd/ftpd.passwd

TraceLog /var/log/ftpd/kafka.log
Trace kafka:20

<Directory "/var/ftp/*" >
<Limit ALL>
AllowAll
</Limit>
</Directory>

<IfModule mod_kafka.c>
  KafkaEngine on
  KafkaLog /var/log/ftpd/kafka.log
  KafkaBroker kafka71:9092,kafka72:9092,kafka73:9092

  LogFormat kafka "%A %a %b %c %D %d %E %{epoch} %F %f %{gid} %g %H %h %I %{iso8601} %J %L %l %m %O %P %p %{protocol} %R %r %{remote-port} %S %s %T %t %U %u %{uid} %V %v %{version}"
  KafkaLogOnEvent ALL kafka topic first-topic
</IfModule>

Start up:

/usr/local/proftpd/sbin/proftpd -d10 -c /etc/proftpd.conf
Castaglia commented 2 years ago

I've just pushed some changes that should hopefully add more logging (to the KafkaLog file) of librdkafka errors/warnings.

Also, in your configuration:

KafkaBroker kafka71:9092,kafka72:9092,kafka73:9092

should be:

KafkaBroker kafka71:9092 kafka72:9092 kafka73:9092

That is, a list of brokers should be separated by spaces, not by ,; the code now errors on this. I've also updated the docs for this.

leocoder0 commented 2 years ago 
./configure  --prefix=/usr/local/proftpd --sysconfdir=/etc/ --enable-nls --enable-openssl --enable-shadow --with-modules=mod_kafka --with-includes=/usr/local/include/librdkafka --with-libraries=/usr/local/lib

I re-installed it, but it still doesn't work, and the log doesn't seem to see any difference.

proftpd.log

image

kafka.log

image
leocoder0 commented 2 years ago 
./configure  --prefix=/usr/local/proftpd --sysconfdir=/etc/ --enable-nls --enable-openssl --enable-shadow --with-modules=mod_kafka --with-includes=/usr/local/include/librdkafka --with-libraries=/usr/local/lib

I re-installed it, but it still doesn't work, and the log doesn't seem to see any difference.

proftpd.log image

kafka.log image

Through the proftpd.log and kafka.log logs, I seem to find some patterns. Before the data connection is established, the commands sent to the mod_kafka module are working normally. The command sent to the mod_kafka module after the data connection is established does not take effect. So I was wondering, is it possible to send STOR commands to mod_kafka module before data connection?

leocoder0 commented 2 years ago

Hi, I still don’t know what to do next to achieve my needs. Can you give me some guidance or suggestions?

leocoder0 commented 2 years ago

Excuse me, are you still following this issue?

Castaglia commented 2 years ago

Yes, I'm still following this issue. I've had some personal issues to deal with in the past month or so.

leocoder0 commented 2 years ago

Ok, if you have time, please help to cut down where the problem is, now the log can not be sent to Kafka after the successful file upload