search

proftpd / proftpd

ProFTPD source code
http://www.proftpd.org
GNU General Public License v2.0
526 stars 200 forks source link

mod_sftp with HiddenStores and sshfs does not go well together (No such file or directory) #1829

Open maltris opened 2 months ago

maltris commented 2 months ago

What I Did

I try to use HiddenStores with mod_sftp but some clients use sshfs.

What I Expected/Wanted

I expect nothing, but I am curious how to prevent "No such file or directory" situations where applications like file editors or simple writes to a new file try to create a file, then open it for writing but the then-opened file (the hidden version with .in) does not exist because right after creation it had been renamed by proftpd.

In my understanding there is no protocol way to lock the file in the hidden position until it had really been written on sshfs. But is there any other way? Or is it possible to disable HiddenStores ONLY for the sshfs clients (they send the normal OpenSSH client identifier).

I have read about writeback_cache in libfuse/sshfs, but it is not available currently:

https://github.com/search?q=repo%3Alibfuse%2Fsshfs+writeback_cache&type=commits

Any ideas?

# touch mytestfile1

# echo 1 > mytestfile2
-bash: mytestfile2: Datei oder Verzeichnis nicht gefunden

# echo 1 > ~/mytestfile3

# cp ~/mytestfile3 .
cp: reguläre Datei './mytestfile3' kann nicht angelegt werden: Datei oder Verzeichnis nicht gefunden

# ls -lah mytestfile*
-rw-r--r-- 1 345 345 0 13. Sep 13:49 mytestfile1
-rw-r--r-- 1 345 345 0 13. Sep 13:49 mytestfile2
-rw-r--r-- 1 345 345 0 13. Sep 13:51 mytestfile3

ProFTPD Version and Configuration

Please help us reproduce the problem/issue you are encountering. To do this, we need to know which version of ProFTPD you are using, how it was built, etc. The following command is an easy way to get all of this information:

proftpd -V
/ # proftpd -V
Compile-time Settings:
  Version: 1.3.8b (maint)
  Platform: LINUX [Linux 5.4.0-120-generic x86_64]
  OS/Release:
    NAME="Alpine Linux"
    ID=alpine
    VERSION_ID=3.19.0
    PRETTY_NAME="Alpine Linux v3.19"
  Built: Mon Jan 8 2024 16:08:53 UTC
  Built With:
    configure  '--enable-ctrls' '--enable-openssl' '--enable-quotatab' '--with-modules=mod_tls:mod_quotatab:mod_sql:mod_sql_passwd:mod_quotatab_sql:mod_quotatab_file:mod_ctrls_admin:mod_auth_otp:mod_sql_sqlite:mod_sql_mysql:mod_ban:mod_readme:mod_sftp:mod_sftp_sql:mod_proxy_protocol:mod_ident:mod_ifsession' '--enable-nls'

  CFLAGS: -g2 -O2 -Wall -fno-omit-frame-pointer -fno-strict-aliasing
  LDFLAGS: -Wl,-L$(top_srcdir)/lib,-L$(top_builddir)/lib  -rdynamic  -L/usr/lib/
  LIBS:  -lssl -lcrypto -lm -lmysqlclient -lz  -lsqlite3  -lssl  -lcrypto 

  Files:
    Configuration File:
      /usr/local/etc/proftpd.conf
    Pid File:
      /usr/local/var/proftpd.pid
    Scoreboard File:
      /usr/local/var/proftpd.scoreboard

  Info:
    + Max supported UID: 4294967295
    + Max supported GID: 4294967295

  Features:
    - Autoshadow support
    + Controls support
    - curses support
    - Developer support
    - DSO support
    + IPv6 support
    + Largefile support
    - Lastlog support
    - Memcache support
    - ncurses support
    + NLS support
    + OpenSSL support (OpenSSL 3.1.4 24 Oct 2023)
    - PCRE support
    - PCRE2 support
    - POSIX ACL support
    - Redis support
    + Sendfile support
    + Shadow file support
    - Sodium support
    + Trace support
    + xattr support

  Tunable Options:
    PR_TUNABLE_BUFFER_SIZE = 1024
    PR_TUNABLE_DEFAULT_RCVBUFSZ = 8192
    PR_TUNABLE_DEFAULT_SNDBUFSZ = 8192
    PR_TUNABLE_ENV_MAX = 2048
    PR_TUNABLE_GLOBBING_MAX_MATCHES = 100000
    PR_TUNABLE_GLOBBING_MAX_RECURSION = 8
    PR_TUNABLE_HASH_TABLE_SIZE = 40
    PR_TUNABLE_LOGIN_MAX = 256
    PR_TUNABLE_NEW_POOL_SIZE = 512
    PR_TUNABLE_PATH_MAX = 4096
    PR_TUNABLE_SCOREBOARD_BUFFER_SIZE = 80
    PR_TUNABLE_SCOREBOARD_SCRUB_TIMER = 30
    PR_TUNABLE_SELECT_TIMEOUT = 30
    PR_TUNABLE_TIMEOUTIDENT = 10
    PR_TUNABLE_TIMEOUTIDLE = 600
    PR_TUNABLE_TIMEOUTLINGER = 10
    PR_TUNABLE_TIMEOUTLOGIN = 300
    PR_TUNABLE_TIMEOUTNOXFER = 300
    PR_TUNABLE_TIMEOUTSTALLED = 3600
    PR_TUNABLE_XFER_SCOREBOARD_UPDATES = 10

In addition, we need to see all of the ProFTPD configuration files you are using (minus any sensitive information like passwords, of course). Armed with the version and configuration data, then, we can set up ProFTPD locally using the same configuration, and see what happens.

<global>
RootLogin off
RequireValidShell off
AllowOverwrite on
WtmpLog off
TransferLog /var/log/xfer.log
DefaultRoot ~

HiddenStores .in. .%P

IdentLookups                    off
DeferWelcome                    off
MultilineRFC2228                on
ShowSymlinks                    on

TimeoutNoTransfer               6000
TimeoutStalled                  6000
TimeoutIdle                     12000

SQLBackend              mysql
SQLAuthTypes            Crypt
SQLAuthenticate         users groups

SQLConnectInfo  redacted

SQLGroupInfo    ftpgroup groupname gid members
SQLMinID        500
CreateHome on

SQLNamedQuery get-user-authorized-keys SELECT "user_key FROM sftpuserkeys WHERE user='%U'"
SQLNamedQuery get-host-authorized-keys SELECT "host_key FROM sftphostkeys WHERE host='%{0}'"

    <Directory /proc>
        <Limit ALL>
            DenyAll
        </Limit>
    </Directory>

    <IfModule mod_proxy_protocol.c>
        ProxyProtocolEngine on
        ProxyProtocolVersion haproxyV2
        AllowForeignAddress on
    </IfModule>

</global>

<IfModule mod_sftp.c>
    <VirtualHost 0.0.0.0>
        Port 22
        SFTPOptions     IgnoreSFTPUploadPerms PessimisticKexinit OldProtocolCompat
        SFTPEngine      on
        SFTPCompression delayed

        SFTPHostKey /etc/ssh/ssh_host_rsa_key
        SFTPHostKey /etc/ssh/ssh_host_ecdsa_key

        SQLUserInfo     ftpusersftp userid passwd uid gid homedir shell

        <IfModule mod_sftp_sql.c>
             SFTPAuthorizedUserKeys sql:/get-user-authorized-keys
             SFTPAuthorizedHostKeys sql:/get-host-authorized-keys
        </IfModule>

    SFTPLog /var/log/sftp.log
    </VirtualHost>
</IfModule>

<IfModule mod_sql_passwd.c>
    <VirtualHost 0.0.0.0>
        <IfModule mod_tls.c>
            TLSEngine on
            TLSRequired off
            TLSRSACertificateFile certfile
            TLSRSACertificateKeyFile keyfile
            TLSVerifyClient            off
            TLSOptions                 NoSessionReuseRequired
        </IfModule>

        Port 21
        DefaultServer on

        PassivePorts 10000 10250
        MasqueradeAddress redacted

        SQLPasswordEngine on

        SQLUserInfo     ftpuser userid passwd uid gid homedir shell
    </VirtualHost>
</IfModule>

Port 0

UseIPv6                         off
UseReverseDNS                   off

ServerType                      standalone

SystemLog /var/log/proftpd.log
SyslogLevel info
Castaglia commented 2 months ago

The mod_sftp module only renames the files on close, not on open.

To better understand what's happening at the protocol level, I would need to see the SFTPLog generated by the sshfs-using client machine, to see the order of specific SFTP requests being used.

maltris commented 1 month ago 
2024-09-18 14:22:54,907 mod_sftp/1.1.1[797854]: user 'myuser' authenticated via 'password' method
2024-09-18 14:22:55,151 mod_sftp/1.1.1[797854]: 'env' channel request: 'LANG' = 'de_DE.UTF-8'
2024-09-18 14:22:55,151 mod_sftp/1.1.1[797854]: 'subsystem' channel request for 'sftp' subsystem
2024-09-18 14:22:55,151 mod_sftp/1.1.1[797854]: using SFTP protocol version 3 for this session (channel ID 0)
2024-09-18 14:24:03,849 mod_sftp/1.1.1[797854]: client sent GLOBAL_REQUEST for 'keepalive@openssh.com', denying
2024-09-18 14:24:21,926 mod_sftp/1.1.1[797854]: error checking '/test/mytestfile' for LSTAT: No such file or directory
2024-09-18 14:24:21,952 mod_sftp/1.1.1[797854]: error checking '/test/mytestfile' for LSTAT: No such file or directory
2024-09-18 14:24:36,998 mod_sftp/1.1.1[797854]: error checking '/test/mytestfile2' for LSTAT: No such file or directory
2024-09-18 14:24:37,022 mod_sftp/1.1.1[797854]: error checking '/test/mytestfile2' for LSTAT: No such file or directory
2024-09-18 14:25:57,467 mod_sftp/1.1.1[797854]: error checking '/test/mytestfile3' for LSTAT: No such file or directory
2024-09-18 14:25:57,491 mod_sftp/1.1.1[797854]: error checking '/test/mytestfile3' for LSTAT: No such file or directory
2024-09-18 14:25:57,516 mod_sftp/1.1.1[797854]: error checking '/test/mytestfile3' for LSTAT: No such file or directory

Thats what I get in the normal SFTPLog. 

[00116] LSTAT
  [00116]         STATUS       38bytes (276ms)
[00117] OPEN
[00118] LSTAT
  [00117]         HANDLE       29bytes (26ms)
  [00118]         STATUS       38bytes (26ms)
[00119] CLOSE
[00120] LSTAT
  [00119]         STATUS       28bytes (26ms)
  [00120]          ATTRS       45bytes (26ms)
[00121] SETSTAT
  [00121]         STATUS       28bytes (23ms)
[00122] LSTAT
  [00122]          ATTRS       45bytes (24ms)

[00123] LSTAT
  [00123]         STATUS       38bytes (25ms)
[00124] OPEN
[00125] LSTAT
  [00124]         HANDLE       29bytes (23ms)
  [00125]         STATUS       38bytes (23ms)
[00126] CLOSE
  [00126]         STATUS       28bytes (27ms)

[00127] LSTAT
  [00127]         STATUS       38bytes (26ms)
[00128] LSTAT
  [00128]         STATUS       38bytes (23ms)
[00129] OPEN
[00130] LSTAT
  [00129]         HANDLE       29bytes (24ms)
  [00130]         STATUS       38bytes (24ms)
[00131] CLOSE
  [00131]         STATUS       28bytes (26ms)

This is what I get from the sshfs debug log. (The attempts from above mytestfile1-3 separated by a few newlines.)

I did another test with an editor (nano) opening the file on the sshfs:

2024-09-18 14:34:41,647 mod_sftp/1.1.1[797854]: error checking '/test/test2' for LSTAT: No such file or directory
2024-09-18 14:34:41,694 mod_sftp/1.1.1[797854]: error checking '/test/.test2.swp' for LSTAT: No such file or directory
2024-09-18 14:34:41,717 mod_sftp/1.1.1[797854]: error checking '/test/.test2.swp' for LSTAT: No such file or directory
2024-09-18 14:34:41,741 mod_sftp/1.1.1[797854]: error checking '/test/.test2.swp' for LSTAT: No such file or directory
2024-09-18 14:34:41,765 mod_sftp/1.1.1[797854]: error checking '/test/.test2.swp' for LSTAT: No such file or directory
2024-09-18 14:34:41,813 mod_sftp/1.1.1[797854]: error checking '/test/test2' for LSTAT: No such file or directory
2024-09-18 14:34:41,836 mod_sftp/1.1.1[797854]: error checking '/test/test2' for LSTAT: No such file or directory
2024-09-18 14:34:41,858 mod_sftp/1.1.1[797854]: error checking '/test/test2' for LSTAT: No such file or directory
2024-09-18 14:34:46,442 mod_sftp/1.1.1[797854]: error checking '/test/test2' for LSTAT: No such file or directory
2024-09-18 14:34:46,467 mod_sftp/1.1.1[797854]: error checking '/test/test2' for LSTAT: No such file or directory
2024-09-18 14:34:46,492 mod_sftp/1.1.1[797854]: error checking '/test/test2' for LSTAT: No such file or directory
2024-09-18 14:34:46,516 mod_sftp/1.1.1[797854]: error checking '/test/test2' for LSTAT: No such file or directory
2024-09-18 14:34:46,539 mod_sftp/1.1.1[797854]: error checking '/test/test2' for LSTAT: No such file or directory
2024-09-18 14:34:46,567 mod_sftp/1.1.1[797854]: error checking '/test/test2' for LSTAT: No such file or directory

[00142] LSTAT
  [00142]         STATUS       38bytes (25ms)
[00143] LSTAT
  [00143]          ATTRS       45bytes (22ms)
[00144] LSTAT
  [00144]         STATUS       38bytes (23ms)
[00145] LSTAT
  [00145]         STATUS       38bytes (22ms)
[00146] LSTAT
  [00146]         STATUS       38bytes (22ms)
[00147] OPEN
[00148] LSTAT
  [00147]         HANDLE       29bytes (23ms)
  [00148]         STATUS       38bytes (23ms)
[00149] CLOSE
[00150] LSTAT
  [00149]         STATUS       28bytes (24ms)
  [00150]          ATTRS       45bytes (24ms)
[00151] LSTAT
  [00151]         STATUS       38bytes (22ms)
[00152] LSTAT
  [00152]         STATUS       38bytes (22ms)
[00153] LSTAT
  [00153]         STATUS       38bytes (22ms)
[00154] LSTAT
  [00154]         STATUS       38bytes (25ms)
[00155] LSTAT
  [00155]         STATUS       38bytes (25ms)
[00156] LSTAT
  [00156]         STATUS       38bytes (23ms)
[00157] LSTAT
  [00157]         STATUS       38bytes (22ms)
[00158] LSTAT
  [00158]         STATUS       38bytes (23ms)
[00159] OPEN
[00160] LSTAT
  [00159]         HANDLE       29bytes (27ms)
  [00160]         STATUS       38bytes (27ms)
[00161] CLOSE
  [00161]         STATUS       28bytes (24ms)
Castaglia commented 1 month ago

Hmm. Could you also enable trace logging, for more detailed logging of the underlying protocol requests?

TraceLog /path/to/proftpd-trace.log
Trace ssh2:30 sftp:30

I'm hoping to see the exact sequence of protocol-level interactions with these files. Thanks!

maltris commented 1 month ago

@Castaglia To not leak any information accidentially, because the trace is very verbose, I sent the trace to you via mail just in this moment.

Castaglia commented 1 month ago

Thanks for the trace logging. I understand the issue now.

When mod_sftp receives the OPEN SFTP request to create a file, and if HiddenStores are enabled, then the actual path created on the filesystem will be the temporary filename -- not the name as requested by the client. Only when the final CLOSE SFTP request is received, for that file, will mod_sftp rename the temporary file atomically to the requested filename. This is how HiddenStores is intended to work.

But your SFTP client (sshfs, in this case) is sending LSTAT requests, with the requested filename, during this file transfer. Hence why those LSTAT requests are properly receiving the "no such file or directory" response -- as that requested filename doesn't exist yet, not until the CLOSE.

The simplest approach would be to not use the HiddenStores directives for your SFTP clients. Or perhaps to create two different SFTP <VirtualHost> sections, differentiated by DNS name or port, so that you point your sshfs clients at the <VirtualHost> configuration that does not enable HiddenStores, while all other SFTP clients use the <VirtualHost> that does enable HiddenStores.

Ideally the sshfs client would be using FSTAT SFTP requests, not LSTAT requests. FSTAT uses the file handle as returned to the client for its OPEN requests; it is effectively the file descriptor, and thus can be used to obtain the desired information regardless of the filesystem path.

maltris commented 1 month ago

Highly interesting!

HiddenStores as of now cannot be disabled with SFTPClientMatch if I understood correctly, so the only solution would be a separate instance or vhost?

Is there any other way to control it?

I will try to ask the sshfs community as well if there could be any compatibility mode for this.

Castaglia commented 1 month ago

Currently HiddenStores cannot be enabled/disabled based on SFTPClientMatch, correct.

Another possibility might be to use mod_ifsession's <IfUser>, <IfGroup>, <IfClass> sections to enable/disable HiddenStores based on the user/group used by the sshfs client, or perhaps the network class. That, in addition to the possibility of using separate <VirtualHost> sections.