statcache SFTP issue?

[tspicer]

Statcache seems to have an issue via SFTP. When statcache is "off" I can transfer files without issue. However, turn it on and I see this in the sftp logs

2016-02-23 00:30:05,463 mod_sftp/1.0.0[3029]: fstat error on '/foo.txt' (fd 23): No such file or directory 2016-02-23 00:30:05,463 mod_sftp/1.0.0[3029]: error checking '/foo.txt': No such file or directory 2016-02-23 00:30:09,275 mod_sftp/1.0.0[3029]: fstat error on '/IMPORTTEST_balance_TST.txt' (fd 23): No such file or directory 2016-02-23 00:30:09,275 mod_sftp/1.0.0[3029]: error checking '/IMPORTTEST_balance_TST.txt': No such file or directory

This does not seem to be an issue vis FTPES.

[Castaglia]

So is this the cause of this issue, do you think? Or is this something else/different?

[tspicer]

Hi, I'm not sure. The only variable for the above was attempting to do a build with mod_statcache. With it on I get the issue with sftp and not ftp(s).

###################

CACHE

###################

StatCacheEngine on StatCacheControlsACLs all allow user root StatCacheCapacity 6000 StatCacheTable /ebs/db/statcache.tab StatCacheMaxAge 300

[Castaglia]

This PR should help with the timeouts. As for the unexpected errors, I am wondering if those are caused by mod_vroot + mod_statcache; attempting to replicate that behavior now.

[tspicer]

I have it turned off, the same as the quota.

Using mod_vroot to set non-kernel permissions

VRootEngine off VRootServerRoot /mnt/ebs/ftpd/ VRootLog /ebs/logs/proftpd/proftpd_vroot.log

I'm wondering if this has to do with the use of hidden stores. I can see the start of the files with the . prefix, but just the initial file.

[tspicer]

One of my users had issues when hidden stores was active so I had to add this in for them <IfUser !dbit> HiddenStores "." "" </IfUser>

My sftp config:

  <VirtualHost 0.0.0.0>
    Port 2222
    SFTPEngine               on
    CreateHome               on 711 dirmode 700
    SFTPAuthMethods          publickey password
    SFTPCryptoDevice         all
   <IfClass !localhost>
       SFTPLog               /ebs/logs/proftpd/proftpd_sftp.log all default
    </IfClass>
    SFTPTrafficPolicy        low
    SFTPHostKey              /etc/ssh/ssh_host_rsa_key
    SFTPHostKey              /etc/ssh/ssh_host_dsa_key
    SFTPAuthorizedUserKeys   file:/etc/pub/%u/.ssh/authorized_keys
    SFTPCompression          delayed
    SFTPClientAlive          3 60
    SFTPOptions              IgnoreSFTPUploadPerms IgnoreSCPUploadPerms IgnoreSFTPSetPerms IgnoreSFTPSetOwners PessimisticKexinit

SFTPRekey none

Default Values: channelPacketSize 32KB channelWindowSize 4GB sftpProtocolVersion 1-6

General fixes from the mod_sftp forum

SFTPClientMatch ".WS_FTP." channelWindowSize 1GB # WS_FTP initial window size SFTPClientMatch ".ClientSftp" sftpProtocolVersion 3 # CuteFTPPro8 SFTPClientMatch ".WinSCP." sftpProtocolVersion 3 # upload/download fix for WinSCP SFTPClientMatch ".CoreFTP." channelWindowSize 1GB # CoreFTP LE (free version 2.2) can not support normal channel 4GB. SFTPClientMatch ".SecureBlackbox." sftpProtocolVersion 3 # SecureBlackbox (SSH-2.0-SecureBlackbox.7) SFTPClientMatch ".J2SSH_Maverick." channelWindowSize 1GB # JaSFtp (SSH-2.0-J2SSH_Maverick_1.2.10_Sterling Commerce) SFTPClientMatch ".WeOnlyDo." sftpProtocolVersion 3 channelWindowSize 1GB # Robo-FTP (SSH-2.0-WeOnlyDo) SFTPClientMatch ".EldoS.SSHBlackbox.3." sftpProtocolVersion 3 channelWindowSize 1GB # Network Automation (SSH-2.0-EldoS.SSHBlackbox.3) SFTPClientMatch ".IP.Works." channelWindowSize 1GB # Software BizTalk SFTP Receive (SSH-2.0-IPWorks! SSH Client v8.0) SFTPClientMatch "JSCH." channelWindowSize 1GB # JSch - Java Secure Channel (SSH-2.0-JSCH-0.1.39) SFTPClientMatch "SecureFX." sftpProtocolVersion 3 # SecureFX - SSH2 client 'SecureFX_2_2_5_225 SecureFX' SFTPClientMatch "1.0" sftpProtocolVersion 2 channelWindowSize 1GB # GoAnywhere (SSH-2.0-1.0) SFTPClientMatch ".Sun_SSH." channelWindowSize 1GB # SSH-2.0-Sun_SSH_1.0.1 SFTPClientMatch ".XFB.Gateway Unix." channelWindowSize 1GB # SSH-2.0-XFB.Gateway Unix SFTPClientMatch ".SharpSSH." channelWindowSize 256MB # SSH-2.0-SharpSSH-1.1.1.13-JSCH-0.1.28 SFTPClientMatch "1.30" channelWindowSize 256MB channelPacketSize 16KB # SSH-2.0-1.30 SFTPClientMatch "^OpenSSH_3\.*" channelWindowSize 8MB # Older OpenSSH clients

SFTPAuthorizedUserKeys sql:/get-user-authorized-keys

SFTPAuthorizedHostKeys sql:/get-host-authorized-keys

   </VirtualHost>
</IfModule>``` 

[Castaglia]

For mod_statcache and mod_sftp, you might try using:

<IfModule mod_sftp.c>
  ...
  <IfModule mod_statcache.c>
    StatCacheMaxAge 5 0
  </IfModule>
</IfModule>

This will tell mod_statcache to not cache any negative stat(2)/fstat(2) calls; this is what made SFTP uploads work in my local testing. I suspect that mod_statcache will need to be more SFTP-aware, due to issues like this.

[Castaglia]

This should now be working with the mod_statcache in the proftpd master branch, without needing the manual StatCacheMaxAge config working (mod_statcache now does this disabling of negative caching for SSH2 sessions automatically).

[tspicer]

OK, will start testing this on my end. Will report back my findings.

[tspicer]

I cant yet confirm the fixes. Been having trouble fully transferring large (450+MB) test file.

This seems unrelated to statcache. Im not sure why this error is coming up but it happens when most of the file seems to have been uploaded.

{"time":"2016-02-24 05:02:11,721","cmd":"STOR","full_path":"/mnt/ebs/ftpd/proftpdtestuser/foo.txt","remote_user":"proftpdtestuser","remote_dns":"xxxxx","remote_ip":"xxxxx","local_ip":"172.17.0.2","local_dns":"172.17.0.2","protocol":"sftp","transfer_time":"238.162","transfer_size":"457269574","response_code":"451"} {"time":"2016-02-24 05:02:11,721","cmd":"CLOSE","full_path":"/mnt/ebs/ftpd/proftpdtestuser/foo.txt","remote_user":"proftpdtestuser","remote_dns":"xxxxx","remote_ip":"xxxxx","local_ip":"172.17.0.2","local_dns":"172.17.0.2","protocol":"sftp","transfer_time":"238.162","transfer_size":"457269574","response_code":"3"}

[Castaglia]

Just in case you were using the mod_statcache from this repo (rather than the mod_statcache from the proftpd master branch), I've updated this repo with the same "fixes".

As for whether this is related to mod_statcache, if you test your SFTP upload with "StatCacheEngine off" in the config, and the upload still fails, then we can rule mod_statcache out.

[Castaglia]

After some conversation on IRC, we debugged the issue; it should now be fixed in the latest code in the master branch for the proftpd repo.