search

owncloud / core

:cloud: ownCloud web server core (Files, DAV, etc.)
https://owncloud.com
GNU Affero General Public License v3.0
8.37k stars 2.06k forks source link

SFTP still not fully working: large files upload #9632

Closed MarvinFS closed 9 years ago

MarvinFS commented 10 years ago

Regarding latest #9605 SFTP fixes, i'm trying the latest nightly https://download.owncloud.org/community/daily/owncloud-daily-master.tar.bz2 new DB new everything

Steps to reproduce

  1. upload multiple files
  2. download multiple files
  3. upload large file (1.5GB, 4GB)

    Expected behaviour

  4. files are uploaded
  5. successful download
  6. successful large file upload

    Actual behaviour

  7. working just fine - fixed now, speed is also great with SFTP
  8. also everything ok here!
  9. while the progress bar fills in pretty fast (depends on network speed) the actual files are failed to upload either 1.5GB or 4GB with several errors in logs. files ends up partially uploaded...

    Server configuration

Ubuntu 14LTS

Web server: Apache 2.4

Database: MySQL

PHP version: PHP 5.5.9-1ubuntu4.3

ownCloud version: (see ownCloud admin page) july 15 nightly

Updated from an older ownCloud or fresh install: the very fresh

List of activated apps: only defaults and external storages app

The content of config/config.php:

<?php
$CONFIG = array (
  'instanceid' => 'occ5e8b3c49a',
  'passwordsalt' => '',
  'trusted_domains' =>
  array (
    0 => '10.0.0.37',
  ),
  'datadirectory' => '/var/www/owncloud/data',
  'dbtype' => 'mysql',
  'version' => '7.8.0.0',
  'dbname' => '',
  'dbhost' => 'localhost',
  'dbtableprefix' => '',
  'dbuser' => '
  'dbpassword' => '',
  'installed' => true,
  'ldapIgnoreNamingRules' => false,
  'loglevel' => '0',
  'forcessl' => true,
  'logtimezone' => 'Asia/Yekaterinburg',
  'log_authfailip' => true,
  'log_rotate_size' => 10485760,
  'debug' => true,
  'xframe_restriction' => true,
  'theme' => '',
  'maintenance' => false,
);

Are you using external storage, if yes which one: sftp

Are you using encryption: yes/no no

Client configuration

Browser: firefox, ie11, chrome Operating system: windows 7, 8.1

Logs

ownCloud log (data/owncloud.log)

{"reqId":"53c4b8bacf667","app":"OC\\Files\\Cache\\Scanner","message":"!!! No reuse of etag for 'files_versions\/WebDAV\/test4' !!! \ncache: Array\n(\n    [fileid] => 89\n    [storage] => home::admin\n    [path] => files_versions\/WebDAV\/test4\n    [parent] => 88\n    [name] => test4\n    [mimetype] => httpd\/unix-directory\n    [mimepart] => httpd\n    [size] => 0\n    [mtime] => 1405401121\n    [storage_mtime] => 1405401121\n    [encrypted] => \n    [unencrypted_size] => 0\n    [etag] => 53c4b823195f0\n    [permissions] => 31\n)\n \ndata: Array\n(\n    [mimetype] => httpd\/unix-directory\n    [mtime] => 1405401125\n    [size] => -1\n    [etag] => 53c4b8bacf331\n    [storage_mtime] => 1405401125\n    [permissions] => 31\n    [parent] => 88\n)\n","level":0,"time":"2014-07-15T11:14:34+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/scan.php?force=false&dir=&requesttoken=f06feb01eac59e487e78"}
{"reqId":"53c4bdd02c1f8","app":"OC\\Files\\Cache\\Scanner","message":"!!! No reuse of etag for 'ocloud\/test4' !!! \ncache: Array\n(\n    [fileid] => 27\n    [storage] => sftp::ocloud@marvinfs.convex.ru\/\/\n    [path] => ocloud\/test4\n    [parent] => 17\n    [name] => test4\n    [mimetype] => httpd\/unix-directory\n    [mimepart] => httpd\n    [size] => 80245\n    [mtime] => 1405400671\n    [storage_mtime] => 1405400671\n    [encrypted] => \n    [unencrypted_size] => 0\n    [etag] => 53c4b6104fe13\n    [permissions] => 31\n)\n \ndata: Array\n(\n    [mimetype] => httpd\/unix-directory\n    [mtime] => 1405402518\n    [size] => -1\n    [etag] => 53c4bdd026181\n    [storage_mtime] => 1405402518\n    [permissions] => 31\n    [parent] => 17\n)\n","level":0,"time":"2014-07-15T11:36:16+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/getstoragestats.php?dir=%2FSFTP%2Focloud%2Ftest4"}
{"reqId":"53c4c0e5778f9","app":"webdav client","message":"Not authenticated","level":3,"time":"2014-07-15T11:49:25+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/scan.php?force=false&dir=&requesttoken=f06feb01eac59e487e78"}
{"reqId":"53c4c3b70df28","app":"PHP","message":"Connection closed prematurely at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2579","level":3,"time":"2014-07-15T12:01:27+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c4c3b70df28","app":"PHP","message":"Connection closed by server at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2825","level":3,"time":"2014-07-15T12:01:27+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c4c3b70df28","app":"PHP","message":"Expected SSH_FXP_STATUS at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SFTP.php#1590","level":3,"time":"2014-07-15T12:01:27+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c4c3b70df28","app":"PHP","message":"Connection closed prematurely at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2992","level":3,"time":"2014-07-15T12:01:27+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c4c3b70df28","app":"OC\\Files\\Cache\\Scanner","message":"!!! No reuse of etag for 'ocloud\/test4\/Delo.Rumjanceva.1955.DVDRip_[Youtracker]_by_AVP Studio.avi' !!! \ncache: Array\n(\n    [fileid] => 174\n    [storage] => sftp::ocloud@marvinfs.convex.ru\/\/\n    [path] => ocloud\/test4\/Delo.Rumjanceva.1955.DVDRip_[Youtracker]_by_AVP Studio.avi\n    [parent] => 27\n    [name] => Delo.Rumjanceva.1955.DVDRip_[Youtracker]_by_AVP Studio.avi\n    [mimetype] => \n    [mimepart] => video\n    [size] => 66953216\n    [mtime] => 1405402656\n    [storage_mtime] => 1405402656\n    [encrypted] => \n    [unencrypted_size] => 0\n    [etag] => 53c4bdd0501bf\n    [permissions] => 27\n)\n \ndata: Array\n(\n    [mimetype] => video\/x-msvideo\n    [mtime] => 1405404166\n    [size] => 817790918\n    [etag] => 53c4c3b81f14a\n    [storage_mtime] => 1405404166\n    [permissions] => 27\n    [parent] => 27\n)\n","level":0,"time":"2014-07-15T12:01:28+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c4c3b70df28","app":"core","message":"Exception during mimetype insertion: An exception occurred while executing 'INSERT INTO `oc_mimetypes`(`mimetype`) VALUES(?)':\n\nSQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'video\/x-msvideo' for key 'mimetype_id_index'","level":0,"time":"2014-07-15T12:01:28+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}

php options in .htaccess

php_value upload_max_filesize 16G
php_value post_max_size 16G
php_value memory_limit 1024M
php_value output_buffering 16384
php_value mbstring.func_overload 0
php_value max_input_time 36484
php_value max_execution_time 36484
PVince81 commented 10 years ago

@bantu look, another case of Expected SSH_FXP_STATUS

@MarvinFS it looks like you're having timeout issues: Connection closed prematurely

MarvinFS commented 10 years ago

@PVince81 yeah but it's a corporate local network - no timeouts here. on my side it looks like it filled the progress bar (time is different, depends on speed and file size) and then it's just hang for a very long time with a full bar doing nothing... then in just stops the upload progress and you left with the partial file on the server... nothing really useful in sftp server's logs (i'm in debug mode)

also the phpseclib's NET_SFTP_RESUME seems useful in case the connection drops and there are partial file on server

bantu commented 10 years ago

@MarvinFS It sounds like you're able to consistently reproduce the issue (possibly due to the large files), which is good. Please replace the contents of the owncloud/3rdparty/phpseclib/phpseclib folder of https://download.owncloud.org/community/daily/owncloud-daily-master.tar.bz2 with the contents of the phpseclib-master folder of https://github.com/phpseclib/phpseclib/archive/master.zip and see whether the issue persists. Thanks.

bantu commented 10 years ago

Possibly related to https://github.com/phpseclib/phpseclib/issues/319#issuecomment-45625669

MarvinFS commented 10 years ago

done! right after apache restart a lot of garbage in logs haven't tried uploading yet

cat ./owncloud.log
{"reqId":"53c500820e612","app":"PHP","message":"Invalid size at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2704","level":3,"time":"2014-07-15T16:20:50+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/list.php?dir=%2F&sort=name&sortdirection=asc"}
{"reqId":"53c500820e612","app":"PHP","message":"Connection closed by server at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#1028","level":3,"time":"2014-07-15T16:20:50+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/list.php?dir=%2F&sort=name&sortdirection=asc"}
{"reqId":"53c500820f24b","app":"PHP","message":"Invalid size at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2704","level":3,"time":"2014-07-15T16:20:50+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/scan.php?force=false&dir=&requesttoken=f06feb01eac59e487e78"}
{"reqId":"53c500820f24b","app":"PHP","message":"Connection closed by server at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#1028","level":3,"time":"2014-07-15T16:20:50+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/scan.php?force=false&dir=&requesttoken=f06feb01eac59e487e78"}
{"reqId":"53c50083e1e5e","app":"PHP","message":"Invalid size at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2704","level":3,"time":"2014-07-15T16:20:51+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/list.php?dir=%2FSFTP&sort=name&sortdirection=asc"}
{"reqId":"53c50083e1e5e","app":"PHP","message":"Connection closed by server at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#1028","level":3,"time":"2014-07-15T16:20:51+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/list.php?dir=%2FSFTP&sort=name&sortdirection=asc"}
{"reqId":"53c50083e1e5e","app":"PHP","message":"Invalid size at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2704","level":3,"time":"2014-07-15T16:20:51+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/list.php?dir=%2FSFTP&sort=name&sortdirection=asc"}
{"reqId":"53c50083e1e5e","app":"PHP","message":"Connection closed by server at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2959","level":3,"time":"2014-07-15T16:20:51+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/list.php?dir=%2FSFTP&sort=name&sortdirection=asc"}
{"reqId":"53c50083e1e5e","app":"PHP","message":"Expected SSH_FXP_ATTRS or SSH_FXP_STATUS at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SFTP.php#1273","level":3,"time":"2014-07-15T16:20:51+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/list.php?dir=%2FSFTP&sort=name&sortdirection=asc"}
{"reqId":"53c500843aec3","app":"PHP","message":"Invalid size at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2704","level":3,"time":"2014-07-15T16:20:52+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/getstoragestats.php?dir=%2FSFTP"}
{"reqId":"53c500843aec3","app":"PHP","message":"Connection closed by server at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#1028","level":3,"time":"2014-07-15T16:20:52+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/getstoragestats.php?dir=%2FSFTP"}
{"reqId":"53c500843aec3","app":"PHP","message":"Invalid size at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2704","level":3,"time":"2014-07-15T16:20:52+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/getstoragestats.php?dir=%2FSFTP"}
{"reqId":"53c500843aec3","app":"PHP","message":"Connection closed by server at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2959","level":3,"time":"2014-07-15T16:20:52+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/getstoragestats.php?dir=%2FSFTP"}
{"reqId":"53c500843aec3","app":"PHP","message":"Expected SSH_FXP_ATTRS or SSH_FXP_STATUS at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SFTP.php#1273","level":3,"time":"2014-07-15T16:20:52+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/getstoragestats.php?dir=%2FSFTP"}
{"reqId":"53c50084e84c5","app":"PHP","message":"Invalid size at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2704","level":3,"time":"2014-07-15T16:20:52+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/list.php?dir=%2FSFTP%2Focloud&sort=name&sortdirection=asc"}
{"reqId":"53c50084e84c5","app":"PHP","message":"Connection closed by server at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#1028","level":3,"time":"2014-07-15T16:20:52+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/list.php?dir=%2FSFTP%2Focloud&sort=name&sortdirection=asc"}
{"reqId":"53c50084e84c5","app":"PHP","message":"Invalid size at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2704","level":3,"time":"2014-07-15T16:20:52+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/list.php?dir=%2FSFTP%2Focloud&sort=name&sortdirection=asc"}
{"reqId":"53c50084e84c5","app":"PHP","message":"Connection closed by server at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2959","level":3,"time":"2014-07-15T16:20:52+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/list.php?dir=%2FSFTP%2Focloud&sort=name&sortdirection=asc"}
{"reqId":"53c50084e84c5","app":"PHP","message":"Expected SSH_FXP_ATTRS or SSH_FXP_STATUS at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SFTP.php#1273","level":3,"time":"2014-07-15T16:20:52+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/list.php?dir=%2FSFTP%2Focloud&sort=name&sortdirection=asc"}
{"reqId":"53c500854f7cd","app":"PHP","message":"Invalid size at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2704","level":3,"time":"2014-07-15T16:20:53+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/getstoragestats.php?dir=%2FSFTP%2Focloud"}
{"reqId":"53c500854f7cd","app":"PHP","message":"Connection closed by server at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#1028","level":3,"time":"2014-07-15T16:20:53+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/getstoragestats.php?dir=%2FSFTP%2Focloud"}
{"reqId":"53c500854f7cd","app":"PHP","message":"Invalid size at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2704","level":3,"time":"2014-07-15T16:20:53+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/getstoragestats.php?dir=%2FSFTP%2Focloud"}
{"reqId":"53c500854f7cd","app":"PHP","message":"Connection closed by server at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2959","level":3,"time":"2014-07-15T16:20:53+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/getstoragestats.php?dir=%2FSFTP%2Focloud"}
{"reqId":"53c500854f7cd","app":"PHP","message":"Expected SSH_FXP_ATTRS or SSH_FXP_STATUS at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SFTP.php#1273","level":3,"time":"2014-07-15T16:20:53+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/getstoragestats.php?dir=%2FSFTP%2Focloud"}
{"reqId":"53c500867fa37","app":"PHP","message":"Invalid size at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2704","level":3,"time":"2014-07-15T16:20:54+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/list.php?dir=%2FSFTP%2Focloud%2Ftest4&sort=name&sortdirection=asc"}
{"reqId":"53c500867fa37","app":"PHP","message":"Connection closed by server at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#1028","level":3,"time":"2014-07-15T16:20:54+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/list.php?dir=%2FSFTP%2Focloud%2Ftest4&sort=name&sortdirection=asc"}
{"reqId":"53c500867fa37","app":"PHP","message":"Invalid size at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2704","level":3,"time":"2014-07-15T16:20:54+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/list.php?dir=%2FSFTP%2Focloud%2Ftest4&sort=name&sortdirection=asc"}
{"reqId":"53c500867fa37","app":"PHP","message":"Connection closed by server at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2959","level":3,"time":"2014-07-15T16:20:54+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/list.php?dir=%2FSFTP%2Focloud%2Ftest4&sort=name&sortdirection=asc"}
{"reqId":"53c500867fa37","app":"PHP","message":"Expected SSH_FXP_ATTRS or SSH_FXP_STATUS at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SFTP.php#1273","level":3,"time":"2014-07-15T16:20:54+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/list.php?dir=%2FSFTP%2Focloud%2Ftest4&sort=name&sortdirection=asc"}
{"reqId":"53c50086d1a2e","app":"PHP","message":"Invalid size at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2704","level":3,"time":"2014-07-15T16:20:54+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/getstoragestats.php?dir=%2FSFTP%2Focloud%2Ftest4"}
{"reqId":"53c50086d1a2e","app":"PHP","message":"Connection closed by server at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#1028","level":3,"time":"2014-07-15T16:20:54+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/getstoragestats.php?dir=%2FSFTP%2Focloud%2Ftest4"}
{"reqId":"53c50086d1a2e","app":"PHP","message":"Invalid size at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2704","level":3,"time":"2014-07-15T16:20:54+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/getstoragestats.php?dir=%2FSFTP%2Focloud%2Ftest4"}
{"reqId":"53c50086d1a2e","app":"PHP","message":"Connection closed by server at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2959","level":3,"time":"2014-07-15T16:20:54+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/getstoragestats.php?dir=%2FSFTP%2Focloud%2Ftest4"}
{"reqId":"53c50086d1a2e","app":"PHP","message":"Expected SSH_FXP_ATTRS or SSH_FXP_STATUS at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SFTP.php#1273","level":3,"time":"2014-07-15T16:20:54+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/getstoragestats.php?dir=%2FSFTP%2Focloud%2Ftest4"}
MarvinFS commented 10 years ago

a lot if bad packet length messages like every second in sshd debug and then reconnects

 Connection from 79.172.0.10 port 32169
 debug1: Client protocol version 2.0; client software version phpseclib_0.3 (gmp)
 debug1: no match: phpseclib_0.3 (gmp)
 debug1: Enabling compatibility mode for protocol 2.0
 debug1: Local version string SSH-2.0-OpenSSH_6.0p1 Debian-4+deb7u1
 debug1: permanently_set_uid: 102/65534 [preauth]
 debug1: list_hostkey_types: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256 [preauth]
 debug1: SSH2_MSG_KEXINIT sent [preauth]
 Bad packet length 3116788250. [preauth]
 Disconnecting: Packet corrupt [preauth]
 debug1: do_cleanup [preauth]
 debug1: monitor_read_log: child log fd closed
 debug1: do_cleanup
 debug1: Killing privsep child 9387
 debug1: Forked child 9388.
 Set /proc/self/oom_score_adj to -1000
 debug1: rexec start in 4 out 4 newsock 4 pipe 6 sock 7
 debug1: inetd sockets after dupping: 3, 3
 Connection from 79.172.0.10 port 48170
bantu commented 10 years ago

@MarvinFS Okay thanks. Will look into that.

MarvinFS commented 10 years ago

the last progress is: on my side the upload bar is full and stays the same for about 20 min (the progress bar filled like may be in 1 min or less - i have a 100mbit here) i was trying to upload a 900MB zip file. nothing new in the logs neither on sshd server nor on owncloud for that waiting time... i quit waiting... the last logs in owncloud were

{"reqId":"53c5017c41946","app":"PHP","message":"Connection closed by server at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2959","level":3,"time":"2014-07-15T16:25:00+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c5017c41946","app":"PHP","message":"Expected SSH_FXP_ATTRS or SSH_FXP_STATUS at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SFTP.php#1273","level":3,"time":"2014-07-15T16:25:00+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c5017c41946","app":"PHP","message":"Invalid size at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2704","level":3,"time":"2014-07-15T16:25:00+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c5017c41946","app":"PHP","message":"Connection closed by server at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#1028","level":3,"time":"2014-07-15T16:25:00+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c5017c41946","app":"PHP","message":"Invalid size at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2704","level":3,"time":"2014-07-15T16:25:00+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c5017c41946","app":"PHP","message":"Connection closed by server at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2959","level":3,"time":"2014-07-15T16:25:00+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c5017c41946","app":"PHP","message":"Expected SSH_FXP_ATTRS or SSH_FXP_STATUS at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SFTP.php#1273","level":3,"time":"2014-07-15T16:25:00+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c5017c41946","app":"PHP","message":"Invalid size at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2704","level":3,"time":"2014-07-15T16:25:00+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c5017c41946","app":"PHP","message":"Connection closed by server at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2959","level":3,"time":"2014-07-15T16:25:00+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c5017c41946","app":"PHP","message":"Expected SSH_FXP_ATTRS or SSH_FXP_STATUS at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SFTP.php#1273","level":3,"time":"2014-07-15T16:25:00+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
bantu commented 10 years ago

@MarvinFS A patch has been accepted into phpseclib potentially fixing the latest issues you were having. Please retry with the latest phpseclib (same instructions as in https://github.com/owncloud/core/issues/9632#issuecomment-49013207). Thanks.

MarvinFS commented 10 years ago

@bantu Dear Andreas, thanks for the fixes, sorry for the delay - was busy! errors changed in logs a bit, but still fails with the same symptoms: i was uploading a 1GB file it filled the progress bar in seconds, after that was just hanging, doing nothing no errors on both sides... for about 35 mins (again 100mb\sec channel) then failed, leaving partially uploaded file (see below the log) also i've measured the bandwidth usage on a server during upload - it was about 25-30KB\sec! which is really ODD! caz if i use any other method of file cope yo SFTP the speed is 6-9MB\sec in the same environment!

the logs from the server start till the end of upload

{"reqId":"53cdcb34f1ac7","app":"core","message":"OC_Image::loadFromFileHandle(): Trying","level":0,"time":"2014-07-22T08:23:48+06:00","method":"GET","url":"\/owncloud\/index.php\/core\/preview.png?file=%2FSFTP%2Focloud%2Fpustota.jpg&c=53c4b5fc45d53&x=36&y=36&forceIcon=0"}
{"reqId":"53cdcb34f1ac7","app":"core","message":"OC_Image->fixOrientation() No readable file path set.","level":0,"time":"2014-07-22T08:23:48+06:00","method":"GET","url":"\/owncloud\/index.php\/core\/preview.png?file=%2FSFTP%2Focloud%2Fpustota.jpg&c=53c4b5fc45d53&x=36&y=36&forceIcon=0"}
{"reqId":"53cdcb34f1ac7","app":"core","message":"OC_Image->fixOrientation() Orientation: -1","level":0,"time":"2014-07-22T08:23:48+06:00","method":"GET","url":"\/owncloud\/index.php\/core\/preview.png?file=%2FSFTP%2Focloud%2Fpustota.jpg&c=53c4b5fc45d53&x=36&y=36&forceIcon=0"}
{"reqId":"53cdcb385209f","app":"OC\\Files\\Cache\\Scanner","message":"!!! No reuse of etag for 'ocloud\/test4' !!! \ncache: Array\n(\n    [fileid] => 27\n    [storage] => sftp::ocloud@marvinfs.convex.ru\/\/\n    [path] => ocloud\/test4\n    [parent] => 17\n    [name] => test4\n    [mimetype] => httpd\/unix-directory\n    [mimepart] => httpd\n    [size] => 1636247838\n    [mtime] => 1405404708\n    [storage_mtime] => 1405404708\n    [encrypted] => \n    [unencrypted_size] => 0\n    [etag] => 53c4cc49abc0e\n    [permissions] => 31\n)\n \ndata: Array\n(\n    [mimetype] => httpd\/unix-directory\n    [mtime] => 1405419982\n    [size] => -1\n    [etag] => 53cdcb3851d91\n    [storage_mtime] => 1405419982\n    [permissions] => 31\n    [parent] => 17\n)\n","level":0,"time":"2014-07-22T08:23:52+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/list.php?dir=%2FSFTP%2Focloud%2Ftest4&sort=name&sortdirection=asc"}
{"reqId":"53cdcf881c7d8","app":"OC\\Files\\Cache\\Scanner","message":"!!! Path 'thumbnails\/655' is not readable !!!","level":0,"time":"2014-07-22T08:42:16+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files_versions\/preview?file=%2FSFTP%2Focloud%2Ftest4%2FDRP13-R399-DVD-Final.iso&version=1405406286"}
{"reqId":"53cdcf881c7d8","app":"core","message":"Generating preview for \"\/SFTP\/ocloud\/test4\/DRP13-R399-DVD-Final.iso.v1405406286\" with \"OC\\Preview\\Unknown\"","level":0,"time":"2014-07-22T08:42:16+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files_versions\/preview?file=%2FSFTP%2Focloud%2Ftest4%2FDRP13-R399-DVD-Final.iso&version=1405406286"}
{"reqId":"53cdcf881c7d8","app":"core","message":"OC_Image->loadFromBase64, couldn't load","level":0,"time":"2014-07-22T08:42:16+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files_versions\/preview?file=%2FSFTP%2Focloud%2Ftest4%2FDRP13-R399-DVD-Final.iso&version=1405406286"}
{"reqId":"53cdcf881c7d8","app":"core","message":"OC_Image->fixOrientation() Orientation: -1","level":0,"time":"2014-07-22T08:42:16+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files_versions\/preview?file=%2FSFTP%2Focloud%2Ftest4%2FDRP13-R399-DVD-Final.iso&version=1405406286"}
{"reqId":"53cdd1f40687c","app":"PHP","message":"Connection closed prematurely at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#3127","level":3,"time":"2014-07-22T08:52:36+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53cdd1f40687c","app":"PHP","message":"Undefined variable: fp at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SFTP.php#1874","level":3,"time":"2014-07-22T08:52:36+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53cdd1f40687c","app":"PHP","message":"fclose() expects parameter 1 to be resource, null given at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SFTP.php#1874","level":3,"time":"2014-07-22T08:52:36+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53cdd1f40687c","app":"OC\\Files\\Cache\\Scanner","message":"!!! Path 'ocloud\/test4\/linuxmint-15-cinnamon-dvd-32bit.iso' is not readable !!!","level":0,"time":"2014-07-22T08:52:36+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53cdd1f40687c","app":"OC\\Files\\Cache\\Scanner","message":"!!! No reuse of etag for 'ocloud\/test4' !!! \ncache: Array\n(\n    [fileid] => 27\n    [storage] => sftp::ocloud@marvinfs.convex.ru\/\/\n    [path] => ocloud\/test4\n    [parent] => 17\n    [name] => test4\n    [mimetype] => httpd\/unix-directory\n    [mimepart] => httpd\n    [size] => 2455009537\n    [mtime] => -1\n    [storage_mtime] => -1\n    [encrypted] => \n    [unencrypted_size] => 0\n    [etag] => 53cdd1f46e605\n    [permissions] => 31\n)\n \ndata: Array\n(\n    [mimetype] => httpd\/unix-directory\n    [mtime] => 1405995992\n    [size] => -1\n    [etag] => 53cdd1f4aa232\n    [storage_mtime] => 1405995992\n    [permissions] => 31\n    [parent] => 17\n)\n","level":0,"time":"2014-07-22T08:52:36+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}

the other interesting issue that the partial fines uploaded to the STFP server are the very same size... 818761699, of cause the source files are different in sizes! (other partial files in that listing were uploaded previously with different settings, that's why the size differs a little bit i think)

root@marvinfs:/opt/ocloud/test4# ls -la
total 3197072
drwxr-xr-x 2 ocloud ocloud      4096 Jul 22 08:26 .
drwxrwxrwx 8 root   root        4096 Jul 15 11:04 ..
-rw-r--r-- 1 ocloud ocloud 817790918 Jul 15 12:02 Delo.Rumjanceva.1955.DVDRip_[Youtracker]_by_AVP Studio.avi
-rw-r--r-- 1 ocloud ocloud 818376675 Jul 15 12:38 DRP13-R399-DVD-Final.iso
-rw-r--r-- 1 ocloud ocloud     11574 Jul 15 11:04 gover_server.ovf
-rw-r--r-- 1 ocloud ocloud     12350 Jul 15 11:04 gover_server.ovf_
-rw-r--r-- 1 ocloud ocloud 818761699 Jul 22 08:49 linuxmint-15-cinnamon-dvd-32bit.iso
-rw-r--r-- 1 ocloud ocloud 818761699 Jul 15 16:51 Video2Brain - Getting Started with Themes in WordPress.zip
-rw-r--r-- 1 ocloud ocloud     56321 Jul 15 11:04 Шаблон письма.docx
root@marvinfs:/opt/ocloud/test4#

Regards, Vladimir

bantu commented 10 years ago

@MarvinFS Please note that phpseclib SFTP does currently not support files larger than 4 GiB.

The following sounds like an issue:

{"reqId":"53cdd1f40687c","app":"PHP","message":"Connection closed prematurely at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#3127","level":3,"time":"2014-07-22T08:52:36+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53cdd1f40687c","app":"PHP","message":"Undefined variable: fp at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SFTP.php#1874","level":3,"time":"2014-07-22T08:52:36+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53cdd1f40687c","app":"PHP","message":"fclose() expects parameter 1 to be resource, null given at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SFTP.php#1874","level":3,"time":"2014-07-22T08:52:36+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
MarvinFS commented 10 years ago

@bantu too bad! 4GB is nothing in modern world! :( but i was definitely trying 1gig file

bantu commented 10 years ago

@MarvinFS phpseclib will likely add support for files larger than 4 GiB. But since that is not a current feature, please try only with files smaller than 4 GiB. Thanks for testing, btw. :-)

MarvinFS commented 10 years ago

Thanks for info! i'm eager to help improving such a nice product, despite that i can't provide any development, still i'm pretty good in beta testing everything, due to rich IT experience! :) Anyway i'm so much waiting for OC to work reliable with different external storages (btw webdav will be the second issue - i will update my opened issue after sftp), because i'm dead tired of evil google corp and wanna be sure about my privacy with personal cloud

MarvinFS commented 10 years ago

@bantu can we expect any progress here?

bantu commented 10 years ago

@MarvinFS I am sorry this is taking so long. Could you give https://github.com/owncloud/core/issues/9632#issuecomment-49013207 another try. There have been some adjustments in phpseclib. Thanks.

MarvinFS commented 10 years ago

@bantu Dear Andreas, thank you for the answer! i've just finished testing a new ver with clean setup - seem to work now, at least 950MB iso file uploaded successfully to remote SFTP server, i wanna try bigger files also, but the issue is speed now - it became just terrible! it took about 40-50 mins to upload a single 950MB file on a 100Mbit\sec network (the real net bandwidth is about 8-9MB\sec) here is the log

{"reqId":"5407f28db8bfe","app":"OC\\Files\\Cache\\Scanner","message":"!!! Path '' is not accessible or present !!!","level":0,"time":"2014-09-04T11:03:09+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/list.php?dir=%2F&sort=name&sortdirection=asc"}
{"reqId":"5407f2993176f","app":"webdav","message":"Sabre\\DAV\\Exception\\NotAuthenticated: No basic authentication headers were found","level":0,"time":"2014-09-04T11:03:21+06:00","method":"PROPFIND","url":"\/owncloud\/remote.php\/webdav\/"}
{"reqId":"5407f2bc870e8","app":"OC\\Files\\Cache\\Scanner","message":"!!! Path '' is not accessible or present !!!","level":0,"time":"2014-09-04T11:03:56+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/list.php?dir=%2F&sort=name&sortdirection=asc"}
{"reqId":"5407f2cfa9db7","app":"OC\\Files\\Cache\\Scanner","message":"!!! Path '' is not accessible or present !!!","level":0,"time":"2014-09-04T11:04:15+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/list.php?dir=%2F&sort=name&sortdirection=asc"}
{"reqId":"5407f2d332256","app":"webdav","message":"Sabre\\DAV\\Exception\\NotAuthenticated: No basic authentication headers were found","level":0,"time":"2014-09-04T11:04:19+06:00","method":"PROPFIND","url":"\/owncloud\/remote.php\/webdav\/"}
{"reqId":"5407f2df54416","app":"webdav","message":"Sabre\\DAV\\Exception\\NotAuthenticated: No basic authentication headers were found","level":0,"time":"2014-09-04T11:04:31+06:00","method":"PROPFIND","url":"\/owncloud\/remote.php\/webdav\/"}
{"reqId":"5407f2f32c4ca","app":"OC\\Files\\Cache\\Scanner","message":"!!! Path '' is not accessible or present !!!","level":0,"time":"2014-09-04T11:04:51+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/list.php?dir=%2F&sort=name&sortdirection=asc"}
{"reqId":"5407f30043377","app":"OC\\Files\\Cache\\Scanner","message":"!!! Path '' is not accessible or present !!!","level":0,"time":"2014-09-04T11:05:04+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/list.php?dir=%2F&sort=name&sortdirection=asc"}
{"reqId":"5407f30ae9ae9","app":"OC\\Files\\Cache\\Scanner","message":"!!! Path '' is not accessible or present !!!","level":0,"time":"2014-09-04T11:05:14+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/list.php?dir=%2F&sort=name&sortdirection=asc"}
{"reqId":"5407f311839eb","app":"webdav","message":"Sabre\\DAV\\Exception\\NotAuthenticated: No basic authentication headers were found","level":0,"time":"2014-09-04T11:05:21+06:00","method":"PROPFIND","url":"\/owncloud\/remote.php\/webdav\/"}
{"reqId":"5407f31b2a686","app":"webdav","message":"Sabre\\DAV\\Exception\\NotAuthenticated: No basic authentication headers were found","level":0,"time":"2014-09-04T11:05:31+06:00","method":"PROPFIND","url":"\/owncloud\/remote.php\/webdav\/"}
{"reqId":"5407f323d5a57","app":"webdav","message":"Sabre\\DAV\\Exception\\NotAuthenticated: No basic authentication headers were found","level":0,"time":"2014-09-04T11:05:39+06:00","method":"PROPFIND","url":"\/owncloud\/remote.php\/webdav\/"}
{"reqId":"5407f3402c326","app":"webdav","message":"Sabre\\DAV\\Exception\\NotAuthenticated: No basic authentication headers were found","level":0,"time":"2014-09-04T11:06:08+06:00","method":"PROPFIND","url":"\/owncloud\/remote.php\/webdav\/"}
{"reqId":"5407f347cb015","app":"OC\\Files\\Cache\\Scanner","message":"!!! Path 'thumbnails\/20' is not accessible or present !!!","level":0,"time":"2014-09-04T11:06:15+06:00","method":"GET","url":"\/owncloud\/index.php\/core\/preview.png?file=%2FSFTP%2Focloud%2Fpustota.jpg&c=5407f3475ccbb&x=36&y=36&forceIcon=0"}
{"reqId":"5407f347cb015","app":"core","message":"Generating preview for \"\/SFTP\/ocloud\/pustota.jpg\" with \"OC\\Preview\\Image\"","level":0,"time":"2014-09-04T11:06:15+06:00","method":"GET","url":"\/owncloud\/index.php\/core\/preview.png?file=%2FSFTP%2Focloud%2Fpustota.jpg&c=5407f3475ccbb&x=36&y=36&forceIcon=0"}
{"reqId":"5407f347cb015","app":"PHP","message":"file_get_contents(\/var\/www\/owncloud\/data\/admin\/files\/tmp\/497e0a422ad7883425c9e1e3394ed06d.jpg): failed to open stream: No such file or directory at \/var\/www\/owncloud\/lib\/private\/files\/storage\/local.php#156","level":3,"time":"2014-09-04T11:06:15+06:00","method":"GET","url":"\/owncloud\/index.php\/core\/preview.png?file=%2FSFTP%2Focloud%2Fpustota.jpg&c=5407f3475ccbb&x=36&y=36&forceIcon=0"}
{"reqId":"5407f347cb015","app":"PHP","message":"imagecreatefromstring(): Empty string or invalid image at \/var\/www\/owncloud\/lib\/private\/image.php#525","level":3,"time":"2014-09-04T11:06:15+06:00","method":"GET","url":"\/owncloud\/index.php\/core\/preview.png?file=%2FSFTP%2Focloud%2Fpustota.jpg&c=5407f3475ccbb&x=36&y=36&forceIcon=0"}
{"reqId":"5407f347cb015","app":"core","message":"OC_Image->loadFromFile, Default","level":0,"time":"2014-09-04T11:06:15+06:00","method":"GET","url":"\/owncloud\/index.php\/core\/preview.png?file=%2FSFTP%2Focloud%2Fpustota.jpg&c=5407f3475ccbb&x=36&y=36&forceIcon=0"}
{"reqId":"5407f347cb015","app":"core","message":"Generating preview for \"\/SFTP\/ocloud\/pustota.jpg\" with \"OC\\Preview\\Unknown\"","level":0,"time":"2014-09-04T11:06:15+06:00","method":"GET","url":"\/owncloud\/index.php\/core\/preview.png?file=%2FSFTP%2Focloud%2Fpustota.jpg&c=5407f3475ccbb&x=36&y=36&forceIcon=0"}
{"reqId":"5407f347cb015","app":"core","message":"OC_Image->loadFromBase64, couldn't load","level":0,"time":"2014-09-04T11:06:15+06:00","method":"GET","url":"\/owncloud\/index.php\/core\/preview.png?file=%2FSFTP%2Focloud%2Fpustota.jpg&c=5407f3475ccbb&x=36&y=36&forceIcon=0"}
{"reqId":"5407f347cb015","app":"core","message":"OC_Image->fixOrientation() Orientation: -1","level":0,"time":"2014-09-04T11:06:15+06:00","method":"GET","url":"\/owncloud\/index.php\/core\/preview.png?file=%2FSFTP%2Focloud%2Fpustota.jpg&c=5407f3475ccbb&x=36&y=36&forceIcon=0"}
{"reqId":"5407f4707ffd6","app":"OC\\Files\\Cache\\Scanner","message":"!!! No reuse of etag for 'ocloud' !!! \ncache: Array\n(\n    [fileid] => 15\n    [storage] => sftp::ocloud@marvinfs.convex.ru\/\/\n    [path] => ocloud\n    [parent] => 14\n    [name] => ocloud\n    [mimetype] => httpd\/unix-directory\n    [mimepart] => httpd\n    [size] => -1\n    [mtime] => 1405400658\n    [storage_mtime] => 1405400658\n    [encrypted] => \n    [unencrypted_size] => 0\n    [etag] => 5407f345f209e\n    [permissions] => 31\n)\n \ndata: Array\n(\n    [mimetype] => httpd\/unix-directory\n    [mtime] => 1409807483\n    [size] => -1\n    [etag] => 5407f4707fb8a\n    [storage_mtime] => 1409807483\n    [permissions] => 31\n)\n","level":0,"time":"2014-09-04T11:11:12+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/getstoragestats.php?dir=%2FSFTP%2Focloud"}
{"reqId":"5407fa0ddafd0","app":"OC\\Files\\Cache\\Scanner","message":"!!! No reuse of etag for 'ocloud\/linuxmint-15-cinnamon-dvd-32bit.iso' !!! \ncache: Array\n(\n    [fileid] => 26\n    [storage] => sftp::ocloud@marvinfs.convex.ru\/\/\n    [path] => ocloud\/linuxmint-15-cinnamon-dvd-32bit.iso\n    [parent] => 15\n    [name] => linuxmint-15-cinnamon-dvd-32bit.iso\n    [mimetype] => application\/octet-stream\n    [mimepart] => application\n    [size] => 116727750\n    [mtime] => 1409807679\n    [storage_mtime] => 1409807679\n    [encrypted] => \n    [unencrypted_size] => 0\n    [etag] => 5407f4708785b\n    [permissions] => 27\n)\n \ndata: Array\n(\n    [mimetype] => application\/octet-stream\n    [mtime] => 1409809117\n    [size] => 973078528\n    [etag] => 5407fa0ddaabb\n    [storage_mtime] => 1409809117\n    [permissions] => 27\n    [parent] => 15\n)\n","level":0,"time":"2014-09-04T11:35:09+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"5407fa0ddafd0","app":"PHP","message":"call_user_func() expects parameter 1 to be a valid callback, class 'OCA\\Gallery\\Thumbnail' not found at \/var\/www\/owncloud\/lib\/private\/hook.php#78","level":3,"time":"2014-09-04T11:35:10+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
bantu commented 10 years ago

@MarvinFS Do you have the mcrypt PHP extension installed and enabled?

MarvinFS commented 10 years ago

@bantu as far as i can tell it is: (from the owncloud server)

PHP Version 5.5.9-1ubuntu4.3
System  Linux ubuntu14 3.13.0-30-generic #55-Ubuntu SMP Fri Jul 4 21:40:53 UTC 2014 x86_64
Build Date  Jul 7 2014 16:33:02
Server API  Apache 2.0 Handler 
Registered Stream Filters   zlib.*, bzip2.*, convert.iconv.*, string.rot13, string.toupper, string.tolower, string.strip_tags, convert.*, consumed, dechunk, mcrypt.*, mdecrypt.*

mcrypt support  enabled
mcrypt_filter support   enabled
Version     2.5.8
Api No  20021217
Supported ciphers   cast-128 gost rijndael-128 twofish arcfour cast-256 loki97 rijndael-192 saferplus wake blowfish-compat des rijndael-256 serpent xtea blowfish enigma rc2 tripledes
Supported modes     cbc cfb ctr ecb ncfb nofb ofb stream
MTRichards commented 10 years ago

Need to put in a release note about the 4GB file size limitation mentioned above.

bantu commented 10 years ago

@MTRichards Good catch.

MarvinFS commented 10 years ago

@bantu what should i do now? i guess this is a totally phpseclib issue... should i move in the phpseclib tree?

PVince81 commented 10 years ago

Even if phpseclib gets fixed, we still need a ticket here as a reminder to upgrade to the next fixed version of phpseclib.

carlaschroder commented 10 years ago

added to release notes.

DeepDiver1975 commented 9 years ago

@bantu is there anything more we can do about this issue? Did we already update phpseclib properly? THX

nickvergessen commented 9 years ago

Well we are on 0.3.6 (Feb 2014) and current is 0.3.9 (Nov 2014)

DeepDiver1975 commented 9 years ago

on stable7 we are hopelessly out dated with respect to your 3rdparty dependencies: https://www.versioneye.com/user/projects/54bf83f36c0035c592000048

master/oc8 doesn't look that bad: https://www.versioneye.com/user/projects/54a548e697427553100000b9

So from my understanding we should update phpseclib for stable7 @bantu can I ask you to update this in the 3rdparty repo stable7 and apply proper testing? Many thanks

DeepDiver1975 commented 9 years ago

@bantu ping

bantu commented 9 years ago

@DeepDiver1975 Not sure there is anything that can be done about this ticket in particular, but if PHP 5.6.1+ compatibility is desired (I think it is), we should upgrade to phpseclib 0.3.9 for PHP 5.6 improvements. Besides that, the improvements to Net_SSH2 are nice to have for our SFTP support. Also see https://github.com/phpseclib/phpseclib/blob/master/CHANGELOG.md

bantu commented 9 years ago

@MarvinFS Could you please tell us a bit more about your SFTP server? Which operating system is it running. Is it running OpenSSH? Which version?

nickvergessen commented 9 years ago

@bantu you dont think updating seclib from 0.3.6 to 0.3.9 will help?

bantu commented 9 years ago

@nickvergessen I don't know. stable7 should receive a phpseclib upgrade for PHP 5.6 compatibility. @MarvinFS here claims to be using master, which already has a newer version of phpseclib (at least right now).

DeepDiver1975 commented 9 years ago

just to get a clear understanding of the current situation: on master we can upload files to sftp with a max size of ??? on stable7 we can upload files to sftp with a max size of ???

nickvergessen commented 9 years ago

@bantu master uses 0.3.6 I just verified it in the repo

bantu commented 9 years ago

@nickvergessen Then https://www.versioneye.com/user/projects/54a548e697427553100000b9 lies.

nickvergessen commented 9 years ago

@bantu appearantly it does, maybe it's confused by the "*" requirement we have. Current 3rdparty master: https://github.com/owncloud/3rdparty/blob/master/composer.lock#L1278-L1279 Current commit of oc master: https://github.com/owncloud/3rdparty/blob/a32d3924bd0012a5410fff4666131cbdfdec2001/composer.lock#L1278-L1279

nickvergessen commented 9 years ago

Also:

nickv@nickv-lt-mint ~/ownCloud/Master/3rdparty $ ~/Tools/composer.phar update phpseclib/phpseclib
Loading composer repositories with package information
Updating dependencies (including require-dev)
  - Removing phpseclib/phpseclib (0.3.6)
  - Installing phpseclib/phpseclib (0.3.9)
    Downloading: 100%         

Writing lock file
Generating autoload files
bantu commented 9 years ago

@DeepDiver1975 I've not actually interpreted this ticket to be about addressing any potentially remaining filesize limitations. Rather, I've always focussed on the "Expected SSH_FXP_STATUS" / "Connection closed prematurely" errors, which were encountered when uploading large files (say 1.8 GiB). I've not been able to reproduce those errors so far.

The version of phpseclib stable7 is using does not clamp filesizes to 4 GiB any more, as per https://github.com/owncloud/core/issues/11432#issuecomment-59217651. As such, there should be no practical filesize limit on 64bit PHP environments.

On 32bit PHP the situation is a bit different. A call to $sftp->put() will result in filesize() being called by phpseclib, which as we know might lie. As such, the limit on 32 bit PHP is 2 GiB. A solution to this part is to extract the filesize tricks into its own library and also use that in phpseclib. I've been meaning to do that.

As per nickvergessen, stable7 and master currently use the same version.

bantu commented 9 years ago

@DeepDiver1975 Additionally, I believe the web interface can not upload files larger than 2 GiB at all on PHP versions below 5.6 (see http://cweiske.de/tagebuch/php-large-file-uploads.htm). And chunking still needs to be investigated on 32bit platforms.

MarvinFS commented 9 years ago

Guys, i'm still greatly concerned about this issue, thank you very much for not letting it go, but i'm on a ski resort right now, will be back on sunday evening and from monday's morning i'll be able to test any new builds (if any) and will provide all the info requested, ok?

MorrisJobke commented 9 years ago

@DeepDiver1975 From IRC:

bantu: move to next version I'd say. I haven't been able to reproduce this problem in particular.

Move to 8.1 then

MarvinFS commented 9 years ago

@bantu

Colleagues, what particular test build i may use to see if these issues are still exists? I'm ready to test 'em with latest debian and ubuntu LTS including sftp transfer and large files upload

MarvinFS commented 9 years ago

@bantu i've just tried again following your previous instructions (replace the contents of the owncloud/3rdparty/phpseclib/phpseclib folder of https://download.owncloud.org/community/daily/owncloud-daily-master.tar.bz2 with the contents of the phpseclib-master folder of https://github.com/phpseclib/phpseclib/archive/master.zip which is 0.3.9 apparently ) and in doesn't work anymore - ocloud is not starting - giving me

{"reqId":"dmJdtH1soUKmfzuP3l3E","remoteAddr":"192.168.1.101","app":"PHP","message":"Class 'phpseclib\Crypt\Rijndael' not found at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Crypt\/AES.php#58","level":3,"time":"2015-02-10T22:43:59+05:00"}

with original phpseclib from latest ocloud daily master the behavior is still the same - uploads about 818MB to the SFTP server and disconnects leaving partially uploaded file

bantu commented 9 years ago

@MarvinFS Branches in phpseclib have been reorganised. You will have to use https://github.com/phpseclib/phpseclib/archive/1.0.zip instead.

MarvinFS commented 9 years ago

@bantu ok i finished the testing with new 0.3.9 branch (all the testing were over public internet) source server: at office, vmware cluster, ubuntu 14.04 LTS: ocloud daily master clean + 0.3.9. phpaseclib (external storage support app enabled)

dest server: at home, debian 7.8

upload 2.5GB file from source to dest by SFTP external storage - SUCCESS speed 1.3-1.6MB/sec upload 4.5GB file from source to dest by FTP external storage - SUCCESS speed 11.2 - 11.5 MB\sec

  1. the low speed for SFTP upload is due to speed limitations for openssh and i failed to find any suitable repo for HPN-SSH for Debian (found only for Ubuntu)
  2. the FTP upload is just fine, although it was good working in the past...
  3. 1st issue: when you start to upload the file to external storage in seems that first file is uploading to the ocloud server itself (the progress bar starts to fill from 0 to full), during this time the dest server is IDLE!!! only after the progress bar fills to full actual upload begins to the dest. server. During the actual upload the progress bar stays full until the upload is finished. But as it's an external storage the actual file is not held on ocloud server itself - so it's not clear what it's doing during initial progress bar fill...
  4. 2nd issue: for the FTP upload the file on dest server is complete and full, but inside ocloud interface, the file size shows as 394MB only... not sure why... refreshing the folder don't help... (screenshot https://yadi.sk/i/w__K8zYQebN98 )

in the logfiles nothing useful - seems everything fine.

Whay else can i do?

MarvinFS commented 9 years ago

you will be laughing but now if i try to download that uploaded file from FTP server though ocloud interface it downloads ONLY the ocloud size which is 359.4 MB, if i try to download the same file from remote FTP server directly is downloads the whole 4.5 gigs... md5 matches - i checked it

PVince81 commented 9 years ago

@MarvinFS if you uploaded through the Web UI that goes through POST, in which case PHP will first store a temp file before giving OC access to it. This might explain the delay. If you try with WebDAV it should be streamed directly.

MarvinFS commented 9 years ago

@PVince81 Got it! makes sense! but the ocloud interface FTP file size issue after upload still remains - should i open other tread? it's not related to phpseclib due to this is plain FTP.

now will try the maximum size upload through SFTP using HPN-SSH on ubuntu - such low transfer speed of standard openssh drives me nuts

PVince81 commented 9 years ago

Yes, please make another ticket for FTP. Thanks.

MarvinFS commented 9 years ago

uploaded 9.2GB file to SFTP external storage just fine! success at last! next will try 16G. i used HPN-SSH as a SFTP external target server - performance improved DRASTICALLY - 5-10 times! (from 1.5 MB\sec to 5-11MB\sec) i used this manual for Ubuntu https://spoutcraft.org/threads/blazing-fast-sftp-ssh-transfer.7682/ although i failed to find a ready repo for Debian