search

owncloud / core

:cloud: ownCloud web server core (Files, DAV, etc.)
https://owncloud.com
GNU Affero General Public License v3.0
8.37k stars 2.06k forks source link

SFTP still not fully working: large files upload #9632

Closed MarvinFS closed 9 years ago

MarvinFS commented 10 years ago

Regarding latest #9605 SFTP fixes, i'm trying the latest nightly https://download.owncloud.org/community/daily/owncloud-daily-master.tar.bz2 new DB new everything

Steps to reproduce

  1. upload multiple files
  2. download multiple files
  3. upload large file (1.5GB, 4GB)

    Expected behaviour

  4. files are uploaded
  5. successful download
  6. successful large file upload

    Actual behaviour

  7. working just fine - fixed now, speed is also great with SFTP
  8. also everything ok here!
  9. while the progress bar fills in pretty fast (depends on network speed) the actual files are failed to upload either 1.5GB or 4GB with several errors in logs. files ends up partially uploaded...

    Server configuration

Ubuntu 14LTS

Web server: Apache 2.4

Database: MySQL

PHP version: PHP 5.5.9-1ubuntu4.3

ownCloud version: (see ownCloud admin page) july 15 nightly

Updated from an older ownCloud or fresh install: the very fresh

List of activated apps: only defaults and external storages app

The content of config/config.php:

<?php
$CONFIG = array (
  'instanceid' => 'occ5e8b3c49a',
  'passwordsalt' => '',
  'trusted_domains' =>
  array (
    0 => '10.0.0.37',
  ),
  'datadirectory' => '/var/www/owncloud/data',
  'dbtype' => 'mysql',
  'version' => '7.8.0.0',
  'dbname' => '',
  'dbhost' => 'localhost',
  'dbtableprefix' => '',
  'dbuser' => '
  'dbpassword' => '',
  'installed' => true,
  'ldapIgnoreNamingRules' => false,
  'loglevel' => '0',
  'forcessl' => true,
  'logtimezone' => 'Asia/Yekaterinburg',
  'log_authfailip' => true,
  'log_rotate_size' => 10485760,
  'debug' => true,
  'xframe_restriction' => true,
  'theme' => '',
  'maintenance' => false,
);

Are you using external storage, if yes which one: sftp

Are you using encryption: yes/no no

Client configuration

Browser: firefox, ie11, chrome Operating system: windows 7, 8.1

Logs

ownCloud log (data/owncloud.log)

{"reqId":"53c4b8bacf667","app":"OC\\Files\\Cache\\Scanner","message":"!!! No reuse of etag for 'files_versions\/WebDAV\/test4' !!! \ncache: Array\n(\n    [fileid] => 89\n    [storage] => home::admin\n    [path] => files_versions\/WebDAV\/test4\n    [parent] => 88\n    [name] => test4\n    [mimetype] => httpd\/unix-directory\n    [mimepart] => httpd\n    [size] => 0\n    [mtime] => 1405401121\n    [storage_mtime] => 1405401121\n    [encrypted] => \n    [unencrypted_size] => 0\n    [etag] => 53c4b823195f0\n    [permissions] => 31\n)\n \ndata: Array\n(\n    [mimetype] => httpd\/unix-directory\n    [mtime] => 1405401125\n    [size] => -1\n    [etag] => 53c4b8bacf331\n    [storage_mtime] => 1405401125\n    [permissions] => 31\n    [parent] => 88\n)\n","level":0,"time":"2014-07-15T11:14:34+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/scan.php?force=false&dir=&requesttoken=f06feb01eac59e487e78"}
{"reqId":"53c4bdd02c1f8","app":"OC\\Files\\Cache\\Scanner","message":"!!! No reuse of etag for 'ocloud\/test4' !!! \ncache: Array\n(\n    [fileid] => 27\n    [storage] => sftp::ocloud@marvinfs.convex.ru\/\/\n    [path] => ocloud\/test4\n    [parent] => 17\n    [name] => test4\n    [mimetype] => httpd\/unix-directory\n    [mimepart] => httpd\n    [size] => 80245\n    [mtime] => 1405400671\n    [storage_mtime] => 1405400671\n    [encrypted] => \n    [unencrypted_size] => 0\n    [etag] => 53c4b6104fe13\n    [permissions] => 31\n)\n \ndata: Array\n(\n    [mimetype] => httpd\/unix-directory\n    [mtime] => 1405402518\n    [size] => -1\n    [etag] => 53c4bdd026181\n    [storage_mtime] => 1405402518\n    [permissions] => 31\n    [parent] => 17\n)\n","level":0,"time":"2014-07-15T11:36:16+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/getstoragestats.php?dir=%2FSFTP%2Focloud%2Ftest4"}
{"reqId":"53c4c0e5778f9","app":"webdav client","message":"Not authenticated","level":3,"time":"2014-07-15T11:49:25+06:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/scan.php?force=false&dir=&requesttoken=f06feb01eac59e487e78"}
{"reqId":"53c4c3b70df28","app":"PHP","message":"Connection closed prematurely at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2579","level":3,"time":"2014-07-15T12:01:27+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c4c3b70df28","app":"PHP","message":"Connection closed by server at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2825","level":3,"time":"2014-07-15T12:01:27+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c4c3b70df28","app":"PHP","message":"Expected SSH_FXP_STATUS at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SFTP.php#1590","level":3,"time":"2014-07-15T12:01:27+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c4c3b70df28","app":"PHP","message":"Connection closed prematurely at \/var\/www\/owncloud\/3rdparty\/phpseclib\/phpseclib\/phpseclib\/Net\/SSH2.php#2992","level":3,"time":"2014-07-15T12:01:27+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c4c3b70df28","app":"OC\\Files\\Cache\\Scanner","message":"!!! No reuse of etag for 'ocloud\/test4\/Delo.Rumjanceva.1955.DVDRip_[Youtracker]_by_AVP Studio.avi' !!! \ncache: Array\n(\n    [fileid] => 174\n    [storage] => sftp::ocloud@marvinfs.convex.ru\/\/\n    [path] => ocloud\/test4\/Delo.Rumjanceva.1955.DVDRip_[Youtracker]_by_AVP Studio.avi\n    [parent] => 27\n    [name] => Delo.Rumjanceva.1955.DVDRip_[Youtracker]_by_AVP Studio.avi\n    [mimetype] => \n    [mimepart] => video\n    [size] => 66953216\n    [mtime] => 1405402656\n    [storage_mtime] => 1405402656\n    [encrypted] => \n    [unencrypted_size] => 0\n    [etag] => 53c4bdd0501bf\n    [permissions] => 27\n)\n \ndata: Array\n(\n    [mimetype] => video\/x-msvideo\n    [mtime] => 1405404166\n    [size] => 817790918\n    [etag] => 53c4c3b81f14a\n    [storage_mtime] => 1405404166\n    [permissions] => 27\n    [parent] => 27\n)\n","level":0,"time":"2014-07-15T12:01:28+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c4c3b70df28","app":"core","message":"Exception during mimetype insertion: An exception occurred while executing 'INSERT INTO `oc_mimetypes`(`mimetype`) VALUES(?)':\n\nSQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'video\/x-msvideo' for key 'mimetype_id_index'","level":0,"time":"2014-07-15T12:01:28+06:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}

php options in .htaccess

php_value upload_max_filesize 16G
php_value post_max_size 16G
php_value memory_limit 1024M
php_value output_buffering 16384
php_value mbstring.func_overload 0
php_value max_input_time 36484
php_value max_execution_time 36484
MarvinFS commented 9 years ago

succeeded with 14GB file SFTP upload! it took about 1.5 hours (5-7MB/sec with HPN-SSH) working fine, i guess 0.3.9 solved the issue are we closing the ticket?

bantu commented 9 years ago

@MarvinFS Yeah, let's close it. Closing this in favour of #12115

bantu commented 9 years ago

@MarvinFS Thanks a lot for your patience and testing.