adrienafl commented 7 years ago

Hi ! Just having this error : I tried to fix perms with the strong perm script. I also tried to use update perms but it didn't change anything.

Thank you

Steps to reproduce

Run sudo -u www-data php occ files:scan --all

Expected behaviour

Scan should complete

Actual behaviour

Exception during scan: Undefined index: size
#0 /var/www/owncloud/lib/private/Files/Cache/Scanner.php(416): OCA\Files\Command\Scan->exceptionErrorHandler(8, 'Undefined index...', '/var/www/ownclo...', 416, Array)
#1 /var/www/owncloud/lib/private/Files/Cache/Scanner.php(377): OC\Files\Cache\Scanner->handleChildren('files/Backup/PI...', true, 3, '57663', true, 119080535)
#2 /var/www/owncloud/lib/private/Files/Cache/Scanner.php(380): OC\Files\Cache\Scanner->scanChildren('files/Backup/PI...', true, 3, '57663', true)
#3 /var/www/owncloud/lib/private/Files/Cache/Scanner.php(380): OC\Files\Cache\Scanner->scanChildren('files/Backup/PI...', true, 3, '512', true)
#4 /var/www/owncloud/lib/private/Files/Cache/Scanner.php(380): OC\Files\Cache\Scanner->scanChildren('files/Backup/PI...', true, 3, '508', true)
#5 /var/www/owncloud/lib/private/Files/Cache/Scanner.php(380): OC\Files\Cache\Scanner->scanChildren('files/Backup', true, 3, '169', true)
#6 /var/www/owncloud/lib/private/Files/Cache/Scanner.php(311): OC\Files\Cache\Scanner->scanChildren('files', true, 3, 2, true)
#7 /var/www/owncloud/lib/private/Files/Utils/Scanner.php(235): OC\Files\Cache\Scanner->scan('files', true, 3)
#8 /var/www/owncloud/apps/files/lib/Command/Scan.php(159): OC\Files\Utils\Scanner->scan('/xxx/files')
#9 /var/www/owncloud/apps/files/lib/Command/Scan.php(228): OCA\Files\Command\Scan->scanFiles('xxx', '/xxx/files', false, Object(Symfony\Component\Console\Output\ConsoleOutput), false)
#10 /var/www/owncloud/lib/composer/symfony/console/Command/Command.php(262): OCA\Files\Command\Scan->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#11 /var/www/owncloud/core/Command/Base.php(158): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#12 /var/www/owncloud/lib/composer/symfony/console/Application.php(826): OC\Core\Command\Base->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 /var/www/owncloud/lib/composer/symfony/console/Application.php(189): Symfony\Component\Console\Application->doRunCommand(Object(OCA\Files\Command\Scan), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /var/www/owncloud/lib/composer/symfony/console/Application.php(120): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /var/www/owncloud/lib/private/Console/Application.php(160): Symfony\Component\Console\Application->run(NULL, NULL)
#16 /var/www/owncloud/console.php(99): OC\Console\Application->run()
#17 /var/www/owncloud/occ(11): require_once('/var/www/ownclo...')
#18 {main}

Server configuration

Operating system: Debian 8 Web server: Apache 2.4.10 Database: mysql Ver 14.14 Distrib 5.5.55, for debian-linux-gnu (x86_64) using readline 6.3 PHP version: PHP 5.6.30-0+deb8u1 ownCloud version: (see ownCloud admin page) 10.0.2.1 Updated from an older ownCloud or fresh install: Upgrade from OC 9 (latest one) Where did you install ownCloud from: Package Signing status (ownCloud 9.0 and above):

No error

The content of config/config.php:

{
    "system": {
        "updatechecker": false,
        "asset-pipeline.enabled": true,
        "instanceid": "ocln9v0ykp6k",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "owncloud.xxx.com"
        ],
        "datadirectory": "\/home\/owncloud",
        "overwrite.cli.url": "http:\/\/owncloud.xxx.com",
        "dbtype": "mysql",
        "version": "10.0.2.1",
        "dbname": "ocdb",
        "dbhost": "localhost",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "logtimezone": "UTC",
        "installed": true,
        "maintenance": false,
        "theme": "",
        "loglevel": 2,
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "memcache.local": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "localhost",
            "port": 6379
        }
    }
}

List of activated apps:

Enabled:

activity: 2.3.4
comments: 0.3.0
configreport: 0.1.1
dav: 0.2.9
federatedfilesharing: 0.3.0
federation: 0.1.0
files: 1.5.1
files_external: 0.7.0
files_pdfviewer: 0.8.2
files_sharing: 0.10.0
files_texteditor: 2.2
files_trashbin: 0.9.0
files_versions: 1.3.0
files_videoplayer: 0.9.8
firstrunwizard: 1.1
gallery: 16.0.2
market: 0.1.0
notifications: 0.3.0
provisioning_api: 0.5.0
systemtags: 0.3.0
templateeditor: 0.1
updatenotification: 0.2.1 Disabled:
encryption
external
files_antivirus
theme-example
user_external Are you using external storage, if yes which one: local/smb/sftp/...

Are you using encryption: no

a) The javascript console log b) The network log c) ...

ghost commented 7 years ago

Most likely a duplicate of https://github.com/owncloud/core/issues/28031 . That one is still open but was solved by the OP by setting the correct permissions.

adrienafl commented 7 years ago

I saw that but as I said i ran the Strong Perm script, every files are www-datat owned...

ghost commented 7 years ago

Just keep in mind that if this is an external mounted storage (like NTFS) the permission script won't help here and might break more stuff then it solves.

adrienafl commented 7 years ago

I see. However my data folder is on the same server on the same part just in another folder.

I have a clue : The file where the script fail are files with UTF8 character like "é à ê" etc

Maybe the buffer isn't made to handle very large strings due to UTF8 characters ?

ghost commented 6 years ago

I performed a short test. I created a folder named は　in the user/files directory via ssh (correct permissions), and the scan failed with "Exception during scan: Undefined index: size" After I removed the folder, everything worked fine.

PVince81 commented 6 years ago

In my case it works for SFTP with OC 10.0.3RC1 :

± % occ files:scan --all -vvvv

Starting scan for user 1 out of 1 (admin)
        Folder /admin/
        Folder /admin/files_external
        Folder /admin/thumbnails
        Folder /admin/thumbnails/19
        File   /admin/thumbnails/19/32-32.png
        File   /admin/thumbnails/19/2048-2048-max.png
        Folder /admin/thumbnails/4
        File   /admin/thumbnails/4/32-32.png
        File   /admin/thumbnails/4/2048-2048-max.png
        Folder /admin/files
        File   /admin/files/welcome.txt
        Folder /admin/cache
        Folder /admin/files/sftp/
        File   /admin/files/sftp/y
        Folder /admin/files/sftp/x
        Folder /admin/files/sftp/は
        File   /admin/files/sftp/は/hello.txt

stuporglue commented 6 years ago

I'm using SFTP with OC 10.0.3.2 and am getting the same error (Undefined index: size) when it tries to scan a folder named "Rīgas Doma zēnu koris".

Also:

Mel Tormé
Les Misérables (Original Broadway Cast Recording)
Alô, paixão.mp3

The scan was able to proceed after renaming these files.

--

The scan also produced the same error when it hit a dead symlink in the SFTP directory. Fixing the symlink let the scan complete.

PVince81 commented 6 years ago

@IljaN didn't you observe something similar ?

patrickjahns commented 6 years ago

@stuporglue Thank's for your findings - the issue with the dead symlinks is addressed in this PR: https://github.com/owncloud/core/pull/28959

However - the issue with the filenames is not yet addressed @IljaN

IljaN commented 6 years ago

@stuporglue / @yakuraku Could you please provide the sshfs line which is used to mount the directory?

stuporglue commented 6 years ago

Unless OwnCloud sets up sshfs automatically, I don't think I'm using sshfs. I think it's just using sftp.

I enabled External Storage and set up the shares as the admin. Users aren't allowed to mount external storage. I chose "SFTP" as the storage type, and used "RSA public key" as the auth method.

I created an "owncloud_sftp" user on my remote machine.

I let OwnCloud generate the SSH key, then added the generated key to ~owncloud_sftp/.ssh/authorizedKeys

ghost commented 6 years ago

@stuporglue Seems SFTP + keypair mode currently doesn't work at all according to https://github.com/owncloud/core/issues/28669 and https://doc.owncloud.org/server/latest/admin_manual/release_notes.html#changes-in-10-0-3

stuporglue commented 6 years ago

Yes, i applied the temp fix in this comment https://github.com/owncloud/core/issues/28669#issuecomment-330791299

And it's working for me, except for files with certain characters. I've renamed my files though so this isn't affecting me anymore at the moment.

-- Michael

On Sunday, October 1, 2017, kdslkdsaldsal notifications@github.com wrote:

@stuporglue https://github.com/stuporglue Seems SFTP + keypair mode currently doesn't work at all according to #28669 https://github.com/owncloud/core/issues/28669 and https://doc.owncloud.org/server/latest/admin_manual/ release_notes.html#changes-in-10-0-3

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/owncloud/core/issues/28181#issuecomment-333384339, or mute the thread https://github.com/notifications/unsubscribe-auth/AABAwIBiDwZx2-Zw7JvNIgT2UNRsKIsyks5sn6-EgaJpZM4OA3FJ .

patrickjahns commented 6 years ago

@stuporglue

And it's working for me, except for files with certain characters. I've renamed my files though so this isn't affecting me anymore at the moment.

Can you tell me, if you have php-intl installed/enabled? If it is missing - could you try installing / enabling php-intl and check if the error would still occur?

stuporglue commented 6 years ago

@patrickjahns Sorry for the delay responding here. I do have php-intl installed.

$ dpkg -l | grep php | grep intl
ii  php7.0-intl                                   7.0.22-3                              amd64        Internationalisation module for PHP

And here's the relevant part of php --info output:

intl

Internationalization support => enabled
version => 1.1.0
ICU version => 57.1
ICU Data version => 57.1

Directive => Local Value => Master Value
intl.default_locale => no value => no value
intl.error_level => 0 => 0
intl.use_exceptions => 0 => 0

PVince81 commented 6 years ago

Can you guys check the database collations ? DB collation, table and column collations. They all should say something like utf8.

stuporglue commented 6 years ago

Thanks for looking into this.

I'm using Postgres. Default collation is UTF-8...

                                  List of databases
    Name     |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
-------------+----------+----------+-------------+-------------+-----------------------
 owncloud    | owncloud | UTF8     | en_US.UTF-8 | en_US.UTF-8 |

And there aren't any tables which override that.

owncloud=# select table_schema,
       table_name,
       column_name,
       collation_name
from information_schema.columns
where collation_name is not null
order by table_schema,
         table_name,
         ordinal_position;
 table_schema | table_name | column_name | collation_name
--------------+------------+-------------+----------------
(0 rows)

PVince81 commented 6 years ago

Assigning to @patrickjahns who was trying to reproduce this, with no success so far.

PVince81 commented 6 years ago

If the files in question use the UTF-8 NFD encoding (created on Mac) which is different than the standard UTF-8 NFC that Windows and Linux uses, then please make sure to set the option "NFD/Mac compatibility mount option" under the cog icon in the external storage section.

This is not enabled by default because it is rather expensive as it needs extra logic to convert back and forth and check for duplicates.

stuporglue commented 6 years ago

If the files in question use the UTF-8 NFD

This is not enabled by default because it is rather expensive as it needs extra logic to convert back and forth and check for duplicates.

I have removed all the accents from my file names, so I can't confirm if this was the issue or not, however this isn't something that a typical user should have to know or care about.

It seems that as long as ownCloud is detecting a scanning error it could check the file name of the file that caused the error and enable NFD compatibility for future syncs at that time - or at the very least add an "NFD detected" message to the error log.

Or maybe that's the long term plan already, and the option in the cog icon is a stopgap measure for now.

PVince81 commented 6 years ago

I seem to remember that the occ files:scan command is supposed to output something about NFD when scanning this type of file: https://github.com/owncloud/core/pull/24341. The message looks like "xyz will not be accessible due to incompatible encoding". Strange if you guys didn't get it.

Auto-enabling NFD compat only during scan wouldn't be helpful because even if the files in question get added, then disable again, anyone with access to that external storage would still get errors.

Also if we'd enable it permanently after detection, the ext storage would permanently have worse performance even if there's just a single file. Even if said file was later on removed. So better keep this opt-in.

Can you confirm that occ files:scan didn't show the above message ?

guillelebon commented 6 years ago

I do too have a problem with file scanning containing certain accentuated caharacters. We have about 200 users on our owncloud instance, for most of tem file scan works flawlessly, but in some accounts, you can clearly see that the scanner throws an error on certain accentuated file names. Many of our users have hundreds of accentuated filenames or folder names that don't pose a problem, but in some accounts, very specific accentued file provoque the following error:

For example, when I run sudo -u apache php occ files:scan deneaulta -vvvv,

I get:

File   /deneaulta/Mes images/Affiches cahier activité/Diapositive51.JPG
    File   /deneaulta/Mes images/Affiches cahier activité/Diapositive52.JPG
    File   /deneaulta/Mes images/Affiches cahier activité/Diapositive53.JPG
    File   /deneaulta/Mes images/Affiches cahier activité/Diapositive54.JPG
    File   /deneaulta/Mes images/Affiches cahier activité/Diapositive6.JPG
    File   /deneaulta/Mes images/Affiches cahier activité/Diapositive7.JPG
    File   /deneaulta/Mes images/Affiches cahier activité/Diapositive8.JPG
    File   /deneaulta/Mes images/Affiches cahier activité/Diapositive9.JPG
    File   /deneaulta/Mes images/Affiches cahier activité/Voyage Boston.jpg
    File   /deneaulta/Mes images/Affiches cahier activité/Grande affiche de la fête 2.pdf
Exception during scan: ErrorException: Undefined index: size
#0 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(421): OCA\Files\Command\Scan->exceptionErrorHandler(8, 'Undefined index...', '/var/www/html/o...', 421, Array)
#1 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(382): OC\Files\Cache\Scanner->handleChildren('Mes images/Affi...', true, 3, '132403', true, 5981118)
#2 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(385): OC\Files\Cache\Scanner->scanChildren('Mes images/Affi...', true, 3, '132403', true)
#3 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(385): OC\Files\Cache\Scanner->scanChildren('Mes images', true, 3, '132392', true)
#4 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(316): OC\Files\Cache\Scanner->scanChildren('', true, 3, 132390, true)
#5 /var/www/html/owncloud/lib/private/Files/Utils/Scanner.php(238): OC\Files\Cache\Scanner->scan('', true, 3)
#6 /var/www/html/owncloud/apps/files/lib/Command/Scan.php(224): OC\Files\Utils\Scanner->scan('/deneaulta', false)
#7 /var/www/html/owncloud/apps/files/lib/Command/Scan.php(310): OCA\Files\Command\Scan->scanFiles('deneaulta', '/deneaulta', true, Object(Symfony\Component\Console\Output\ConsoleOutput), false, false)
#8 /var/www/html/owncloud/lib/composer/symfony/console/Command/Command.php(262): OCA\Files\Command\Scan->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#9 /var/www/html/owncloud/core/Command/Base.php(159): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#10 /var/www/html/owncloud/lib/composer/symfony/console/Application.php(826): OC\Core\Command\Base->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#11 /var/www/html/owncloud/lib/composer/symfony/console/Application.php(189): Symfony\Component\Console\Application->doRunCommand(Object(OCA\Files\Command\Scan), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#12 /var/www/html/owncloud/lib/composer/symfony/console/Application.php(120): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 /var/www/html/owncloud/lib/private/Console/Application.php(162): Symfony\Component\Console\Application->run(NULL, NULL)
#14 /var/www/html/owncloud/console.php(106): OC\Console\Application->run()
#15 /var/www/html/owncloud/occ(11): require_once('/var/www/html/o...')
#16 {main}

If I rename in Windows the same file, and I replace accentuated characters with the corresponding Windows accentuated characters, the file scanner completes the task.

So it goes from:

Grande affiche de la fête 2.pdf (before renaming, crashes scanner) to
Grande affiche de la fête 2.pdf (after renaming, scanner is ok and completes task)

which looks exactly the same on screen, but is somehow different.

Obviously removing accentuated chars is not an option for us, and I've tried many other fixes for our owncloud instance (permissions, check php version, apache modules etc. etc.), but everything seems to point towards a very specific set of accentuated chars crashing the file scanner.

Hope this helps.

Thx.

guillelebon commented 6 years ago

Forgot to post the technical details of our installation

Owncloud version 10.0.0.4, fresh install OS: CentOS Linux release 7.4.1708 (Core) PHP: 5.6.32 Php internationalization activated Internationalization support => enabled version => 1.1.0 ICU version => 50.1.2 ICU Data version => 50.1

Apache/2.4.6 (CentOS)

Owncloud database engine: mariaDB (mySQL), InnoDB, charset: UTF8_bin

Thank you

guillelebon commented 6 years ago

Here is a zipped file with a filename that crashes owncloud scanner Inscription aux activités_Classe_verte 2015.zip

Run the scan on it, without renaming the file, then rename the file by replacing the é in the filename with another é, and scan will complete...

PVince81 commented 6 years ago

@guillelebon this file name is using the NFD encoding:

  % convmv -f utf8 -t utf8 --nfc --notest Inscription\ aux\ activite<0301>s_Classe_verte\ 2015.docx
mv "./Inscription aux activités_Classe_verte 2015.docx" "./Inscription aux activités_Classe_verte 2015.docx"
Ready! I converted 1 files in 0 seconds.

  % unzip Inscription.aux.activites_Classe_verte.2015.zip 
Archive:  Inscription.aux.activites_Classe_verte.2015.zip
  inflating: Inscription aux activités_Classe_verte 2015.docx  

  % ll
total 4.9M
-rw-r--r-- 1 vincent users 1.7M Mar  9  2015 'Inscription aux activités_Classe_verte 2015.docx'
-rw-r--r-- 1 vincent users 1.7M Mar  9  2015 'Inscription aux activités_Classe_verte 2015.docx'
-rw-r--r-- 1 vincent users 1.6M Dec 14 09:33  Inscription.aux.activites_Classe_verte.2015.zip

See how both file names exist, one with NFC (the one I converted) and one with NFD, the wicked Mac version of UTF-8...

Please enable Mac compatibility in your external storage settings, it's under the cog icon. Note that it will make the storage slower because ownCloud needs to do some gymnastics to check for the existence of both NFC and NFD to provide a transparent normalized experience. (ownCloud only works with NFC file names).

PVince81 commented 6 years ago

I made a zip with both versions here: encoding-madness.zip

If you're not using Mac file names in general on your external storage you could als simply rename this file to its NFC UTF-8 name and the problem will disappear for this one file.

PVince81 commented 6 years ago

Okay, the file scanner is indeed broken. It's supposed to tell you that you have a NFD file name.

I'll take over this to investigate

PVince81 commented 6 years ago

This PR https://github.com/owncloud/core/pull/29739 makes the "Undefined index" error disappear, but the warning still doesn't appear and the NFD file is skipped instead of a warning.

PVince81 commented 6 years ago

PR to bring back the warning: https://github.com/owncloud/core/pull/29845

caguiar commented 6 years ago

I have the same problem, i.e. the file scanner crashes, and the "problem" files which exist on the smb external storage, dosen't appear on the owncloud side.

From what I read above enabling the Mac compatibility in the external storage should have resolved the issue(? did I understand wrong?) however, I have enabled the Mac compatibility in the external storage settings, which solved 99% of the problems, but I still have at least one file that dos not "sync".

Here is the owncloud.log content:

{"reqId":"NTjJlqJ7xJUeKaugL09z","level":3,"time":"2017-12-24T13:09:36+00:00","remoteAddr":"10.244.6.61","user":"97C5B7E0-4380-41D7-B203-5EC3195AD4CC","app":"PHP","method":"PROPFIND","url":"\/remote.php\/dav\/files\/97C5B7E0-4380-41D7-B203-5EC3195AD4CC\/_OUTRAS-ATIVIDADES_\/Prestacao_servicos\/UBQ%20-%20Prest.servi%C3%A7os\/Pedido%20Remb.%2020-02-2015","message":"Undefined index: size at \/var\/www\/owncloud\/lib\/private\/Files\/Cache\/Scanner.php#421"}
{"reqId":"NTjJlqJ7xJUeKaugL09z","level":3,"time":"2017-12-24T13:09:36+00:00","remoteAddr":"10.244.6.61","user":"97C5B7E0-4380-41D7-B203-5EC3195AD4CC","app":"PHP","method":"PROPFIND","url":"\/remote.php\/dav\/files\/97C5B7E0-4380-41D7-B203-5EC3195AD4CC\/_OUTRAS-ATIVIDADES_\/Prestacao_servicos\/UBQ%20-%20Prest.servi%C3%A7os\/Pedido%20Remb.%2020-02-2015","message":"Undefined index: size at \/var\/www\/owncloud\/lib\/private\/Files\/Cache\/Scanner.php#424"}

And here is a zip with a file with a name that issued the error message:

problem_file.zip

ownclouders commented 6 years ago

Hey, this issue has been closed because the label status/STALE is set and there were no updates for 7 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.)

GregorB54321 commented 6 years ago

Short Braindump of my analysis:

`sudo -uwww-data ./occ files:scan --repair OCLogonUser [sudo] Passwort für server: Starting scan (and repair) for user 1 out of 1 (OCLogonUser) 0 [>---------------------------] Exception during scan: ErrorException: Undefined index: size

0 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(421): OCA\Files\Command\Scan->exceptionErrorHandler(8, 'Undefined index...', '/var/www/html/o...', 421, Array)

1 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(382): OC\Files\Cache\Scanner->handleChildren('files_versions/...', true, 3, '83097', true, 0)

2 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(385): OC\Files\Cache\Scanner->scanChildren('files_versions/...', true, 3, '83097', true)

3 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(385): OC\Files\Cache\Scanner->scanChildren('files_versions/...', true, 3, '83096', true)

4 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(385): OC\Files\Cache\Scanner->scanChildren('files_versions/...', true, 3, '83095', true)

5 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(385): OC\Files\Cache\Scanner->scanChildren('files_versions', true, 3, '24448', true)

6 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(316): OC\Files\Cache\Scanner->scanChildren('', true, 3, '32', true)

7 /var/www/html/owncloud/lib/private/Files/Utils/Scanner.php(238): OC\Files\Cache\Scanner->scan('', true, 3)

8 /var/www/html/owncloud/apps/files/lib/Command/Scan.php(224): OC\Files\Utils\Scanner->scan('/OCLogonUser', true)

9 /var/www/html/owncloud/apps/files/lib/Command/Scan.php(310): OCA\Files\Command\Scan->scanFiles('OCLogonUser', '/OCLogonUser', false, Object(Symfony\Component\Console\Output\ConsoleOutput), false, true)

10 /var/www/html/owncloud/lib/composer/symfony/console/Command/Command.php(262): OCA\Files\Command\Scan->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

11 /var/www/html/owncloud/core/Command/Base.php(159): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

12 /var/www/html/owncloud/lib/composer/symfony/console/Application.php(826): OC\Core\Command\Base->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

13 /var/www/html/owncloud/lib/composer/symfony/console/Application.php(189): Symfony\Component\Console\Application->doRunCommand(Object(OCA\Files\Command\Scan), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

14 /var/www/html/owncloud/lib/composer/symfony/console/Application.php(120): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

15 /var/www/html/owncloud/lib/private/Console/Application.php(162): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

16 /var/www/html/owncloud/console.php(106): OC\Console\Application->run()

17 /var/www/html/owncloud/occ(11): require_once('/var/www/html/o...')

18 {main}

+---------+-------+--------------+ | Folders | Files | Elapsed time | +---------+-------+--------------+ | 2221 | 28005 | 00:00:58 | +---------+-------+--------------+ `

MariaDB [owncloud]> select fileid, path from oc_filecache where fileid in (83097 ,83096 ,83095 ,24448 ,32 ); +--------+---------------------------------------------------+ | fileid | path | +--------+---------------------------------------------------+ | 32 | | | 24448 | files_versions | | 83095 | files_versions/.thunderbird | | 83096 | files_versions/.thunderbird/xyz123ab.default | | 83097 | files_versions/.thunderbird/xyz123ab.default/Mail | +--------+---------------------------------------------------+ 5 rows in set (0.00 sec)

server-p560 Mail # pwd /var/www/owncloud_data/OCLogonUser/files_versions/.thunderbird/xyz123ab.default/Mail server-p560 Mail # ls -lah insgesamt 0 drwxr-xr-x 1 www-data www-data 214 Feb 26 21:17 . drwxr-xr-x 1 www-data www-data 26K Mär 15 17:30 .. drwxr-xr-x 1 www-data www-data 5,0K Mär 15 17:30 Surname.Lastname drwxr-xr-x 1 www-data www-data 5,4K Mär 15 17:30 LastnameSurname drwxr-xr-x 1 www-data www-data 4,0K Mär 15 17:27 www.domain.de drwxr-xr-x 1 www-data www-data 4,2K Mär 15 17:27 adress_one drwx------ 1 root root 5,9K Mär 14 11:25 adress_two drwxr-xr-x 1 www-data www-data 6,9K Mär 15 17:30 adress_three drwxr-xr-x 1 www-data www-data 734 Mär 15 17:27 Local Folders drwxr-xr-x 1 www-data www-data 868 Mär 15 11:00 smart mailboxes server-p560 Mail #

` server@server-p560 /var/www/html/owncloud $ sudo -uwww-data ./occ files:scan --repair OCLogonUser Starting scan (and repair) for user 1 out of 1 (OCLogonUser) 0 [>---------------------------]

+---------+-------+--------------+ | Folders | Files | Elapsed time | +---------+-------+--------------+ | 2302 | 29225 | 00:00:59 | +---------+-------+--------------+ `

please reformat anybody my post. i have no clue how to do that!

PVince81 commented 6 years ago

Please apply the patch from https://github.com/owncloud/core/pull/30302 to make the "Undefined index" error disappear. Maybe a more useful message will appear.

Please also try setting the options for the external storage to use the encoding compatibility mode. You likely have files which names are encoded with UTF-8 NFD.

jean151515 commented 6 years ago

I had the same issue with owncloud-10.0.7 with external storage hooked up using SFTP.

What did the trick for me was mass converting my files using a linux package named convmv: convmv -f utf8 -t utf8 --nfc --replace --nosmart --notest -r "Your Folder"/

PVince81 commented 6 years ago

no time to finish this, moving to planned for next

PVince81 commented 6 years ago

The error will be gone and scanning will go forward. However the NFD error message is still missing.

IljaN commented 6 years ago

@PVince81 It is a bit tricky to add a message because on a low-level php can`t read the metadata (size) of the file, AFAIR there are different conditions which can lead to this so we would need some reliable way to detect NFD-Filenames beforehand. But need to dig deeper again to have the proper context for this problem.

COArikan commented 3 years ago

Hi, I've made a temporary fix for this issue in /var/www/owncloud/lib/private/Files/Cache/Scanner.php Line 421 changed from } else if ($data['size'] === -1) { to } else if ( (isset($data['size'])) && ($data['size'] === -1) ) {

and before Line 424, added a line as: if (isset($date['size'])) $size += $data['size'];

It is fixed the error and scanning continue..

phil-davis commented 3 years ago

The line numbers look out-of-date compared to current master https://github.com/owncloud/core/blob/master/lib/private/Files/Cache/Scanner.php

PR https://github.com/owncloud/core/pull/32408/files did some changes there in 2018.

@COArikan what version of ownCloud are you running?

Can you make a PR to owncloud/core with the changes? That will avoid confusion about exactly what changes you are suggesting.

stale[bot] commented 2 years ago

This issue has been automatically closed.

owncloud / core

External file scanning error #28181

Steps to reproduce

Expected behaviour

Actual behaviour

Server configuration

0 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(421): OCA\Files\Command\Scan->exceptionErrorHandler(8, 'Undefined index...', '/var/www/html/o...', 421, Array)

1 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(382): OC\Files\Cache\Scanner->handleChildren('files_versions/...', true, 3, '83097', true, 0)

2 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(385): OC\Files\Cache\Scanner->scanChildren('files_versions/...', true, 3, '83097', true)

3 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(385): OC\Files\Cache\Scanner->scanChildren('files_versions/...', true, 3, '83096', true)

4 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(385): OC\Files\Cache\Scanner->scanChildren('files_versions/...', true, 3, '83095', true)

5 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(385): OC\Files\Cache\Scanner->scanChildren('files_versions', true, 3, '24448', true)

6 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(316): OC\Files\Cache\Scanner->scanChildren('', true, 3, '32', true)

7 /var/www/html/owncloud/lib/private/Files/Utils/Scanner.php(238): OC\Files\Cache\Scanner->scan('', true, 3)

8 /var/www/html/owncloud/apps/files/lib/Command/Scan.php(224): OC\Files\Utils\Scanner->scan('/OCLogonUser', true)

9 /var/www/html/owncloud/apps/files/lib/Command/Scan.php(310): OCA\Files\Command\Scan->scanFiles('OCLogonUser', '/OCLogonUser', false, Object(Symfony\Component\Console\Output\ConsoleOutput), false, true)

10 /var/www/html/owncloud/lib/composer/symfony/console/Command/Command.php(262): OCA\Files\Command\Scan->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

11 /var/www/html/owncloud/core/Command/Base.php(159): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

12 /var/www/html/owncloud/lib/composer/symfony/console/Application.php(826): OC\Core\Command\Base->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

13 /var/www/html/owncloud/lib/composer/symfony/console/Application.php(189): Symfony\Component\Console\Application->doRunCommand(Object(OCA\Files\Command\Scan), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

14 /var/www/html/owncloud/lib/composer/symfony/console/Application.php(120): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

15 /var/www/html/owncloud/lib/private/Console/Application.php(162): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

16 /var/www/html/owncloud/console.php(106): OC\Console\Application->run()

17 /var/www/html/owncloud/occ(11): require_once('/var/www/html/o...')

18 {main}