S3 external storage doesn't show objects uploaded by "aws s3 sync" command line tool.

[stevenmcastano]

Steps to reproduce

1. Create a new Amazon AWS S3 storage bucket with all the default options
2. Configure owncloud to mount this external storage
3. Create a new directory, or next text file from the owncloud web interface, upload a file through the owncloud web interface and all your files and folders show up fine.
4. Install the "aws" command line tool. Mine is the latest version running on macOS Sierra
5. Use the "aws s3 sync" command to upload an entire folder, preferable with sub folders and some test files to the root of your s3 bucket. A command like "aws s3 sync . s3://mybucketname"
6. Log into the AWS console and look at the contents of your s3 bucket, you should see the files you created through owncloud and the files you uploaded via the sync command
7. Now browse that externally mounted folder through owncloud, you won't see any of the files or folders that were uploaded with the sync command. It's as if they're invisible for some reason.

Expected behaviour

You should see all the files, weather you upload through the owncloud web interface, an owncloud sync client, the aws cli tools, or even upload through the aws web portal.

Actual behaviour

You see all files except files created with the sync command. If you upload through anything else they work. Even if you use the "aws s3 cp" command to "copy" a file to S3, you can see it.

Server configuration

Operating system: Ubuntu 14.04

Web server: NGINX 1.12

Database: MariaDB 10.0.x

PHP version: 7.0

ownCloud version: 10.0.2 (stable)

Updated from an older ownCloud or fresh install: Updated from 9.0.3 (I think, 9.0.something for sure)

Where did you install ownCloud from: owncloud.org

Signing status (ownCloud 9.0 and above):

Login as admin user into your ownCloud and access 
http://example.com/index.php/settings/integrity/failed 
paste the results into https://gist.github.com/ and put the link here.

No need for the above, it reports "No errors have been found."

Are you using external storage, if yes which one: S3

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

Client configuration

Browser: Chrome, Firefox and IE all test

Operating system: macOS, Windows 8, Windows 10

Logs

ownCloud log (data/owncloud.log)

Lots of errors like the following:

{"reqId":"********","level":3,"time":"2017-06-14T22:00:14+00:00","remoteAddr":"1********","user":"********","app":"PHP","method":"PROPFIND","url":"\/remote.php\/webdav\/myfilesystem-s3","message":"Undefined index: size at \/usr\/share\/nginx\/owncloud\/lib\/private\/Files\/Cache\/Scanner.php#416"}
{"reqId":"********","level":3,"time":"2017-06-14T22:00:14+00:00","remoteAddr":"1********","user":"********","app":"PHP","method":"PROPFIND","url":"\/remote.php\/webdav\/myfilesystem-s3","message":"Undefined index: size at \/usr\/share\/nginx\/owncloud\/lib\/private\/Files\/Cache\/Scanner.php#419"}
{"reqId":"********","level":3,"time":"2017-06-14T22:00:14+00:00","remoteAddr":"1********","user":"********","app":"PHP","method":"PROPFIND","url":"\/remote.php\/webdav\/myfilesystem-s3","message":"Undefined index: size at \/usr\/share\/nginx\/owncloud\/lib\/private\/Files\/Cache\/Scanner.php#416"}
{"reqId":"********","level":3,"time":"2017-06-14T22:00:14+00:00","remoteAddr":"1********","user":"********","app":"PHP","method":"PROPFIND","url":"\/remote.php\/webdav\/myfilesystem-s3","message":"Undefined index: size at \/usr\/share\/nginx\/owncloud\/lib\/private\/Files\/Cache\/Scanner.php#419"}

[SamuAlfageme]

I think this is https://github.com/owncloud/core/issues/26819 but using the CLI instead of the web console (the underlying issue might be the same)

[stevenmcastano]

This could possibly be a similar issue, with a few differences. The errors I'm seeing in my logs reference a different function and line number, but it mat be caused be some central code.

Also, if I do an "aws 3s cp /local/textfile.txt s3://mybucket" the files DO show up. So for me, I can COPY from the command line and it works, but not the sync.

Also, just for fun, I sync a directory with a sub-directory called "testing", couldn't see it through the owncloud mount, but then made a directory called "testing" through owncloud. It make the directory, didn't throw any error messages and when I added files to it through owncloud, the wound up in the right place, i.e. the PRE-exisiting "testing" directory that I couldn't see before.

[mitiih]

Same problem here... If I use "Cloudberry Explorer for Amazon S3" to upload folders OR upload a folder through AWS S3 web console, the folder does not show up in owncloud storage. Then if I create (not upload) a folder in AWS S3, that does show up in owncloud. Folder size does not seem to matter in uploading problem.

Uploaded files does show up, no problem there but uploaded folders do not.

I'm just testing owncloud with AWS S3 so using t2 micro EC2 Ubuntu 16.04 and owncloud version 10.0.2. The setup is just installed and updated.

[samwilson]

I think this is because the empty pseudo-directory items are not created (the ones with no content and keys that end in slashes). It's not OwnCloud's fault.

A workaround seems to be as follows.

After running the sync e.g.

s3cmd sync /path/to/owncloud_data/theuser/files/ s3://thebucket

Then run through all the directories and create their (empty) objects:

for F in $(find /path/to/owncloud_data -type d -print '%P\n'); do
  aws s3api put-object --bucket thebucket --key "$F/"
done

Note the trailing slash on the key name.

[PVince81]

duplicate of https://github.com/owncloud/core/issues/13507 then

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.