A Shared folder is not displaying all of the data to users - lots of errors in owncloud log

gig13 commented 9 years ago

Steps to reproduce

WIP

Expected behaviour

Sharing a folder with other users should allow all users to see the entire folder. The owner of the data should also be able to view all of the content in the folder via the desktop client, web interface or via mobile apps

Actual behaviour

Most of the content from a shared folder appears to have disappeared when viewing from the web interface, desktop client or via the mobile apps. When checking the ownCloud server, the data still resides in the users directory.

Server configuration

Operating system: Ubuntu -- see config report https://s3.owncloud.com/owncloud/index.php/apps/files?dir=%2FShared%2Fowncloud%2Fsupport%2Fgithub-issues%2Fcore%2F17640

Web server: Apache 2.2

Database: PostgreSQL

PHP version: 5.4.42

ownCloud version: 8.0.5

Updated from an older ownCloud or fresh install: Updated

List of activated apps: See report config on s3

The content of config/config.php: see report config on s3

Are you using external storage, if yes which one: NO

Are you using encryption: yes/no

Are you using an external user-backend, if yes which one: OpenLDAP

LDAP configuration

See s3 report config https://s3.owncloud.com/owncloud/index.php/apps/files?dir=%2FShared%2Fowncloud%2Fsupport%2Fgithub-issues%2Fcore%2F17640

Client configuration

Browser: Internet Explorer

Operating system: Windows

Logs

ownCloud log (data/owncloud.log)

See s3 -- https://s3.owncloud.com/owncloud/index.php/apps/files?dir=%2FShared%2Fowncloud%2Fsupport%2Fgithub-issues%2Fcore%2F17640

Lots of these errors:

{"reqId":"8ebb786130164e097a9912b4cb7f02a8","remoteAddr":"123456789","app":"PHP","message":"Illegal offset type in isset or empty at \/x\/a\/owncloud-8.0.5\/lib\/private\/user\/database.php#184","level":3,"time":"2015-07-14T10:54:38+00:00"}
{"reqId":"8ebb786130164e097a9912b4cb7f02a8","remoteAddr":"123456789","app":"PHP","message":"Array to string conversion at \/x\/a\/owncloud-8.0.5\/apps\/user_ldap\/user_ldap.php#195","level":3,"time":"2015-07-14T10:54:38+00:00"}

gig13 commented 8 years ago

@blizzz The scan was listed above

> I ran the file scan and it exited after some time with this:
>
> Scanning folder /alex/cache
>
>
>
> [OC\ServerNotAvailableException]
> Lost connection to LDAP server.
>
>
>
> files:scan [-p|--path="..."] [-q|--quiet] [--all] [user_id1] ... [user_idN]

LDAP info is in the config report on s3.

blizzz commented 8 years ago

LDAP info is in the config report on s3.

But not the information is asked about. Unless I oversaw it, then point me please.

gig13 commented 8 years ago

@blizzz What are you looking for? If there is a proxy being used? They have upgraded several times and the issues never go away. And without being able to perform a file:scan how can this every be reconciled?

I even provided an entire database dump on s3 -- so you can review the tables. What else do you need?

gig13 commented 8 years ago

@blizzz @bboule Any updates?

blizzz commented 8 years ago

What are you looking for? If there is a proxy being used?

How their LDAP setup is. Are they having just one server, everything is talking to, or are there LDAP slaves and their OC servers is talking to one, or are there many LDAP servers and OC connects with one of them via proxy, or somehow else?

They have upgraded several times and the issues never go away.

So they are now on 8.0.6 or 8.0.8?

Do we have log output from the scan command attempts (ideally with logging set to DEBUG level)? In none of the provided log files i can find anything about Lost connection to LDAP server.

gig13 commented 8 years ago

@blizzz

They are on 8.0.6

The ldap server is on the same subnet on the same switch as the owncloud server. It also serves all my users in the office and is monitored, and there have been no alerts.

also i tried running it again and php spins at 100% for a really long time trying to process thumbnails:

Scanning file /root/thumbnails/407681/36-36.png
Scanning folder /root/thumbnails/407684
Scanning file /root/thumbnails/407684/36-36.png
Scanning folder /root/thumbnails/415194
Scanning file /root/thumbnails/415194/36-36.png
Scanning folder /root/thumbnails/407674
Scanning file /root/thumbnails/407674/36-36.png
Scanning folder /root/thumbnails/414044
Scanning file /root/thumbnails/414044/36-36.png
Scanning folder /root/thumbnails/6913
Scanning file /root/thumbnails/6913/36-36.png
Scanning folder /root/thumbnails/405362
Scanning file /root/thumbnails/405362/36-36.png
Scanning folder /root/thumbnails/6911
Scanning file /root/thumbnails/6911/36-36.png

PVince81 commented 8 years ago

I can't help with "LDAP temporarily unavailable" issues and @blizzz is sick this week.

Ideal would be to have the "files:scan" for the "trusef" user complete and then see whether the file disappearance case occurs again.

It is likely that the problem will not appear again due to recent fixes related to LDAP not being available while syncing. In the past, an unavailable LDAP server could make the user's home folder appear empty to the outside and make the sync client believe that the files were gone.

It is likely that when some users saw the sync client's dialog "all files are gone, do you want to keep or delete", they clicked "keep", which caused their sync client to reupload the files they had locally, which explains why the folders all have a more recent timestamp. The reason why not all the files are there is likely to be because that user used selective sync, so didn't have all the files locally.

From what I understood from @blizzz and @MorrisJobke is that now in the newer OC 8 versions this situation cannot happen any more, because now an unavailable LDAP server will throw an error to the outside (503 service unavailable) instead of exposing an empty folder.

Now to confirm this we need to find a way to make the "files:scan" of the user "trusef" finish properly, and then let the system run as normal. After the scan all files should be visible again.

MorrisJobke commented 8 years ago

From what I understood from @blizzz and @MorrisJobke is that now in the newer OC 8 versions this situation cannot happen any more, because now an unavailable LDAP server will throw an error to the outside (503 service unavailable) instead of exposing an empty folder.

This is what we think. And it's not that unlikely.

@gig13 Is there any chance to get this instance upgrade to at least 8.0.6 and run the files:scan again?

gig13 commented 8 years ago

@MorrisJobke They are on 8.0.6 and have attempted the files scan

MorrisJobke commented 8 years ago

They are on 8.0.6 and have attempted the files scan

Oh. Sorry. Just read it above.

@PVince81 Then it seems that it is still another issue with the filesystem setup. cc @icewind1991

PVince81 commented 8 years ago

So I guess the missing folders are still missing now ?

Now thinking of it, the output @gig13 gave here https://github.com/owncloud/core/issues/17640#issuecomment-121606403 is directly from the data folder. So if the missing folders aren't there, a rescan will not help.

The folders will have to be restored manually by moving/copying them from "new_trusef".

MorrisJobke commented 8 years ago

This is an old log, but maybe it's still present. And it looks somehow fishy.

{"reqId":"c8fc413304cfaf11504ed47f6b6588b1","remoteAddr":"10.10.162.27","app":"PHP","message":"Illegal offset type in isset or empty at \/x\/a\/owncloud-8.0.5\/lib\/private\/user\/manager.php#109","level":3,"time":"2015-07-12T06:49:18+00:00"}
{"reqId":"c8fc413304cfaf11504ed47f6b6588b1","remoteAddr":"10.10.162.27","app":"PHP","message":"Illegal offset type in isset or empty at \/x\/a\/owncloud-8.0.5\/lib\/private\/user\/database.php#184","level":3,"time":"2015-07-12T06:49:18+00:00"}
{"reqId":"c8fc413304cfaf11504ed47f6b6588b1","remoteAddr":"10.10.162.27","app":"PHP","message":"Array to string conversion at \/x\/a\/owncloud-8.0.5\/3rdparty\/doctrine\/dbal\/lib\/Doctrine\/DBAL\/Driver\/PDOStatement.php#91","level":3,"time":"2015-07-12T06:49:18+00:00"}
{"reqId":"c8fc413304cfaf11504ed47f6b6588b1","remoteAddr":"10.10.162.27","app":"PHP","message":"Illegal offset type in isset or empty at \/x\/a\/owncloud-8.0.5\/lib\/private\/user\/database.php#225","level":3,"time":"2015-07-12T06:49:18+00:00"}
{"reqId":"c8fc413304cfaf11504ed47f6b6588b1","remoteAddr":"10.10.162.27","app":"PHP","message":"Array to string conversion at \/x\/a\/owncloud-8.0.5\/apps\/user_ldap\/user_ldap.php#194","level":3,"time":"2015-07-12T06:49:18+00:00"}
{"reqId":"c8fc413304cfaf11504ed47f6b6588b1","remoteAddr":"10.10.162.27","app":"PHP","message":"Array to string conversion at \/x\/a\/owncloud-8.0.5\/apps\/user_ldap\/user_ldap.php#195","level":3,"time":"2015-07-12T06:49:18+00:00"}

PVince81 commented 8 years ago

Regarding the "Array to string conversion", see https://github.com/owncloud/core/issues/17640#issuecomment-125656748, still not clear why an Array was returned instead of the user object. Could be due to the user not existing or another bug. There is a chance that it might be fixed already, see https://github.com/owncloud/core/issues/17640#issuecomment-125676360

MorrisJobke commented 8 years ago

Okay. Just had a screen sharing session.

As it turned out this was a leftover from old filesystem setups. The folders were actually present in the data dir (and showed up as non shared in the "Shared" folder). This was quite confusing in the first time. What now is done:

cleanup those leftover folders (most of them are completely empty and are just the structure of the old shared folder, that is not shared anymore) in the data dir
run a filescan to let owncloud recognize the changed filesystem
clients should clean up the folders on the desktop PCs

Case closed. :)

owncloud / core