Closed Bazon closed 8 years ago
PS: permissions of the files generated by streamripper (run as http) should be OK, ls - l gives -rw-r--r-- 1 http http
In general it isn't recommended to store files directly inside the data folder. You might want to use the "Local" external storage type and point it at the folder in question.
Regarding the issue, the scanner probably doesn't update the etag of the parent folders for some reason.
Also the scanner never generates activity events because the file wasn't changed through ownCloud, this has never worked before and was never intended to work that way.
I tried the following in v9.0.0:
cd data/admin/files/"
mkdir test
cp ~vincent/temp/bacon.txt test/
(test file)chown -R wwwrun:www test
(the web server user)sudo -uwwwrun ./occ files:scan --path="admin/files/test/bacon.txt" -vvv
sudo -uwwwrun ./occ files:scan --path="admin/files/test/bacon.txt" -vvv
However if I run it on the parent folder, it works: sudo -uwwwrun ./occ files:scan --path="admin/files/test" -vvv
I checked the database and it looks like whenever a file gets scanned directly, only that file's etag changes, not the one of the parents. That's likely the regression you were mentionned.
This isn't too critical because the workaround is to rescan the parent folder, and there aren't many use cases where one would rescan only a single file.
Note that "occ files:scan" is used a lot with external storages.
External mount error There was an error with message: Empty response from the server. Do you want to review mount point config in admin settings page?
Yes, the web-user (http for me) has all permissions for this folder (drwxr-xr-x 2 http http ) and I set that very folder in the configuration in the admin-webinterface. But that's really another problem I suppose...
More detail about 2: My streamripper writes in /owncloud/data/tobi/files/music/streamripper/ and there in a directory named automatically after the radio station, in this case /owncloud/data/tobi/files/music/streamripper/Deutschlandradio Kultur
sudo -u http php /usr/share/webapps/owncloud/occ files:scan --path="/tobi/files/music/streamripper" -vvv
gives:
Starting scan for user 1 out of 1 (tobi) Folder /tobi/files/music/streamripper Folder /tobi/files/music/streamripper/try1-Deutschlandradio Kultur OGG/incomplete Folder /tobi/files/music/streamripper/Deutschlandradio Kultur File /tobi/files/music/streamripper/Deutschlandradio Kultur/Bednarz - O-Ton 2.mp3 File /tobi/files/music/streamripper/Deutschlandradio Kultur/Astrud Gilberto & Walter Wanderley - That sadness.mp3 ..... ..... ...... +---------+-------+--------------+ | Folders | Files | Elapsed time | +---------+-------+--------------+ | 8 | 114 | 00:00:00 | +---------+-------+--------------+
--> Seems to work. And indeed, all of this can be seen in the webinterface: http://i.imgur.com/Q6q8zT9.png But again, nothing of this triggers an activity: http://i.imgur.com/zWmk0pf.png
Please tell me if I can give you any more information, I would be really happy if I could use the resan --path occ-function again. as I said, all this worked in owncloud 8. Thanks!
OK. solved that local external storage problem (path has to be in php_admin_value open_basedir in /etc/httpd/conf/extra/owncloud.conf), but still exactly the same problem: Scanning seems to work:
sudo -u http php /usr/share/webapps/owncloud/occ files:scan --path="/tobi/files/local" -vvv
Starting scan for user 1 out of 1 (tobi) Folder /tobi/files/local/ File /tobi/files/local/testfile Folder /tobi/files/local/streamripper Folder /tobi/files/local/streamripper/Deutschlandradio Kultur Folder /tobi/files/local/streamripper/Deutschlandradio Kultur/incomplete File /tobi/files/local/streamripper/Deutschlandradio Kultur/incomplete/Mareike Maage & SebastianPeter - Geliehene Woerter.mp3
+---------+-------+--------------+ | Folders | Files | Elapsed time | +---------+-------+--------------+ | 4 | 2 | 00:00:00 | +---------+-------+--------------+
New file appears in the webinterface: http://i.imgur.com/dOvcm7Y.png (--> 11 minutes ago) But no sync-activity or download: http://i.imgur.com/SmRAa7z.png (-->last activity: 2 hours ago)
Ah! Look what's that: The foldername "Deutschlandradio Kultur" seems to be a problem!
After renaming it to "dradio" (mv Deutschlandradio\ Kultur/ dradio
), scanning
sudo -u http php /usr/share/webapps/owncloud/occ files:scan --path="/tobi/files/local" -vvv
Starting scan for user 1 out of 1 (tobi) Folder /tobi/files/local/ File /tobi/files/local/testfile Folder /tobi/files/local/streamripper Folder /tobi/files/local/streamripper/dradio Folder /tobi/files/local/streamripper/dradio/incomplete File /tobi/files/local/streamripper/dradio/incomplete/Mareike Maage & SebastianPeter - Geliehene Woerter.mp3
+---------+-------+--------------+ | Folders | Files | Elapsed time | +---------+-------+--------------+ | 4 | 2 | 00:00:00 | +---------+-------+--------------+
and downloading works! whyever, it doesn't appear in the activity-list, but in the clients sync-protocol tab: http://i.imgur.com/WXsFkDu.png
Edit: Yet another confusing picture: http://i.imgur.com/Rr7k3pe.png See: (local: 0B), but INSIDE (streamripper: 861kB) and right: nothing in the local dir on client side.
nothing of this triggers an activity
Yes, as I told you above, this is not implemented. Detecting a changed file on an external storage doesn't qualify as an activity.
If downloading with the desktop client works after scanning, then it's working as expected.
The only bug I think is that when scanning a direct file it doesn't propagate the etag any more.
If downloading with the desktop client works after scanning, then it's working as expected.
Well, it doesn't for me: No download with the desktop client after scanning with occ files:scan. (*) That's the problem, the missing download to clients. (I added the description about activity only to get a fuller picture.)
Before OwnCloud 9, I could create or modify any file directly on the server, scan the containing folder with occ files:scan --path="...", and that triggered an activity and download to client directly and immediately and was very useful for me.
The only bug I think is that when scanning a direct file it doesn't propagate the etag any more.
I don't scan direct files, I scan the containing folder. (Again, sorry for the misleading description in the initial report, I'll refine it.)
So let's hope that behaviour will come back with etag propagation. Thanks for your work!
(*) Usually. I still don't understand why it worked after the renaming described in https://github.com/owncloud/core/issues/23394#issuecomment-198759529
Well, it doesn't for me: No download with the desktop client after scanning with occ files:scan. (*)
This works for me. Not sure what difference there is on your setup.
You could check the "oc_filecache" table, the "etag" column. After rescanning, the "etag" of the folder and all its parents up to "files" must change.
Not sure what difference there is on your setup.
Me neither. I tried to find differences in the way I did it and you did and found 2:
You could check the "oc_filecache" table, the "etag" column.
Eh, sorry, I don't know how. I tried to understand, but didn't succeed. Not enough understanding of mariadb here. Could you give me a bash command to do so? For 'datadirectory' => '/magnet/owncloud/data', 'dbtype' => 'mysql', 'version' => '9.0.0.19', 'dbname' => 'owncloud', 'dbhost' => 'localhost', 'dbtableprefix' => 'oc_', 'dbuser' => 'owncloud',
mysql -u owncloud -p
(then enter the database password)select * from oc_filecache where name='files'
(that's the root)select * from oc_filecache where name='files'
(that's the root)The etag of streamripper and the root must have changed after the scan.
It keeps being strange: the etag DOES change for me! (Still no download to client.)
Here is what I did step by step:
sudo -u http mkdir testdir
cd testdir/
sudo -u http touch testdirtesfile
sudo -u http /usr/share/webapps/owncloud/occ files:scan --path="/tobi/files/local/testdir" -vvv
Starting scan for user 1 out of 1 (tobi)
Folder /tobi/files/local/testdir
File /tobi/files/local/testdir/testdirtesfile
+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 1 | 1 | 00:00:00 |
+---------+-------+--------------+ mysql -u root -p owncloud
[+ entering root password, not db-password], just mysql -u owncloud -p
ended up with an MariaDB prompt without selcted database: MariaDB [(none)]> )MariaDB [owncloud]> select * from oc_filecache where path like '%testdir%';
+--------+---------+------------------------+----------------------------------+--------+----------------+----------+----------+------+------------+---------------+-----------+------------------+----------------------------------+-------------+----------+
| fileid | storage | path | path_hash | parent | name | mimetype | mimepart | size | mtime | storage_mtime | encrypted | unencrypted_size | etag | permissions | checksum |
+--------+---------+------------------------+----------------------------------+--------+----------------+----------+----------+------+------------+---------------+-----------+------------------+----------------------------------+-------------+----------+
| 203328 | 18 | testdir | 3ca0a3620b59afb57cf5fd77cee6432c | 202832 | testdir | 2 | 1 | 0 | 1458557601 | 1458557601 | 0 | 0 | 56efd2c840cf6 | 31 | NULL |
| 203329 | 18 | testdir/testdirtesfile | 740da632f2191ab53bf8f423943f6ff7 | 203328 | testdirtesfile | 10 | 3 | 0 | 1458557601 | 1458557601 | 0 | 0 | fffcbe753184d23203bbe7b9be9bc5ae | 27 | NULL |
+--------+---------+------------------------+----------------------------------+--------+----------------+----------+----------+------+------------+---------------+-----------+------------------+----------------------------------+-------------+----------+
2 rows in set (0.34 sec)
nano testdirtesfile
to contain the text "blabla".sudo -u http /usr/share/webapps/owncloud/occ files:scan --path="/tobi/files/local/testdir" -vvv
(...)MariaDB [owncloud]> select * from oc_filecache where path like '%testdir%';
+--------+---------+------------------------+----------------------------------+--------+----------------+----------+----------+------+------------+---------------+-----------+------------------+----------------------------------+-------------+----------+
| fileid | storage | path | path_hash | parent | name | mimetype | mimepart | size | mtime | storage_mtime | encrypted | unencrypted_size | etag | permissions | checksum |
+--------+---------+------------------------+----------------------------------+--------+----------------+----------+----------+------+------------+---------------+-----------+------------------+----------------------------------+-------------+----------+
| 203328 | 18 | testdir | 3ca0a3620b59afb57cf5fd77cee6432c | 202832 | testdir | 2 | 1 | 7 | 1458557601 | 1458557601 | 0 | 0 | 56efd2c840cf6 | 31 | NULL |
| 203329 | 18 | testdir/testdirtesfile | 740da632f2191ab53bf8f423943f6ff7 | 203328 | testdirtesfile | 10 | 3 | 7 | 1458557754 | 1458557754 | 0 | 0 | 9ccbf37cf3f4f3f0141ea7320e320411 | 27 | NULL |
+--------+---------+------------------------+----------------------------------+--------+----------------+----------+----------+------+------------+---------------+-----------+------------------+----------------------------------+-------------+----------+
2 rows in set (0.34 sec)
So as the etag column is the third last column as I understand, it DID change for testdir/testdirtesfile.
sudo -u http touch testdirtesfile2
sudo -u http /usr/share/webapps/owncloud/occ files:scan --path="/tobi/files/local/testdir" -vvv
(...)MariaDB [owncloud]> select * from oc_filecache where path like '%testdir%';
+--------+---------+-------------------------+----------------------------------+--------+-----------------+----------+----------+------+------------+---------------+-----------+------------------+----------------------------------+-------------+----------+
| fileid | storage | path | path_hash | parent | name | mimetype | mimepart | size | mtime | storage_mtime | encrypted | unencrypted_size | etag | permissions | checksum |
+--------+---------+-------------------------+----------------------------------+--------+-----------------+----------+----------+------+------------+---------------+-----------+------------------+----------------------------------+-------------+----------+
| 203328 | 18 | testdir | 3ca0a3620b59afb57cf5fd77cee6432c | 202832 | testdir | 2 | 1 | 7 | 1458558716 | 1458558716 | 0 | 0 | 56efd72274407 | 31 | NULL |
| 203329 | 18 | testdir/testdirtesfile | 740da632f2191ab53bf8f423943f6ff7 | 203328 | testdirtesfile | 10 | 3 | 7 | 1458557754 | 1458557754 | 0 | 0 | 9ccbf37cf3f4f3f0141ea7320e320411 | 27 | NULL |
| 203330 | 18 | testdir/testdirtesfile2 | 2809bcc8ab675e10dcab736f202a3792 | 203328 | testdirtesfile2 | 10 | 3 | 0 | 1458558716 | 1458558716 | 0 | 0 | 6da8e414931dcbac71ff85d6932c2138 | 27 | NULL |
+--------+---------+-------------------------+----------------------------------+--------+-----------------+----------+----------+------+------------+---------------+-----------+------------------+----------------------------------+-------------+----------+
3 rows in set (0.33 sec)
So yes, the scan moves new files into the database and changes their etag after the file is being changed. (Stil no download to the client or activity.)
Testing the root in my next comment.....
@Bazon for the client to detect the change, the etag of the "files" entry needs to change too. What if you restart the desktop client ?
OK, indeed, the etag of the root doesn't change after changing or creating a file that way.
MariaDB [owncloud]> select * from oc_filecache where name='files';
+--------+---------+----------------------------------------------+----------------------------------+--------+-------+----------+----------+--------------+------------+---------------+-----------+------------------+---------------+-------------+----------+
| fileid | storage | path | path_hash | parent | name | mimetype | mimepart | size | mtime | storage_mtime | encrypted | unencrypted_size | etag | permissions | checksum |
+--------+---------+----------------------------------------------+----------------------------------+--------+-------+----------+----------+--------------+------------+---------------+-----------+------------------+---------------+-------------+----------+
| 2 | 1 | files | 45b963397aa40d4a0063e0d85e4fe7a1 | 1 | files | 2 | 1 | 464657382300 | 1458496630 | 1454762530 | 0 | 0 | 56eee4768cc99 | 31 | NULL |
| 13 | 2 | files | 45b963397aa40d4a0063e0d85e4fe7a1 | 12 | files | 2 | 1 | 6038748 | 1454790809 | 1406182116 | 0 | 0 | 56b65899940a7 | 31 | NULL |
| 218 | 3 | tobi/files | 3bbcf051db4bdf15d5f0b2efaa06eb03 | 217 | files | 2 | 1 | 10214059312 | 1401221750 | 1401221750 | 0 | 0 | 53878dd9b2aef | 0 | NULL |
| 7020 | 1 | files_trashbin/files | 3014a771cbe30761f2e9ff3272110dbf | 7019 | files | 2 | 1 | 20052124202 | 1458414634 | 1458414634 | 0 | 0 | 56eda42a4c0cd | 31 | NULL |
| 33697 | 1 | files/music/mp3-alt/.Trash-1000/files | 855828d17e147303b562d26d824fb8ef | 33696 | files | 2 | 1 | 1138455848 | 1411808812 | 1411808812 | 0 | 0 | 54267e2c9c90e | 31 | NULL |
| 140707 | 1 | files/arch/bin/Bridge Construction Set/files | 54ab5b7f0ce6e92176651e490c40fcbc | 140698 | files | 2 | 1 | 192698970 | 1063756940 | 1435984069 | 0 | 0 | 559760ca46eb6 | 31 | NULL |
| 178289 | 3 | tobi/files_trashbin/files | 850a0de8653e9e556676f75addbcbc26 | 16253 | files | 2 | 1 | 9270951048 | 1445278733 | 1445278733 | 0 | 0 | 56287eef0cf35 | 31 | NULL |
+--------+---------+----------------------------------------------+----------------------------------+--------+-------+----------+----------+--------------+------------+---------------+-----------+------------------+---------------+-------------+----------+
7 rows in set (0.18 sec)
MariaDB [owncloud]> select * from oc_filecache where name='files';
+--------+---------+----------------------------------------------+----------------------------------+--------+-------+----------+----------+--------------+------------+---------------+-----------+------------------+---------------+-------------+----------+
| fileid | storage | path | path_hash | parent | name | mimetype | mimepart | size | mtime | storage_mtime | encrypted | unencrypted_size | etag | permissions | checksum |
+--------+---------+----------------------------------------------+----------------------------------+--------+-------+----------+----------+--------------+------------+---------------+-----------+------------------+---------------+-------------+----------+
| 2 | 1 | files | 45b963397aa40d4a0063e0d85e4fe7a1 | 1 | files | 2 | 1 | 464657382300 | 1458496630 | 1454762530 | 0 | 0 | 56eee4768cc99 | 31 | NULL |
| 13 | 2 | files | 45b963397aa40d4a0063e0d85e4fe7a1 | 12 | files | 2 | 1 | 6038748 | 1454790809 | 1406182116 | 0 | 0 | 56b65899940a7 | 31 | NULL |
| 218 | 3 | tobi/files | 3bbcf051db4bdf15d5f0b2efaa06eb03 | 217 | files | 2 | 1 | 10214059312 | 1401221750 | 1401221750 | 0 | 0 | 53878dd9b2aef | 0 | NULL |
| 7020 | 1 | files_trashbin/files | 3014a771cbe30761f2e9ff3272110dbf | 7019 | files | 2 | 1 | 20052124202 | 1458414634 | 1458414634 | 0 | 0 | 56eda42a4c0cd | 31 | NULL |
| 33697 | 1 | files/music/mp3-alt/.Trash-1000/files | 855828d17e147303b562d26d824fb8ef | 33696 | files | 2 | 1 | 1138455848 | 1411808812 | 1411808812 | 0 | 0 | 54267e2c9c90e | 31 | NULL |
| 140707 | 1 | files/arch/bin/Bridge Construction Set/files | 54ab5b7f0ce6e92176651e490c40fcbc | 140698 | files | 2 | 1 | 192698970 | 1063756940 | 1435984069 | 0 | 0 | 559760ca46eb6 | 31 | NULL |
| 178289 | 3 | tobi/files_trashbin/files | 850a0de8653e9e556676f75addbcbc26 | 16253 | files | 2 | 1 | 9270951048 | 1445278733 | 1445278733 | 0 | 0 | 56287eef0cf35 | 31 | NULL |
+--------+---------+----------------------------------------------+----------------------------------+--------+-------+----------+----------+--------------+------------+---------------+-----------+------------------+---------------+-------------+----------+
7 rows in set (0.18 sec)
We fixed an issue with --path that cause missing size and etag propagation: https://github.com/owncloud/core/pull/24875
Change was backported 9.0.3, so would be nice to test this again on 9.0.3
I redid the tests from https://github.com/owncloud/core/issues/23394#issuecomment-198406120 on stable9 (upcoming 9.0.3) and syncing works correctly.
Thanks! I'll test as soon as ownCloud 9.0.3 arrives in arch linux.
ownCloud 9.0.3 arrived in arch linux and occ files:scan --path="..." works again, thanks! :-)
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Short Description:
I use my owncloud server as a radio-recorder with streamripper, too. After recording (in a owncloud share directory), I run a scan only in the path of the recording (with occ files:scan --path=....). That worked with owncloud 8.x. What happens with owncloud 9.0 is: The scan is successful, when I use my browser and the owncloud webinterface, the new files appear where they should. BUT!: This doesn't trigger any sync activity. Neither on the activity page in the webinterface, nor in the server-activity tab in the sync-client. The new files aren't downloaded to the clients.
### Steps to reproduce 1. On the server, cd to a path in an owncloud share files directory and create a new file there, e.g. with `touch testfile` 2. run occ files:scan --path="..." for the directory containing that new testfile, e.g. with `sudo -u http php /usr/share/webapps/owncloud/occ files:scan --path="/ocusername/files/path/to/testfile"` **[EDIT] I scan the containing folder,** not the file directly, so `sudo -u http php /usr/share/webapps/owncloud/occ files:scan --path="/ocusername/files/path/to/folder-containing-testfile"` is less misleading. **[/EDIT]** 3. wait for that new file to be synced: it won't. although it appears in the right path in the owncloud webinterface, but it doesn't appear in activities. ### Expected behaviour The scanned new file should appear in server activity and should be downloaded to clients using that share. (that worked with owncloud 8.x) ### Actual behaviour The scanned new file appears in the webinterface and the android app, but it doesn't appear in server activity and isn't downloaded to clients using that share. ### Server configuration **Operating system**: arch linux Linux 4.4.5-1-ARCH #1 SMP PREEMPT Thu Mar 10 07:38:19 CET 2016 x86_64 GNU/Linux **Web server:** apache 2.4.18-2 **Database:** mariadb 10.1.12-3 **PHP version:** php 7.0.4-2 php-apache 7.0.4-2 php-apcu 5.1.3-1 php-apcu-bc 1.0.3-1 php-gd 7.0.4-2 php-intl 7.0.4-2 php-pear 5.6.16-3 **ownCloud version:** (see ownCloud admin page) owncloud 9.0.0-1 **Updated from an older ownCloud or fresh install:** updated **Where did you install ownCloud from:** arch repository **Signing status (ownCloud 9.0 and above):** ``` Login as admin user into your ownCloud and access http://example.com/index.php/settings/integrity/failed paste the results here. ``` Integrity checker has been disabled. Integrity cannot be verified. **List of activated apps:** ``` If you have access to your command line run e.g.: sudo -u www-data php occ app:list from within your ownCloud installation folder ``` Enabled: - activity: 2.2.1 - admin_migrate: 0.1 - comments: 0.2 - dav: 0.1.5 - federatedfilesharing: 0.1.0 - federation: 0.0.4 - files: 1.4.4 - files_odfviewer: 0.1 - files_sharing: 0.9.1 - files_texteditor: 2.1 - files_trashbin: 0.8.0 - files_versions: 1.2.0 - provisioning_api: 0.4.1 - systemtags: 0.2 - templateeditor: 0.1 - updatenotification: 0.1.0 - user_migrate: 0.1 Disabled: - apptemplate - django_auth - emoji - encryption - encryption_dummy - external - files_archive - files_external - files_pdfviewer - files_sgfviewer - files_svgedit - files_videoviewer - firstrunwizard - fluxx_compensator - galleryplus - impress - imprint - music - ownpad_lite - pong - pushnotifications - reader - search - tattoo - testing - user_external - user_ldap - user_oauth - user_openid_provider - user_persona - user_saml - user_vd - user_webfinger **The content of config/config.php:** ``` If you have access to your command line run e.g.: sudo -u www-data php occ config:list system from within your ownCloud installation folder system { "system": { "instanceid": "oc9bd3e2992c", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "192.168.2.222", "replaced.noip.me" ], "datadirectory": "\/magnet\/owncloud\/data", "dbtype": "mysql", "version": "9.0.0.19", "dbname": "owncloud", "dbhost": "localhost", "dbtableprefix": "oc_", "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "maxZipInputSize": 838860800, "allowZipDownload": true, "forcessl": true, "loglevel": 4, "theme": "", "maintenance": false, "secret": "***REMOVED SENSITIVE VALUE***", "trashbin_retention_obligation": "auto", "overwrite.cli.url": "https:\/\/replaced.noip.me\/owncloud", "memcache.local": "\\OC\\Memcache\\APCu" } } **Are you using external storage, if yes which one:** local/smb/sftp/... no **Are you using encryption:** yes/no no **Are you using an external user-backend, if yes which one:** LDAP/ActiveDirectory/Webdav/... Webdav ### Client configuration **Browser:** firefox 45.0-1 **Operating system:** arch linux ### Logs #### Web server error log ``` Insert your webserver log here ``` no related entries, I'll add it later if you really need it. #### ownCloud log (data/owncloud.log) ``` Insert your ownCloud log here ``` really? it's 1,7MB. If you really want to, I'll post it later #### Browser log this isn't browser related. ```