Closed solracsf closed 3 years ago
lukasmu
commented
3 years ago That's a bit weird. The script should first create the folder structure (A) and then copy the data (B). The errors you describe actually occur in B but probably they are due to something not working properly in A.
Are you in Linux or Windows and what filesystem does your disk use? I suspect that something odd occurs between lines 89-106... Let's debug this!
solracsf
commented
3 years ago Ubuntu 20.04, XFS (RAID5), NC 20, PHP 7.4.
> composer show
aws/aws-sdk-php 3.176.3 AWS SDK for PHP - Use Amazon Web Services in your PHP project
guiguiboy/php-cli-progress-bar 0.0.4 Progress bar for PHP CLI scripts
guzzlehttp/guzzle 7.3.0 Guzzle is a PHP HTTP client library
guzzlehttp/promises 1.4.1 Guzzle promises library
guzzlehttp/psr7 1.8.1 PSR-7 message implementation that also provides common utility methods
mtdowling/jmespath.php 2.6.0 Declaratively specify how to extract elements from a JSON document
psr/http-client 1.0.1 Common interface for HTTP clients
psr/http-message 1.0.1 Common interface for HTTP messages
ralouphie/getallheaders 3.0.3 A polyfill for getallheaders.
symfony/polyfill-mbstring v1.22.1 Symfony polyfill for the Mbstring extension
symfony/polyfill-php80 v1.22.1 Symfony polyfill backporting some PHP 8.0+ features to lower PHP versions
symfony/process v5.2.4 Executes commands in sub-processesOkay just to doublecheck: In between the console output below you did not receive any errors? And there are no folders in your new data directory but only files?
Creating folder structure started...
33088/33088 [============================================>] 100.00% 00:00:00
Creating folder structure finishedConfiguration:
// Note: Preferably use absolute path without trailing directory separators
$PATH_NEXTCLOUD = '/srv/users/myuser/apps/myuser/public'; // Path of the main Nextcloud directory
$PATH_DATA = '/srv/users/myuser/data'; // Path of the new Nextcloud data directory
$PATH_BACKUP = '/srv/users/myuser/backup'; // Path for backup of MySQL databaseRun it:
# php -f transfer.php
Setting everything up started...
Setting everything up finished
Copying existing data started...
Copying existing data finished
Creating folder structure started...
Creating folder structure finished
Copying files started...
Failed to transfer: 7 (Error executing "GetObject" on "https://mybucket.s3.fr-par.scw.cloud/urn%3Aoid%3A7";
AWS HTTP error: Unable to open "/srv/users/myuser/data/" using mode "w+": fopen(/srv/users/myuser/data/): failed to open stream: Is a directory)
Failed to transfer: 64 (Error executing "GetObject" on "https://mybucket.s3.fr-par.scw.cloud/urn%3Aoid%3A64";
AWS HTTP error: Unable to open "/srv/users/myuser/data/" using mode "w+": fopen(/srv/users/myuser/data/): failed to open stream: Is a directory)Another hint, when i run this line, i get an empty set.
mysql> SELECT st.id, fc.fileid, fc.path, fc.storage_mtime FROM oc_filecache as fc, oc_storages as st WHERE st.numeric_id = fc.storage AND st.id LIKE 'object::%' AND fc.mimetype = 2;
Empty set (0.04 sec) The result of the query should indeed not be empty. Can you check your mimetypes table ('oc_mimetypes')? Maybe the entry with ID 2 is not 'httpd/unix-directory'?
solracsf
commented
3 years ago Data is on a Galera Cluster so IDs are not serialized (maybe the script could be optimized to auto-detect the correct id?)
mysql> SELECT * FROM oc_mimetypes ORDER BY oc_mimetypes.id ASC;
+-----+---------------------------------------------------------------------------+
| id | mimetype |
+-----+---------------------------------------------------------------------------+
| 1 | httpd |
| 4 | httpd/unix-directory |
| 7 | image |
| 10 | image/png |
| 13 | application |
| 16 | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| 19 | application/pdf |
| 22 | text |
| 25 | text/markdown |
| 28 | video |
| 31 | video/mp4 |
| 34 | image/jpeg |
| 37 | application/octet-stream |
| 40 | application/json |
| 43 | application/javascript |
| 46 | application/x-gzip |
| 49 | text/css |
| 52 | image/svg+xml |
| 55 | application/vnd.openxmlformats-officedocument.spreadsheetml.sheet |
| 58 | application/msword |
| 61 | application/zip |
| 64 | text/plain |
| 67 | application/vnd.ms-excel |
| 70 | application/vnd.ms-powerpoint |
| 73 | application/vnd.openxmlformats-officedocument.presentationml.presentation |
| 76 | text/rtf |
| 79 | application/font-sfnt |
| 82 | application/vnd.openxmlformats-officedocument.wordprocessingml.template |
| 85 | application/vnd.openxmlformats-officedocument.presentationml.template |
| 88 | application/vnd.oasis.opendocument.text |
| 91 | application/x-rar-compressed |
| 94 | video/quicktime |
| 97 | application/internet-shortcut |
| 100 | application/xml |
| 103 | text/vcard |
| 106 | image/tiff |
| 109 | audio |
| 112 | audio/mpeg |
| 115 | text/html |
| 118 | image/gif |
| 121 | image/bmp |
| 124 | application/x-photoshop |
| 127 | video/x-ms-wmv |
| 130 | application/vnd.ms-excel.sheet.macroEnabled.12 |
| 133 | application/msaccess |
| 136 | text/csv |
| 139 | application/x-shockwave-flash |
| 142 | application/x-ms-dos-executable |
| 145 | application/x-msi |
| 148 | audio/wav |
| 151 | application/vnd.oasis.opendocument.presentation |
| 154 | text/x-shellscript |
| 157 | application/illustrator |
| 158 | application/postscript |
| 161 | application/x-gimp |
| 164 | application/x-iwork-pages-sffpages |
| 167 | application/vnd.openxmlformats-officedocument.spreadsheetml.template |
| 170 | application/vnd.ms-powerpoint.presentation.macroEnabled.12 |
| 173 | audio/mp4 |
| 176 | application/x-php |
| 179 | application/x-iwork-keynote-sffkey |
| 181 | image/jp2 |
| 184 | image/heic |
| 187 | application/epub+zip |
| 190 | application/x-mobipocket-ebook |
| 193 | audio/mpegurl |
| 196 | video/mpeg |
| 199 | application/vnd.oasis.opendocument.text-template |
| 202 | application/vnd.oasis.opendocument.spreadsheet-template |
| 205 | application/vnd.oasis.opendocument.presentation-template |
| 208 | application/vnd.oasis.opendocument.graphics-template |
| 210 | application/vnd.oasis.opendocument.graphics |
| 213 | video/webm |
| 216 | application/vnd.oasis.opendocument.spreadsheet |
| 219 | application/msonenote |
| 221 | application/vnd.openxmlformats-officedocument.presentationml.slideshow |
| 222 | application/x-tex |
| 223 | image/webp |
| 224 | audio/ogg |
| 225 | application/x-7z-compressed |
+-----+---------------------------------------------------------------------------+
80 rows in set (0.00 sec)I've replaced 2 by 4 and script is running now.
php -f transfer.php
Setting everything up started...
Setting everything up finished
Copying existing data started...
Copying existing data finished
Creating folder structure started...
28522/28522 [============================================>] 100.00% 00:00:00
Creating folder structure finished
Copying files started...
802/83565 [>-----------------------------------------------] 0.96% 02:27:54I'll let it run until it ends and i'll provide feedback 👍
lukasmu
commented
3 years ago I hope the transfer worked for you. I also extended the query a bit to make sure this does not occur again. Thanks for helping to debug this!
Hi,
Thanks for this script. I gave it a try on a test env, but it (only) ouputs thousands of errors like these:
When I check the
/srv/users/myuser/data/folder, it has createdappdata_oc69ql39b6tras regular file, not as folder.Another example, same applies for:
/srv/users/myuser/data/__groupfoldersis created as a regular file, not folder.See also https://github.com/aws/aws-sdk-php/issues/1398