Closed tomcatcw1980 closed 6 months ago
Hi mrAceT,
I modified some settings in the my.cnf so that the script ran to the end. But unfortunately I don't see any shared files. In the overview of files I see the shared folders with the size, but when I enter the folders, they are empty. Do you have any ideas?
occ files:scan + files:scan-app-data did not did the trick.
Thank you for your help.
Greetings Christian
Hi @tomcatcw1980,
First of all, it is rarely wise to add comments to a closed issue.
Second, in response:
first of all, thank you very much for your script.
You are welcome (buy me a cup of coffee ;) )
Who is the clouduser? www-data is probably not, because with sudo www-data I get the message that www-data is logically not sudoer. That's why I just left it at root.
The 'cloud user' is the user of you NextCloud installation. When you perform actions as root, all file-actions are performed as root and thus it is very likely your NextCloud installation does not have the rights to access the migrated files.. If you are unable to run as your NextCloud installation user, you will need to manually set the owner of files and folders to you nextcloud user!
What is meant by this variable? $PATH_BASE = ''; // Path to the base of the main Nextcloud directory. What has to go in there? or I have left it empty, otherwise the path is not correct and the script will not start.
Looking at the next line, the 'base path' in your case will be something like 'var/www/nextcloud'
What is the difference to this variable? $PATH_NEXTCLOUD = $PATH_BASE.'/var/www/nextcloud'; // Path of the public Nextcloud directory
I am guessing simeting like '/public' or '/public_html' (the web part of your NextCloud installation.
Is it intended in your script that it retains the shares etc.?
Good question! I expect that the NextCloud structure will retain shares. This because my script only redirects "the pointer" to the file from S3 to local (or the other way around).
PS: I just got this error message. It copied all data, then asked to continue: [...] /nc_data/appdata_ocoyq3be73jp/css/terms_of_service/6aaf-32d3-overlay.css.deps /nc_data/appdata_ocoyq3be73jp/css/theming/d71e-32d3-settings-admin.css.gzip
Continue?Y
I expect the few lines above your quote will be more reveling to me..
Copying files finished ######################################################################################### Modifying database started... PHP Fatal error: Uncaught mysqli_sql_exception: MySQL server has gone away in /var/www/nextcloud-s3-to-disk-migration/s3tolocal.php:367 Stack trace:
0 /var/www/nextcloud-s3-to-disk-migration/s3tolocal.php(367): mysqli->query()
https://github.com/nextcloud/server/issues/1 {main} thrown in /var/www/nextcloud-s3-to-disk-migration/s3tolocal.php on line 367
Yikes, does this happen every time? This means the connection to your MySql server has been lost. The part above line 367 regularly uses the $mysqli connection.. so I am thinking database corruption? place NextCloud in maintenance mode and perform some database repair actions to be sure on the table the error given in your line 367. If that doesn't do the trick, try and do a re connection (the part "connect to sql-database..." somewhere around line 80 and copy those lines directly below "Copying files finished") see if that does the trick.
You are welcome (buy me a cup of coffee ;) ) I really will. Tell me how.
But first, it would be nice, if you had any idea, where the shares have been gone.
By modifiying the my.cnf the script went till the end. When I log into the NC-instance with migrated data, the share don't work anymore. See the screenshots above Therefore I have >200 users I can't restore them manually by each users himself.
I ran occ sharing:delete-orphan-shares, then all share are not shown anymore because they are orphaned.
I would be very grateful for help
Greetings
If still possible I'dd restore the backup first (reconnecting to your S3, use the backup.sql)
I can all undone. I set up a second instance and can try and try.
What do you mean exactly: Shall I restore the backup.sql before running the script again?
Ah, you red my manual.. wow, people who read the manual and act on it really exist! ;)
So my theory the shares would remain in tact doesn't hold. Darn..
That will mean I will need to (re)build my test setup and find out how that share changes..
I need to dive into the share structure and find out what needs to be migrated :-/
About a year ago I have done some digging into shares (for my project https://github.com/GeoArchive) and need to brush up on it for my automation in creating an account in that platform (which uses NextCloud for the data).. but I haven't had the time for that yet.. in how much of a hurry are you?
AD: If you can "correct" one share and EXACTLY tell me what you've changed I'll try to build the code migration code on that, idea?
Ah, you red my manual.. wow, people who read the manual and act on it really exist! ;)
yes ;-) for my own safety. this instance is productive. If I would destroy I will be dead ;-)
About a year ago I have done some digging into shares (for my project https://github.com/GeoArchive) and need to brush up on it for my automation in creating an account in that platform (which uses NextCloud for the data).. but I haven't had the time for that yet.. in how much of a hurry are you?
I can't push you. I'm really grateful if you take care of this problem.
AD: If you can "correct" one share and EXACTLY tell me what you've changed I'll try to build the code migration code on that, idea?
You mean I should reshare an old share, that didn't work anymore? I think this isn't a good idea, because i'm in staging environment. I guess, then the user would be informed by mail that a new share on another instance is made. Can we avoid that?
If I understood you correctly you have done a test migration in a copied instance.
I need to (re)build my test setup to find out what needs to be changed. I was hoping you could tell me what needs to be changed (in the MySql database table(s) ). Based upon that I could extend the migration script.
But I will admit that'll require a bit of digging (trust me, I know ;) ) so I won't blame you if you leave that to me ;)
I have done the following:
1) I created an exact copy of the instance to be migrated. -- on the testmaschine: -- 2) Then I adjusted the variables in your script 3) Then I set it to Live "1" and let the script run. The fact that the script then terminated was probably due to the incorrect settings in my.cnf. I was able to fix this so that the script ran. 4) Since I ran everything via root, I had to manually adjust the folders to www-data via chown and then run files:scan --all and files:scan-app-data again.
I noticed the following thing, but I have no idea if it matters: The data that is downloaded from the S3 bucket apparently does not contain a file named .ocdata. I also had to copy this over from the old location to the new NC data folder. I was then able to log into the staging instance, can see all migrated data from S3. But unfortunately the shares no longer work, although they are apparently displayed.
I can delete everything on this instance again, restore the original DB of the old instance and turn everything back to the beginning. Since the S3 data is only copied, nothing is broken.
But I'm afraid I can contribute less here. I would therefore be grateful if you could somehow get to the bottom of the problem.
Hi, I did a bit of research again and made an extract of the oc_share table. The table contains the following data. In the production system the tables ist filled with data:
<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns="http://www.w3.org/TR/REC-html40">
id | share_type | share_with | password | uid_owner | uid_initiator | parent | item_type | item_source | item_target | file_source | file_target | permissions | stime | accepted | expiration | token | mail_send | share_name | password_by_talk | note | hide_download | label | password_expiration_time | attributes -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
Hi There,
I don't want to duplicate, but don't know if my issue is here better placed:
https://github.com/nextcloud/server/issues/34407#issuecomment-1938179733
Greetings Chrstian