search

MichaIng / DietPi

Lightweight justice for your single-board computer!
https://dietpi.com/
GNU General Public License v2.0
4.89k stars 498 forks source link

DietPi-Software | ownCloud/Nextcloud: Properly handle multiple "local::" db entries #5626

Closed BristolPiggy closed 2 years ago

BristolPiggy commented 2 years ago

Details:

Joulinar commented 2 years ago

same has been opened on our forum https://dietpi.com/forum/t/following-a-system-update-system-broken-and-i-have-not-been-able-to-restore-it/13680

MichaIng commented 2 years ago

Hmm, can you show:

mysql -e 'select * from nextcloud.oc_storages'

Indeed on uninstall a backup of database and Nextcloud files is created and loaded automatically on an install if present.

The /var/www/nextcloud/config/config.php that was recovered is mostly empty now, is it? Do you have a backup of it?

BristolPiggy commented 2 years ago

Certainly, here is the output to: mysql -e 'select * from nextcloud.oc_storages'

+------------+---------------------------------------------+-----------+--------------+
| numeric_id | id                                          | available | last_checked |
+------------+---------------------------------------------+-----------+--------------+
|          1 | home::admin                                 |         1 |         NULL |
|          2 | local::/mnt/dietpi_userdata/nextcloud_data/ |         1 |         NULL |
|          3 | home::John                                  |         1 |         NULL |
|          4 | home::TheAdminBear                          |         1 |         NULL |
|          6 | local::/mnt/2TBHDD/'nextcloud sync'/        |         0 |   1651181784 |
|          7 | local::/mnt/2TBHDD/nextcloud sync/          |         0 |   1655554150 |
|          8 | local::/120GBTVShows/                       |         0 |   1655281824 |
|         10 | local::/mnt/ExtraStorage/                   |         1 |   1655651445 |
|         11 | local::/mnt/BackUpDrive/                    |         1 |   1655650999 |
|         12 | local::/mnt/ExtraStorage/LongTermStorage/   |         0 |   1657159444 |
+------------+---------------------------------------------+-----------+--------------+

I have looked through my backup copies made previously and every config.php file is the same. It opens with the instance reference and that is it, nothing else:

#<?php
$CONFIG = array (
  'instanceid' => 'ocy8smh5k3st',
);
BristolPiggy commented 2 years ago

Certainly, here is the output to: mysql -e 'select * from nextcloud.oc_storages'

+------------+---------------------------------------------+-----------+--------------+
| numeric_id | id                                          | available | last_checked |
+------------+---------------------------------------------+-----------+--------------+
|          1 | home::admin                                 |         1 |         NULL |
|          2 | local::/mnt/dietpi_userdata/nextcloud_data/ |         1 |         NULL |
|          3 | home::John                                  |         1 |         NULL |
|          4 | home::TheAdminBear                          |         1 |         NULL |
|          6 | local::/mnt/2TBHDD/'nextcloud sync'/        |         0 |   1651181784 |
|          7 | local::/mnt/2TBHDD/nextcloud sync/          |         0 |   1655554150 |
|          8 | local::/120GBTVShows/                       |         0 |   1655281824 |
|         10 | local::/mnt/ExtraStorage/                   |         1 |   1655651445 |
|         11 | local::/mnt/BackUpDrive/                    |         1 |   1655650999 |
|         12 | local::/mnt/ExtraStorage/LongTermStorage/   |         0 |   1657159444 |
+------------+---------------------------------------------+-----------+--------------+

I have looked through my backup copies made previously and every config.php file is the same. It opens with the instance reference and that is it, nothing else:

#<?php
$CONFIG = array (
  'instanceid' => 'ocy8smh5k3st',
);

Sorry, am I meant to hit a button or select something to show that I have answered? I think Joulinar can vouch for me that I'm pretty new to this and learning as I go.... but at my usual snails pace (thank for your understanding in advance).

MichaIng commented 2 years ago

I'm confused about all the different local:: entries. Are those all valid? dietpi-software expects one such entry only.

BristolPiggy commented 2 years ago

I'm confused about all the different local:: entries. Are those all valid? dietpi-software expects one such entry only.

I'm not 100% sure by what you are asking, but I'll give it a go in answering.

Looking on my system /mnt/dietpi_userdata/nextcloud_data/ is still a location on my system and I am able to cd into it. The same goes for /mnt/BackUpDrive (stores only backup files, no live ones).

However the others were past iterations that have since been removed- although obviously not well enough it seems.

I am still confused by how the config.php file has managed to be wiped clean. I have tried re-writing one but I don't know critical information like the name of the data file, user, passwords etc.

TBH my BackUpDrive is 2 TB in size, and it seems that despite me opting for every type of backup option going... it's worth nothing (am finding this all a bit much and wondering if privacy is worth this level of stress).

BristolPiggy commented 2 years ago

So since I sent you a reply a week ago I found out how to delete off the erroneous databases but I am still unable to find out how I can recover my data. I have been looking into how to input data into a new mariadb, but unsure how to perform a sql dump- or if it is even possible at present. If so I'm hoping a fresh install of everything will provide me with the system I had over a month ago, and then let me return the data from the old MariaDB to the new one (but I keep reading different opinions as to whether this would be possible).

MichaIng commented 2 years ago

Looking on my system /mnt/dietpi_userdata/nextcloud_data/ is still a location on my system and I am able to cd into it. The same goes for /mnt/BackUpDrive (stores only backup files, no live ones).

So I guess the actual Nextcloud data which you access through its web interface are in /mnt/dietpi_userdata/nextcloud_data? Are the other storages added to Nextcloud in any way, e.g. with the external storages app as additional local storage?

BristolPiggy commented 2 years ago

I resolved this issue by go back to basics and learning how to spin up a nextcloud instance running off a ubuntu server. From there I used various websites, but mostly that of mariadb to learn about how the database could be accessed and the data held in all my backups accessed, long story short: I needed a password held in the ever important config.php file and without this the 2TB of back up data was worthless.

Luckily I found at some point in time I had set the nextcloud instance to back up my main data to a drive running on my router through FTP, so managed to get the data from there. The data is not in mariadb format so learnt about rsync to transfer the data.

My plan was to wipe nextcloud from my dietpi completely, and then spin up a new Nextcloud instance using all the new knowledge I had gained from this "episode", but whilst setting up nextcloud I managed to brick it some how. An error keeps coming up stating "Your composer dependencies require a PHP version >8.0.2 You are running 7.4.30". I have tried installing all new PHP modules but this hasn't changed the situation, so I guess this is the end of my time with dietpi.

The concept of Dietpi is a good one, but being able to create all these amazingly complex systems without understanding the backbones of them gives you a false sense of belief. People need to understand the fundamentals behind the systems they are creating, understand what programmes are needed to make them their creations possible eg, mysql & maria databases, apache2, and PHP's. The reason being that when things go wrong you are left on your lonesome to try and figure out how to fix this complex pile of syntax and instruction through scouring the internet and online forums, reading article after article that never supply all the information you need, skipping over the difficult topics with "but that's too big a topic for this video". At this point you may find your anger- that has been building for four weeks , that's four weeks of stress through fear you have lost a lifetimes worth of memories- which when vented gets you kicked off a site, like, for example YouTube!.

There are too many issues with the OS for one person to help, whilst maintaining the OS, improving it, and all the rest they have to do. The amount Michalng does is astounding for one person but still never enough for the person going through a melt down (you). The forum is no help.... everyone there seems to be having a melt down and too few are able, let alone willing to offer help. So I guess I'm saying that Dietpi may seem like a good idea at first, but you will do yourself a much better service if you put the hard work in at the beginning and learn the base systems like mysql/mariadb, apache, PHP, etc etc.

Joulinar commented 2 years ago

Feel free to join our little project if you feel there is not enough support from our side. Volunteers are always welcome. It would be great if you could share your knowledge and assist others who having issues.

Expecting everybody to learn everything yourself is a little bit of an interesting approach but will not work in reality. On none OS. Therefore we would like to welcome you to improve our code, develop new features or create guide's on how to setup backup concepts.

MichaIng commented 2 years ago

being able to create all these amazingly complex systems without understanding the backbones of them gives you a false sense of belief. People need to understand the fundamentals behind the systems they are creating, understand what programmes are needed to make them their creations possible eg, mysql & maria databases, apache2, and PHP's.

I agree that at some point, based on use case, one needs to start learning these things when running a server. I wouldn't be developing DietPi, if I didn't go through all these things manually 7 years ago, setting up ownCloud for my personal use and being a perfectionist, notoriously digging deep into every aspect. But IMO it is not correct (and wouldn't be practical) that everyone with every use case needs to learn everything, agreeing with Joulinar on this.

Please don't take it as offence, but the additional issues you ran into, and knowledge required to solve it, seems to be based on an uncommon or unintended setup of the software. There is no general problem with this, but that it bears the risk to run into more complex issues is just a logical consequence:

Using Nextcloud just the way it is installed via dietpi-software (which follows official best practice instructions), on system level I mean, wouldn't have caused any of the issues, and works well for thousands of users without deep knowledge of the underlying setup. Of course there are use cases which require a more customised setup, and in turn deeper knowledge, that is clear, but something an automated installer for such server software stacks cannot cover.

The forum is no help.... everyone there seems to be having a melt down and too few are able, let alone willing to offer help.

That is a false and unfair statement. We address nearly every issues in short time, aside of Joulinar and me there are other very helpful community members with their specific knowledge fields. Not everyone is Nextcloud or database expert, but the only reason why you didn't get help on the forum is that you opened just the same issue here on GitHub. Dealing with the same issue on two platforms in parallel is just a waste of time, not just for us, but also for you. You opened just this single and duplicate issue on our forum, please be careful when drawing conclusions about our support quality from this.

MichaIng commented 2 years ago

Okay I have to take back the part about local::. Indeed adding a local directory via external storages app adds another such entry (and also removes it once the storage is "disconnected" from Nextcloud UI). Is this what you did?

+------------+---------------------------------------------+-----------+--------------+
| numeric_id | id                                          | available | last_checked |
+------------+---------------------------------------------+-----------+--------------+
|          1 | home::admin                                 |         1 |         NULL |
|          2 | local::/mnt/dietpi_userdata/nextcloud_data/ |         1 |         NULL |
|          3 | local::/mnt/dietpi_userdata/Music/          |         1 |   1659875171 |
+------------+---------------------------------------------+-----------+--------------+

What we hence can do to fix reinstalls for this case:

EDIT: Ah, the issue btw only appears if the config.php (installed or as backup) is missing or incomplete, like yours was. We replace the "old" entry explicitly, but in your case there was no old entry present:

local datadir_old=$(grep -m1 "^[[:blank:]]*'datadirectory'" "$config_php" | mawk -F\' '{print $4}')
G_EXEC mysql -e "update nextcloud.oc_storages set id='local::$datadir/' where id rlike 'local::$datadir_old';"

This is indeed harder to repair as one needs to recover a bunch of config.php entries manually based on info from the database. I'm wondering how the instance worked without complete config.php as all your backups of this file are nearly empty as well. And such an config.php should lead to 5xx server errors on every attempt to access Nextcloud.

BristolPiggy commented 2 years ago

I have no idea where to start (this will take a few re-writes as I will no doubt run my mouth/ fingers off at times).

You argue that it is not needed for everyone to understand all the "ins and outs" of a programme as this is not practical. I agree in part on this, I do not need to experience all the set backs, failures, problems, and dead ends you and others have that have brought you to the place your knowledge is today. I need to know what those avenues were are so I can spend my time pursuing other routes, this is after all the basis of the acquisition of knowledge.

However, I firmly believe if we blindly allow others to take control of aspects of our lives then we are left at their whim when things go wrong and in life you can only ever rely on yourself*. This is exactly where I found myself with this situation. I ran an update as when I logged in there was a coloured warning to do so, at which point my system broke. The guides on what to do in such situations repeated to first look to see if others had experienced this (and then reading and re-reading in every forum request "has this not been answered in forum post...", which you read and are still left with no answer) then gave specific instructions how post must be set out. Completing just the request for help was an effort in itself... as I said, if you don't learn the basics, when things go wrong, simply the action of completing a forum request becomes daunting.

[On the note of putting in two requests for help... if both github and the dietpi forum perform the same function, then get rid of one! Oh and I didn't put two requests in, I put over 10. These included here, the dietpi forum, YouTube channels, Reddit, Facebook, Twitter, Discord, and any other site where there were articles stating "dietpi is the easy option". I was stressed and upset.]

In respect to me not answering your questions I reply that you need to learn how to ask them. If you make a statement of understanding, adding a question mark does not make it a question (it just makes it grammatically incorrect):

"So I guess the actual Nextcloud data which you access through its web interface are in /mnt/dietpi_userdata/nextcloud_data"

There was nothing to answer, so I didn't! Had you typed for example:

"Is the main data, which you see when you access through the web interface (usually under the 'Files' tab), located in the folder /mnt/dietpi_userdate/nextcloud_data/"username"/files? If not, where is it stored? And regarding the other storages, can you remember how you added them to Nextcloud? For example, did you connect external drives to your Raspberry Pi and then use the external storage app to connect it or, perhaps, did you create new directories (using the mkdir command) on your storage drive before changing the config.php file to look for the data under the new mount points? Or some other means?"

Either way, the experience has been had. I have come away from it with more knowledge than I had before. I accept from your position it is not an easy one when it comes to helping people Especially those who are maybe stressed and upset, because for every 1 person who does what you ask regarding trying to help themselves and is at the end of their tether, there will always be the 9 others that are too lazy and privileged shouting just as much for help. [I get it, honestly I do, I worked in IKEA for 10 years and was driven round the twist by the never ending deluge of being asked "where's the way out?" by the stupid (insert your favoured derogatory term here) but every now and then, you would find yourself, face to face, with someone genuinely lost and confused, who had followed the yellow line, who had gone past you three times already, was descending into a panic attack and really needed your help**)

Do what you like with this diatribe. Either learn from it and question whether there are ways in which you may be able to help people better or write it off as someone you disagree with completely, a loon, "one of those you can never help", and carry on as you were before.

Either way, I'm only closing this just one more time... if you open it again, then that's how it stays!

*I have come to understand this mind set is due to my personality disorder, schizoids don't rely on others as typically in the past we have been let down or suffered pain (either or both mental and physical) when we required help. As much as you will never be able to understand, let alone try to change your personality to fall in line with mine, I too find it incomprehensible to understand your personality (on which like everyone, all view points, behaviours, thought processes.... life is set by). On the good side though, of the 10 Personality Disorders, Schizoid is the least understood and more rare of the bunch (mainly because we keep to ourselves lol).

** in such situations the answer was always straight out through a fire escape and screw the commotion it causes. I've experienced both sides and no amount of talking can bring a person off that ledge. They need to see the sky, so get them outside asap!

MichaIng commented 2 years ago

However, I firmly believe if we blindly allow others to take control of aspects of our lives then we are left at their whim when things go wrong and in life you can only ever rely on yourself*.

Well, everyone has full control, all scripts are open source (as scripts anyway), all commands are transparently printed when executed. So it is perfectly possible to review everything immediately or on demand when an issue occurs, without the need to execute all steps manually.

However, it has no point to argue on this. It is an individual decision with ups and downs whether you use automated installers or do things yourself. I personally also do most things manually myself, since I usually want to understand in detail what happens, also as a burden of the perfectionist I am. But I'm coding the findings into DietPi's scripts for others to benefit from it, which do not have the time, interest or ability to do everything manually.

Think of Windows or macOS installers, where you not even have the option to know in details what happens, and still, Windows and macOS have their reason and point to exist, based on individual use cases.

Either learn from it and question whether there are ways in which you may be able to help people better or write it off as someone you disagree with completely, a loon, "one of those you can never help", and carry on as you were before.

You basically gave up before we were able to help you. Indeed your situation with a lost config.php, even in all backups (not sure how this is possible, without recognition long before), is not trivial to repair, but it is possible. And such situation is very rare. I have no percentage, but we are able to help most users on forum and GitHub to solve their issue. I say "help users to solve their issue", since it is often a combination of hints or directions our end, with the final solution found/done by users, learning something that way, just not in the first place but on demand. Of course there are always such situations which cannot be solved, or where the solution implies fore effort than starting from scratch, but this is true regardless whether you setup the system yourself of whether you used scripts from others for this.

However, I'm happy that you found your favourite way of setting up your system. One just cannot assume that this is the preferred or even possible way for everyone 😉.