Debconf setting for which user account to run the system service

[benjaoming]

Imported, refined and improved the implementation from KA Lite.

The change comes without changelog, it's unreleased. ~I think we should import it into master and release it as an improvement for 0.9.3 as well.~ 0.9.3 was skipped.

This is destined for 0.10.0

[benjaoming]

Test installer here: https://files.slack.com/files-pri/T0KT5DC58-FBFB9EWA3/download/kolibri_0.10.0_b7-0ubuntu1_all.deb

[lyw07]

@benjaoming somehow it says the deb installer is not found through the link...

[benjaoming]

@lyw07 I removed it -- will be making changes and uploading a 0.9.3 build instead.

[benjaoming]

This is now released in kolibri-proposed and ready for testing

https://launchpad.net/~learningequality/+archive/ubuntu/kolibri-proposed

[benjaoming]

One more thing remaining:

To ensure that we don't mess up old <0.10 installations upgrading: This is handled by keeping the kolibri user.

We will need an instruction somewhere*) telling people to run apt-get purge kolibri and then manually copy /var/kolibri/.kolibri to /home/username if they want access to USB devices.

*) Somewhere = community forums, documents, release notes - but which is better?

[benjaoming]

So the intended outcomes are:

1. User installs 0.10 on a clean system: A prompt is displayed, asking which user to run the system service as. The default is to use the first desktop user.

2. User upgrades from 0.9=>0.10: The system service remains unchanged and runs as kolibri. None of the channel data etc. is touched or moved. *)

3. User upgrades from 0.10=>0.10+: The system service remains unchanged and runs under the user originally chosen. No prompt is displayed.

4. User purges kolibri and then re-installs: Same as 1.

*) Motivation: Moving data is dangerous. We cannot know if /home is on the same partition, nor how much space is available. Large processes like copying 20 GB should always come with progress feedback, which is not possible in this case. The user might run out of battery, close the terminal etc. leaving the system in an invalid state. So it's entirely infeasible to support such a scenario, and people who have installed 0.9 are likely to want to keep their data seamlessly, unharmed and unchanged during an upgrade!

In the eventual case that someone upgrading from <0.10 to 0.10+, and they want to import/export with USB, they will have to change the user name for the Kolibri system service:

# Stop kolibri
sudo systemctl kolibri stop
# Move data to your desktop user:
sudo mv /var/kolibri/.kolibri /home/$USER/.kolibri
# Change ownership
sudo chown -R $USER /home/$USER/.kolibri
# Change the username configuration
sudo echo -n $USER > /etc/kolibri/username
# Start kolibri again
sudo systemctl kolibri start

[radinamatic]

The new 0.10.0~b8-0ubuntu1 was working as expected on systems without previous versions: during the installation it selects the currently active desktop user as the default owner of Kolibri, therefore has full access to attached USB drives, and can import content offline correctly.

However, in systems that have had a previous version of Kolibri installed, the new desktop user as owner of Kolibri leads to the loss of previous content and user database. On another hand, I did try the manual copy of the ./kolibri folder inside /var/kolibri/ to the home of my desktop user, and only then proceeded to upgrade to 0.10.0b8. Everything was seamless and after the upgrade Kolibri picked up the previous content and database without issues.

When I tried the 0.10.0~b8-0ubuntu2, it was supposed to slide over the user account selection, and choose the previous kolibri owner without prompting the user, but is still displaying that step. @benjaoming will review and check why, and release another proposed.

[indirectlylit]

We will need an instruction somewhere*) telling people to run apt-get purge kolibri and then manually copy /var/kolibri/.kolibri to /home/username if they want access to USB devices. Somewhere = community forums, documents, release notes - but which is better?

...

User upgrades from 0.9=>0.10: The system service remains unchanged and runs as kolibri. None of the channel data etc. is touched or moved.

Recommendation:

1. Create a dedicated page in our support docs:

   Upgrading from 0.9.x to 0.10.x on Debian and switching users

   In there, we describe how to do the transition from kolibri user to standard user. (Might be apt-get purge kolibri or chown? There are two strategies described above)

2. When user upgrades from 0.9=>0.10 debian: proceed as described above, but pause the installation process and include a message like:

   Your installation of Kolibri uses a 'kolibri' user; however that user may not be able to access mounted drives which prevents importing via USB. If you would like to switch to using a standard user account, please follow these instructions: http://kolibri.readthedocs.io/en/latest/upgrade_debian_09x_to_010x.html

With this strategy we ensure that we reach all affected users, while at the same time not bothering users of other operating systems with irrelevant information.

[benjaoming]

Create a dedicated page in our support docs

We could introduce an item in the FAQ: Problems connecting a USB device

(because people don't always read instructions during installations and upgrades, and they might not realize the issue before they are actually failing to access a USB device)

It's also to keep the normal install/upgrade docs as clean as possible. The instruction is relevant for people whom a) started using Kolibri before 0.10, and b) need to import/export channel data via USB afterwards.

When user upgrades from 0.9=>0.10 debian: proceed as described above, but pause the installation process and include a message like: (...)

We will have to verify what state it's paused in. If it is after the system service is stopped, that's fine, then the user could open up another terminal window and follow the instructions. However, if they close the terminal window, they will end up with a package with an unfinished install state and then have to run apt-get install -f or something like that to resume. Good to add something about (...) after finishing the installation.

Suggesting 2 things then:

1. That we add an FAQ item about USB devices. I've expanded it a bit, so you know why this candidates to its entire own chapter :)

Problems importing and exporting with USB devices

Kolibri needs read and write access to your USB devices.

User account access: Running Kolibri from a normal desktop instance of your operating system is safe, but if you have either: 1) Installed Kolibri in your own environment running as a non-desktop user (for instance UWSGI) or b) you have upgraded Kolibri on Debian from a version <0.10, then you can follow these instructions to move Kolibri from one system user account to another. In order to grant access to USB devices for other accounts, we refer to the documentation of your operating system.

Write mode: Some USB devices will experience problems when they are unplugged from the computer in an "unclean" way. If you are denied access to write to the device, look for options to "fix" or "repair" the file system.

Inconsistent or broken devices: It will happen that SD storage "expires". So always keep your receipts! When reading and writing content data of many gigabytes, there is an increased chance that you will discover the limits of the device's quality.

Inconsistent or broken data: Copying data can take a long time. If you do not see a final success message after the copy is done, do not assume that the data has been imported/exported correctly. Instead, start over. Otherwise you risk inconsistent and malfunctioning content data.

2. To change the documentation text from an "upgrade note" about 0.x=>0.10, and instead to be about changing the user running Kolibri in general. This will make the documentation more universal in understanding the basic layout of system data.

So the text for the installation step could be:

You are updating from an older version of Kolibri and we recommend that you switch the user account that runs the system service to your normal desktop user. This allows Kolibri to access your USB devices for importing and exporting content channels.

After finishing the installation/upgrade, follow these instructions to change the user setting and move your previous content channels and database:

    https://learningequality.org/r/deb-installer-upgrade-note-system-user

The chapter about changing the system user would be nested under Install Kolibri > Debian/Ubuntu:

Changing system user

You may need to change the system service to run with the permissions of a different user account. Before 0.10, we used "kolibri" as the default account, but since 0.10 we have changed to expect a desktop user's account for USB device access.

To change the user, you need to change the system service configuration, move the data and change permissions of the data such that it is owned by the new user:

# Stop kolibri
sudo systemctl kolibri stop
# Move data to your desktop user:
sudo mv /var/kolibri/.kolibri /home/$USER/.kolibri
# Change ownership
sudo chown -R $USER /home/$USER/.kolibri
# Change the username configuration
sudo echo -n $USER > /etc/kolibri/username
# Start kolibri again
sudo systemctl kolibri start

[benjaoming]

Alright, the new upgrade note is implemented. I tested that:

1. It doesn't display during new installations
2. It displays during upgrades

[benjaoming]

If manual tests are positive, I would conclude that this PR is done and ready for merge: @radinamatic is doing updates for the docs regarding changing system user over in https://github.com/learningequality/kolibri-docs/pull/28 and there is a separate issue tracking the FAQ entry about USB device troubleshooting.

[radinamatic]

Trying to run the instructions for system service ownership change I may be missing something.

This was on system updated from 0.8 to 0.10 where kolibri user is still the one with permissions for system service, and the ./kolibri folder is in /var/kolibri/. But when I run the first line:

osboxes@osboxes:~$ sudo systemctl kolibri stop
[sudo] password for osboxes: 
Unknown operation kolibri.
osboxes@osboxes:~$ 

[indirectlylit]

Interesting. Looking at the docs, I actually don't even see a reference to systemctl or service, which (from my understanding) is how users are supposed to interact with the Kolibri service when it's daemonized in Debian.

[indirectlylit]

In @lyw07's recent PR, we tell people to use this form:

sudo su kolibri -c 'kolibri stop'

[radinamatic]

Ok, so far I have the first 3 steps working:

osboxes@osboxes:~$ sudo service kolibri stop
[sudo] password for osboxes: 
osboxes@osboxes:~$ sudo mv /var/kolibri/.kolibri /home/osboxes/.kolibri
osboxes@osboxes:~$ sudo chown -R osboxes /home/osboxes/.kolibri

But then the next step fails:

# Change the username configuration
osboxes@osboxes:~$ sudo echo -n osboxes > /etc/kolibri/username
bash: /etc/kolibri/username: Permission denied
osboxes@osboxes:~$

[benjaoming]

@radinamatic quick note: Any issues with the system service are 99% likely to be unrelated, so that would be better to post elsewhere. It would be helpful to have that feedback in the #testing channel or similar, so we can resolve these things outside of PR reviews etc. :)

This error:

osboxes@osboxes:~$ sudo systemctl kolibri stop
[sudo] password for osboxes: 
Unknown operation kolibri.
osboxes@osboxes:~$ 

...was because the command was wrong. It is sudo systemctl stop kolibri.

[radinamatic]

Thanks @benjaoming, I was missing the proper order... 🤦‍♀️

However, the sequence still fails at Change the username configuration step:

osboxes@osboxes:~$ sudo systemctl stop kolibri
[sudo] password for osboxes: 
osboxes@osboxes:~$ sudo mv /var/kolibri/.kolibri /home/osboxes/.kolibri
osboxes@osboxes:~$ sudo chown -R osboxes /home/osboxes/.kolibri
osboxes@osboxes:~$ sudo echo -n osboxes > /etc/kolibri/username
bash: /etc/kolibri/username: Permission denied
osboxes@osboxes:~$ 

I tried closing the Terminal window and restarting the system, but the /etc/kolibri/username is still owned by the root and not my desktop user.

[radinamatic]

Basically, when the service is restarted, setup wizard appears again, and the previous DB is not loaded even though it is now in the Home of my desktop user.

[radinamatic]

With help from @jamalex and @indirectlylit, we now have two alternatives (see below). Both work, but the first one creates some unexpected Terminal output issues (see the image below).

Let's decide with which to go on, so I can finish updating the docs!

osboxes@osboxes:~$ sudo systemctl stop kolibri
[sudo] password for osboxes: 
osboxes@osboxes:~$ sudo mv /var/kolibri/.kolibri /home/osboxes/.kolibri
osboxes@osboxes:~$ sudo chown -R osboxes /home/osboxes/.kolibri
osboxes@osboxes:~$ echo -n osboxes | sudo tee /etc/kolibri/username
osboxes@osboxes:~$ sudo systemctl start kolibri
osboxes@osboxes:~$ 

osboxes@osboxes:~$ sudo systemctl stop kolibri
[sudo] password for osboxes: 
osboxes@osboxes:~$ sudo mv /var/kolibri/.kolibri /home/osboxes/.kolibri
osboxes@osboxes:~$ sudo chown -R osboxes /home/osboxes/.kolibri
osboxes@osboxes:~$ sudo sh -c 'sudo echo -n osboxes > /etc/kolibri/username'
osboxes@osboxes:~$ sudo systemctl start kolibri
osboxes@osboxes:~$ 

[benjaoming]

@radinamatic sorry, writing sudo [command] > /some/file is an old habbit... a mistake I often make :)

Please do go for the second option, still. It's a bit more readable IMO. I often do sudo bash and strip commands from the "sudo" part when I execute them.

Since all this is a documentation issue, I'm assuming this PR is good to go!

Thanks! :)