Migration from ownCloud 10.4.0

[daladim]

Hello,

I am using owncloud 10.4.0 (from Raspbian's package owncloud-files 10.4.0-1+1.1). I have used the migrator from https://download.nextcloud.com/server/installer/migrator/index.php. It works great, until I have to finish the upgrade. Running php occ upgrade greets me with the following error:

Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Set log level to debug
Turned on maintenance mode
Exception: Updates between multiple major versions and downgrades are unsupported.
Update failed
Maintenance mode is kept active
Reset log level

It looks I am not the only one to encounter this error :-( (e.g. #9 ) Is the migration process broken since 10.1 or 10.2? Nextcloud's documentation is not clear about this.

Do you think this migration could be possible? Or do I have to either stick with OC, or start with a fresh install of NC? Thanks for your work anyway

[tridoan]

Hi, did you end up figuring out a way to migrate from oC 10.4.x?

[daladim]

No, I finally gave up migrating :-(

It is actually quite easy to migrate the files (there are articles on the Internet that describe this: you basically need to copy/move/rsync the data folder, then run some occ command to re-build the database). However, I am not aware of any easy way to migrate the existing shares. Nextcloud team, do you have any idea about how to do it?

I have not searched how to migrate the contacts and calendars. I would guess that some desktop clients could be able to copy-and-paste items from one account to another, hence migrating them...

Too bad the migrator is not maintained anymore! I am still sticking to Owncloud for now

[thetrial]

Same happened to me. Tried to migrate from OC 10.4.1 … no chance. Of course, also one could just set up a virgin NC … but my setup has been grown through years with certificate, database, redis-server …

No, one does not want to try that. So it seems the crossgrade chapter of OC to NC has been closed. Well, this is a chance for OC, though a bit unintended. The only plus for me is the capability of delta syncs in OC.

[tlhonmey]

I don't even care if the web migrator works. Just having the migration page updated with what versions are compatible with what would be really nice. The current "You can't upgrade more than one version or downgrade" message doesn't give me enough data to bisect the releases and find the one that works so the only choice is to try with every... single... release...

And even then it might not work.

I mean, come on, if you've decided to not care about users migrating from owncloud past a certain point, just say so so I know not to waste my time.

[tlhonmey]

I just finished testing with every Nextcloud release from 12 to 19 with no luck. I guess wait until 20 comes out and hope for the best.

[heikohaensge]

In the hope this article might help anybody: https://julian.pustkuchen.com/en/migrating-owncloud-1041-nextcloud-18-odyssee

[Spanni26]

Any work on this issue? I try to migrate from OC 10.5, but it did not work.

[daladim]

Hi.

I finally managed to migrate from Owncloud to Nextcloud in last December. That was not easy. Basically, I used the migrator to NC12, then performed upgrades from version to version. I've been using Nextcloud smoothly since then, with my migrated data, registered users, share links, etc.

I based this migration on the article previously mentioned (https://julian.pustkuchen.com/en/migrating-owncloud-1041-nextcloud-18-odyssee). However I ran into many various problems. I'm pasting the procedure which, after several failed attempts, worked for me. This might help some of you, but I do not guarantee anything. It worked for me, but maybe the errors I had encountered were specific to my instance, maybe not.

I have annotated a few things, commented out stuff that I think were not important. This duplicates everything (data and DB) before proceeding. Never attempt an upgrade without such a kind of backup first, because the migration has little chance to work at first attempt. I had tried several times, starting back from my backup everytime.

Please read this script and the comments first. You'll probably want to run them line by line rather than from a script file, because some comments are actually requests for some manual action. You may need to fix some paths, I'm assuming everything is under /srv, that was my case but that's not the standard owncloud location.

Enjoy, I hope this will help some of you.

#!/bin/bash

set -e
export nc="/srv/nextcloud" # That's a variable, but still, double-check this script if your instance is from somewhere else
export upgd="$nc/data/updater-occ*"
export SUDO="sudo -u www-data"

echo "This will take a long time. You're advised to use 'screen' (or similar) in case you are working over SSH, to cope with possible disconnections"
echo "Press return to acknowledge"
read proceed

# Disable your web server here so that your Owncloud/Nextcloud instances are not used
# e.g. sudo systemctl stop apache2

# You should have already created a "nextcloud" database, handled by user nextcloud

echo "Clearing the database..."
mysql -vv -u nextcloud -p nextcloud <<eof
drop table if exists oc_accounts;
drop table if exists oc_activity;
drop table if exists oc_activity_mq;
drop table if exists oc_addressbookchanges;
drop table if exists oc_addressbooks;
drop table if exists oc_appconfig;
drop table if exists oc_authtoken;
drop table if exists oc_bruteforce_attempts;
drop table if exists oc_calendar_invitations;
drop table if exists oc_calendar_reminders;
drop table if exists oc_calendar_resources;
drop table if exists oc_calendar_resources_md;
drop table if exists oc_calendar_rooms;
drop table if exists oc_calendar_rooms_md;
drop table if exists oc_calendarchanges;
drop table if exists oc_calendarobjects;
drop table if exists oc_calendarobjects_props;
drop table if exists oc_calendars;
drop table if exists oc_calendarsubscriptions;
drop table if exists oc_cards;
drop table if exists oc_cards_properties;
drop table if exists oc_collres_accesscache;
drop table if exists oc_collres_collections;
drop table if exists oc_collres_resources;
drop table if exists oc_comments;
drop table if exists oc_comments_read_markers;
drop table if exists oc_credentials;
drop table if exists oc_dav_job_status;
drop table if exists oc_dav_properties;
drop table if exists oc_dav_shares;
drop table if exists oc_direct_edit;
drop table if exists oc_directlink;
drop table if exists oc_external_applicable;
drop table if exists oc_external_config;
drop table if exists oc_external_mounts;
drop table if exists oc_external_options;
drop table if exists oc_federated_reshares;
drop table if exists oc_file_locks;
drop table if exists oc_filecache;
drop table if exists oc_filecache_extended;
drop table if exists oc_files_trash;
drop table if exists oc_flow_checks;
drop table if exists oc_flow_operations;
drop table if exists oc_flow_operations_scope;
drop table if exists oc_group_admin;
drop table if exists oc_group_user;
drop table if exists oc_groups;
drop table if exists oc_jobs;
drop table if exists oc_login_flow_v2;
drop table if exists oc_mail_accounts;
drop table if exists oc_mail_aliases;
drop table if exists oc_mail_attachments;
drop table if exists oc_mail_coll_addresses;
drop table if exists oc_mail_mailboxes;
drop table if exists oc_migrations;
drop table if exists oc_mimetypes;
drop table if exists oc_mounts;
drop table if exists oc_notifications;
drop table if exists oc_notifications_pushtokens;
drop table if exists oc_oauth2_access_tokens;
drop table if exists oc_oauth2_clients;
drop table if exists oc_ownnote;
drop table if exists oc_ownnote_parts;
drop table if exists oc_persistent_locks;
drop table if exists oc_preferences;
drop table if exists oc_privacy_admins;
drop table if exists oc_privatedata;
drop table if exists oc_properties;
drop table if exists oc_recent_contact;
drop table if exists oc_richdocuments_wopi;
drop table if exists oc_schedulingobjects;
drop table if exists oc_share;
drop table if exists oc_share_external;
drop table if exists oc_storages;
drop table if exists oc_systemtag;
drop table if exists oc_systemtag_group;
drop table if exists oc_systemtag_object_mapping;
drop table if exists oc_text_documents;
drop table if exists oc_text_sessions;
drop table if exists oc_text_steps;
drop table if exists oc_trusted_servers;
drop table if exists oc_twofactor_backupcodes;
drop table if exists oc_twofactor_providers;
drop table if exists oc_user_transfer_owner;
drop table if exists oc_users;
drop table if exists oc_vcategory;
drop table if exists oc_vcategory_to_object;
drop table if exists oc_webauthn;
drop table if exists oc_whats_new;
eof

echo "Duplicating databases..."
echo "First type nextcloud's password"
echo "Then type owncloud's password"
mysqldump --single-transaction -u owncloud -p owncloud | mysql -u nextcloud -p nextcloud

cd /srv

echo "Copying app files..."
sudo rsync -var --info=progress2 --partial --delete --exclude "data/" owncloud/ nextcloud/

echo "Copying user files..."
sudo rsync -var --info=progress2 --partial --delete owncloud/data/ nextcloud/data/

cd "$nc"

function waitForFolder() {
    while true; do
        echo "Waiting for folder..."
        ls $upgd >/dev/null && return 0
        sleep 1
    done
}

echo "Clearing the previous update, in case you've tried again after a failed attempt"
rm -rf $upgd

echo "Removing old logs..."
rm -f "$nc"/data/crash* "$nc"/data/owncloud.log*

echo "Tweaking..."
rm -f "$nc/README.md"
rm -rf "$nc/ocm-provider"
rm -rf "$nc/etc"
rm -rf "$nc/assets"
rm -rf "$nc/apps-external"
# Not sure why I needed this (if it is ever important), but I needed this in my own migration. Maybe that's related to the maintenance mode value?
# cp /some/temp/place/config.php "$nc/config/config.php"

echo "Fixing permissions..."
sudo chown www-data:www-data "$nc/console.php"
sudo chown www-data:www-data "$nc/index.php"
sudo chown www-data:www-data "$nc/public.php"
sudo chown www-data:www-data "$nc/cron.php"
sudo chown www-data:www-data "$nc/CHANGELOG.md"
sudo chown www-data:www-data "$nc/status.php"
sudo chown www-data:www-data "$nc/occ"
sudo chown www-data:www-data "$nc/db_structure.xml"
sudo chown www-data:www-data "$nc/version.php"
sudo chown www-data:www-data "$nc/AUTHORS"
sudo chown www-data:www-data "$nc/COPYING"
sudo chown www-data:www-data "$nc/robots.txt"
sudo chown www-data:www-data "$nc/index.html"
sudo chown www-data:www-data "$nc/remote.php"

sudo chown -R www-data:www-data "$nc/ocs"
sudo chown -R www-data:www-data "$nc/lib"

echo "Tweaking the databases..."

# You probably want to move the root folder
# See https://doc.owncloud.com/server/10.1/admin_manual/maintenance/manually-moving-data-folders.html

# This will remove foreign keys
# see https://github.com/nextcloud/server/issues/14549
#
# Note: to find these foreign keys:
#
# select table_name,column_name,constraint_name,referenced_table_name,referenced_column_name
#   from
#   information_schema.key_column_usage
#   where
#   referenced_table_schema = 'nextcloud' and
#   referenced_table_name = 'oc_accounts';
#
# Then  show create table oc_persistent_locks;
#
# We get
#  CONSTRAINT `FK_F0C3D55B93CB796C` FOREIGN KEY (`file_id`) REFERENCES `oc_filecache` (`fileid`) ON DELETE CASCADE,
#  CONSTRAINT `FK_F0C3D55BC901C6FF` FOREIGN KEY (`owner_account_id`) REFERENCES `oc_accounts` (`id`) ON DELETE CASCADE
#
mysql -vv -u nextcloud -p nextcloud <<eof
update oc_storages set id="local::/srv/nextcloud/data/" where id="local::/srv/owncloud/data/";
update oc_accounts set home = REPLACE(home, "/srv/owncloud/data/", "/srv/nextcloud/data/");

alter table oc_persistent_locks drop foreign key FK_F0C3D55BC901C6FF, drop owner_account_id;
alter table oc_persistent_locks drop foreign key FK_F0C3D55B93CB796C, drop file_id;
eof

# Here, you should ensure that oc_persistent_locks is empty

# This is more-or-less what is described at https://nextcloud.com/blog/get-up-to-date-with-the-new-nextcloud-updater/
if [ -d "$nc/updater" ]; then
    rm -rf "/tmp/updater-from-owncloud"
    mv "$nc/updater" "/tmp/updater-from-owncloud"
fi
$SUDO mkdir "$nc/updater"

echo "Getting the migrator..."
$SUDO wget -O "$nc/updater/index.php" https://download.nextcloud.com/server/installer/migrator/index.php
# Downloading the zip failed for me, I'll rather do it myself
echo "Patching it..."
sed -i 's $updater->downloadUpdate() //$updater->downloadUpdate() ' "$nc/updater/index.php"

echo "Restart your web server for Nextcloud, thens tart the upgrade from a web browser (https://your.nextcloud.url/updater/)"
echo "Press return after the upgrade has been initiated"
read suite
waitForFolder

cd $upgd
$SUDO mkdir -p downloads
# This is the latest version supported by the official migrator
$SUDO wget -O downloads/nextcloud-12.0.12.zip http://download.nextcloud.com/server/releases/nextcloud-12.0.12.zip

echo "When the upgrade is done, keep maintenance mode on and press return"
echo -n "  ...[press return] "
read suite

# Older owncloud and nextcloud versions only support PHP7.2, newer ones only support php7.3)
# Depending on your distro, you may have to do stuff like this
# echo "Setting the PHP version..."
# sudo systemctl stop apache2
# sudo a2dismod php7.3
# sudo a2enmod php7.2
# sudo systemctl restart apache2

echo "Patching the second updater"
$SUDO sed -i '/public function isUpgradePossible/a\ \ \ \ \ \ \ \ \ return true;' "$nc/lib/private/Updater.php"

echo "Running the upgrade..."
$SUDO php7.1 "$nc/occ" upgrade

# TODO: uninstall the following apps which are now disabled? Do we care?
# calendar
# contacts
# market
# files_mediaviewer
# tasks

function do_upgrade_from() {
    from="$1"
    to=$((from + 1))
    echo -e "$(date) \033[1mUpgrading from $from to $to...\033[0m"

    PHP="php7.3"
    if [ "$from" -lt "15" ]; then
        PHP="php7.1"
    fi
    alias php="$PHP"

    cd "$nc"
    backup=""
    if [ "$from" -ge 19 ]; then
        backup="--no-backup"
    fi
    # Using yes | ... turns the updater into a non-interactive one, which sucks because it continues with occ upgrade, sometimes with the wrong PHP version...
    $SUDO $PHP updater/updater.phar $backup --ansi
    if [ "${PIPESTATUS[0]}" -ne 0 ]; then
        false
    fi
    $SUDO $PHP occ upgrade
    if [ "${PIPESTATUS[0]}" -ne 0 ]; then
        false
    fi

    echo "Fixing stuff, because why not"
    if [ "$from" -ge "13" ]; then
        $SUDO $PHP occ db:add-missing-indices
        $SUDO $PHP occ db:convert-filecache-bigint
    fi
    if [ "$from" -ge "18" ]; then
        $SUDO $PHP occ db:add-missing-columns
    fi
    #  php ./occ maintenance:repair -vv # not needed, run at every update
    #  php ./occ files:scan --all -vv   # maybe once at the end?

    echo -e "$(date) \033[1mDone upgrading to version $to...\033[0m"
    echo "You could check for misconfiguration warnings"
    echo "----------------------------------"
    echo "----------------------------------"
    echo "----------------------------------"
    echo "----------------------------------"
    echo ""
    echo ""
    echo ""
}

do_upgrade_from 12
do_upgrade_from 13
do_upgrade_from 14
do_upgrade_from 15
do_upgrade_from 16

# complains about oc_direct_edit and oc_user_transfer_owner already exists
# Since they were empty, I dropped the tables:
# mysql -vv -u nextcloud -p nextcloud <<eof
# SELECT * FROM oc_direct_edit;
# DROP TABLE IF EXISTS oc_direct_edit;
#
# SELECT * FROM oc_user_transfer_owner;
# DROP TABLE IF EXISTS oc_user_transfer_owner;
# eof

# Second time I did it, I lacked a column:
# mysql -vv -u nextcloud -p nextcloud <<eof
# ALTER TABLE oc_flow_operations ADD COLUMN entity VARCHAR(255) NOT NULL;
# eof
do_upgrade_from 17
do_upgrade_from 18
# maybe there are newer NC version that have been released since I migrated

echo "Workaround for a bug: emptying the trash..."
$SUDO $PHP occ trashbin:cleanup --all-users

echo "You can now check file permissions/ownership, restart your web server and install required Nextcloud apps"