Closed kneiser closed 7 years ago
I wrote the script for Nextcloud 10 I think. Let me try to adjust it
is it possible to write the script for both Nextcloud 9 and 10?
Hmm it should work in Nextcloud 9 as well, but you need to run it after updating to Nextcloud 9.
@nickvergessen the fatal error "IS" keeping Nextcloud 9 from upgrading dude. When the error is raised there is no indication that the upgrade was successful. If I go to the supposed upgraded instance Nextcloud is still thinking it has to do the upgrade process. If you want me to try fine, I would not be suprised to see Nextcloud 9 still requiring to do the upgrade.
This script is intended to repair data in ownCloud 8.2.1 data before upgrade to Nextcloud 9.0.54.
This is wrong. The problem is in 8.2 the stuff does not exist, and therefor can not be fixed. E.g.:
Fixing missing records for app: systemtags
This app is new in 9.0, so how is the script event trying to fix missing records? Or did you update your files already? Again, all the script does is fix an brokenly update 9.0 instance.
What you can do is:
oc_trusted_servers
)We will have a look to somewhat better detect this automatically
@nickvergessen You will see from my previously posted incidents, per your "What you can do is:", I had already did that. Eventually it leads raising another fatal error to one of the existing tables that already has data in it. This was suggested in past forum posts by other users having the same issue. As I mentioned in that incident (#1669) I was not willing to lose any of my data stored in existing tables.
I am not sure what you mean did I update my files already. Other than making sure that the database instance was empty before the restore from backups, I did nothing, other than perform the recommended upgrade steps per the Nextcloud Manual Update process in the on-line documentation.
Can you run the following script:
https://gist.github.com/nickvergessen/94b57082a5ab83e47bbbd1bef6d13f66#file-app-state-detection-php
As well as post your config report:
occ config:list
That would really help a lot to make sure we cover all the right cases.
@nickvergessen you want this done on the ownCloud 8.2.1 instance right?
On the instance that makes the trouble, yes
@nickvergessen here you go, I hope the results are of help:
[OK] App admin_audit is not found, not installed and disabled [OK] App comments is not found, not installed and disabled [OK] App dav is not found, not installed and disabled [Error] Table addressbooks of app dav exists although the app is not installed [Error] Table addressbooks of app dav should not exist but has 5 rows [Error] Table cards of app dav exists although the app is not installed [Error] Table cards of app dav should not exist but has 240 rows [Error] Table addressbookchanges of app dav exists although the app is not installed [Error] Table addressbookchanges of app dav should not exist but has 11520 rows [Error] Table calendarobjects of app dav exists although the app is not installed [Error] Table calendarobjects of app dav should not exist but has 1578240 rows [Error] Table calendars of app dav exists although the app is not installed [Error] Table calendars of app dav should not exist but has 17360640 rows [Error] Table calendarchanges of app dav exists although the app is not installed
the PHP script freezes at the point above and does not exit. I had to CTRL/C it...
{
"system": {
"instanceid": "oc5zrx7pkm6x",
"passwordsalt": "***REMOVED SENSITIVE VALUE***",
"secret": "***REMOVED SENSITIVE VALUE***",
"trusted_domains": [
"192.168.0.244"
],
"datadirectory": "\/var\/files\/owncloud\/data",
"overwrite.cli.url": "http:\/\/192.168.0.244\/owncloud",
"dbtype": "mysql",
"version": "8.2.1.4",
"dbname": "owncloud",
"dbhost": "localhost",
"dbtableprefix": "oc_",
"dbuser": "***REMOVED SENSITIVE VALUE***",
"dbpassword": "***REMOVED SENSITIVE VALUE***",
"logtimezone": "UTC",
"installed": true,
"updatechecker": true,
"has_internet_connection": true,
"loglevel": 4,
"log_rotate_size": 26214400,
"log_rotate_sizerotation": 209715200,
"trashbin_retention_obligation": "14, auto",
"memcache.local": "\\OC\\Memcache\\APCu",
"theme": "",
"maintenance": false,
"appstore.experimental.enabled": true
},
"apps": {
"activity": {
"enabled": "no",
"installed_version": "2.0.1",
"types": "filesystem"
},
"backgroundjob": {
"lastjob": "24"
},
"bookmarks": {
"enabled": "yes",
"installed_version": "0.6.2",
"ocsid": "168710",
"types": ""
},
"calendar": {
"enabled": "yes",
"installed_version": "0.8.2",
"ocsid": "168707",
"types": ""
},
"contacts": {
"enabled": "yes",
"installed_version": "0.5.0.0",
"ocsid": "168708",
"types": ""
},
"core": {
"incoming_server2server_share_enabled": "no",
"installedat": "1442069383.2677",
"lastcron": "1478125861",
"lastupdateResult": "{\"version\":\"8.2.8\",\"versionstring\":\"ownCloud 8.2.8\",\"url\":\"https:\\\/\\\/download.owncloud.org\\\/community\\\/owncloud-8.2.8.zip\",\"web\":\"https:\\\/\\\/doc.owncloud.org\\\/server\\\/8.2\\\/admin_manual\\\/maintenance\\\/upgrade.html\"}",
"lastupdatedat": "1477749904",
"outgoing_server2server_share_enabled": "no",
"public_caldav": "calendar\/share.php",
"public_calendar": "calendar\/share.php",
"public_documents": "documents\/public.php",
"public_files": "files_sharing\/public.php",
"public_gallery": "gallery\/public.php",
"public_webdav": "files_sharing\/publicwebdav.php",
"remote_caldav": "calendar\/appinfo\/remote.php",
"remote_calendar": "calendar\/appinfo\/remote.php",
"remote_carddav": "contacts\/appinfo\/remote.php",
"remote_contacts": "contacts\/appinfo\/remote.php",
"remote_files": "files\/appinfo\/remote.php",
"remote_webdav": "files\/appinfo\/remote.php",
"repairlegacystoragesdone": "yes"
},
"delete_confirmation": {
"enabled": "yes",
"installed_version": "2.0.1",
"ocsid": "169968",
"types": ""
},
"documents": {
"enabled": "yes",
"installed_version": "0.11.2",
"ocsid": "168711",
"types": ""
},
"external": {
"enabled": "yes",
"installed_version": "1.2",
"ocsid": "166046",
"sites": "[[\"CNN\",\"http:\\\/\\\/cnn.com\",\"external.png\"]]",
"types": ""
},
"files": {
"enabled": "yes",
"installed_version": "1.2.0",
"types": "filesystem"
},
"files_epubviewer": {
"enabled": "no",
"installed_version": "7.0.5",
"types": ""
},
"files_external": {
"\/shrMedia": "1442070568",
"enabled": "yes",
"installed_version": "0.3.0",
"ocsid": "166048",
"types": "filesystem"
},
"files_locking": {
"enabled": "yes",
"installed_version": "",
"types": "filesystem"
},
"files_mv": {
"enabled": "yes",
"installed_version": "0.8.2",
"ocsid": "150271",
"types": "filesystem"
},
"files_pdfviewer": {
"enabled": "yes",
"installed_version": "0.7",
"ocsid": "166049",
"types": ""
},
"files_sharing": {
"2": "1458412605.0635",
"3": "1458412605.1362",
"enabled": "yes",
"incoming_server2server_share_enabled": "no",
"installed_version": "0.7.0",
"outgoing_server2server_share_enabled": "no",
"types": "filesystem"
},
"files_texteditor": {
"enabled": "yes",
"installed_version": "2.0",
"ocsid": "166051",
"types": ""
},
"files_trashbin": {
"enabled": "yes",
"installed_version": "0.7.0",
"types": "filesystem"
},
"files_versions": {
"enabled": "no",
"installed_version": "1.0.6",
"types": "filesystem"
},
"files_videoviewer": {
"enabled": "yes",
"installed_version": "0.1.3",
"ocsid": "166054",
"types": ""
},
"firstrunwizard": {
"enabled": "no",
"installed_version": "1.1",
"ocsid": "166055",
"types": ""
},
"gallery": {
"enabled": "yes",
"installed_version": "14.2.0",
"types": ""
},
"galleryplus": {
"enabled": "no",
"installed_version": "15.0.0",
"ocsid": "169116",
"types": ""
},
"logreader": {
"enabled": "yes",
"installed_version": "1.1.5",
"ocsid": "170871",
"types": ""
},
"music": {
"enabled": "yes",
"installed_version": "0.3.12",
"ocsid": "164319",
"types": "filesystem"
},
"ownnote": {
"enabled": "yes",
"folder": "Notes",
"installed_version": "1.08",
"ocsid": "168512",
"types": ""
},
"passwords": {
"backup_allowed": "true",
"days_orange": "150",
"days_red": "365",
"enabled": "no",
"https_check": "false",
"icons_allowed": "true",
"icons_service": "ddg",
"installed_version": "16",
"types": ""
},
"provisioning_api": {
"enabled": "yes",
"installed_version": "0.3.0",
"types": "filesystem"
},
"quicknotes": {
"enabled": "yes",
"installed_version": "0.1.0",
"ocsid": "174716",
"types": ""
},
"search_lucene": {
"enabled": "yes",
"installed_version": "0.6.0",
"ocsid": "168709",
"types": "filesystem"
},
"sharelinks": {
"enabled": "yes",
"installed_version": "1.1.0",
"ocsid": "170527",
"types": ""
},
"templateeditor": {
"enabled": "yes",
"installed_version": "0.1",
"types": ""
},
"updater": {
"enabled": "no",
"installed_version": "0.6",
"types": ""
}
}
}
I also did this:
Enabled:
- bookmarks: 0.6.2
- calendar: 0.8.2
- contacts: 0.5.0.0
- delete_confirmation: 2.0.1
- documents: 0.11.2
- external: 1.2
- files: 1.2.0
- files_external: 0.3.0
- files_mv: 0.8.2
- files_pdfviewer: 0.7
- files_sharing: 0.7.0
- files_texteditor: 2.0
- files_trashbin: 0.7.0
- files_videoviewer: 0.1.3
- gallery: 14.2.0
- logreader: 1.1.5
- music: 0.3.12
- ownnote: 1.08
- provisioning_api: 0.3.0
- quicknotes: 0.1.0
- search_lucene: 0.6.0
- sharelinks: 1.1.0
- templateeditor: 0.1
Disabled:
- activity
- encryption
- files_versions
- firstrunwizard
- notifications
- updater
- user_external
- user_ldap
Table calendars of app dav should not exist but has 17360640 rows
Oo seems like your database has 17million calendars Oo. I updated the script to also check the number of entries in the original tables, which should be copied over. Can you run it again: https://gist.github.com/nickvergessen/94b57082a5ab83e47bbbd1bef6d13f66
@nickvergessen that's interesting. Over time there has been a lot of issues with ownCloud and even with Nextcloud with the importing of external calendar files ( i.e. ics). Even if I delete a calendar there is no feedback if it was deleted. Shouldn't there be some sort of clean up process in the update/upgrade process to make sure that orphaned calendars are deleted. I don't understand how this has anything to do with the issues posted in #297 and #1669?
I will however run you updated script and report back.
Well, the "funny" thing is, that those tables are not used yet. The update you are failing to execute would pull them from the old table to the new one, yet this seems to have already happened or something. That's why the stats for both (old and new) tables would be nice to have.
Btw about the timeout, not sure what's up with that. If that happens again, it would be nice if you could execute it again, after deleting the table names from the array in https://gist.github.com/nickvergessen/94b57082a5ab83e47bbbd1bef6d13f66#file-app-state-detection-php-L27-L36 until the script finishes without an error.
@nickvergessen this time the script did not freeze. The results follow...
1478468601 [OK] App admin_audit is not found, not installed and disabled
1478468601 [OK] App comments is not found, not installed and disabled
1478468601 [OK] App dav is not found, not installed and disabled
1478468601 [Error] Table addressbooks of app dav exists although the app is not installed
1478468602 [Info] Table addressbooks has 5 entries
1478468602 [Error] Table cards of app dav exists although the app is not installed
1478468602 [Info] Table cards has 48 entries
1478468602 [Error] Table addressbookchanges of app dav exists although the app is not installed
1478468602 [Info] Table addressbookchanges has 48 entries
1478468602 [Error] Table calendarobjects of app dav exists although the app is not installed
1478468602 [Info] Table calendarobjects has 137 entries
1478468602 [Error] Table calendars of app dav exists although the app is not installed
1478468602 [Info] Table calendars has 11 entries
1478468602 [Error] Table calendarchanges of app dav exists although the app is not installed
1478468602 [Info] Table calendarchanges has 137 entries
1478468602 [Error] Table calendarsubscriptions of app dav exists although the app is not installed
1478468602 [Info] Table calendarsubscriptions has 0 entries
1478468602 [Error] Table schedulingobjects of app dav exists although the app is not installed
1478468602 [Info] Table schedulingobjects has 0 entries
1478468602 [Error] Table cards_properties of app dav exists although the app is not installed
1478468602 [Info] Table cards_properties has 286 entries
1478468602 [Error] Table dav_shares of app dav exists although the app is not installed
1478468602 [Info] Table dav_shares has 0 entries
1478468602 [OK] App federatedfilesharing is not found, not installed and disabled
1478468602 [Error] Table federated_reshares of app federatedfilesharing exists although the app is not installed
1478468602 [Info] Table federated_reshares has 0 entries
1478468602 [OK] App federation is not found, not installed and disabled
1478468602 [Error] Table trusted_servers of app federation exists although the app is not installed
1478468602 [Info] Table trusted_servers has 0 entries
1478468602 [OK] App files_external is installed and enabled
1478468602 [OK] Table external_mounts of app files_external exists
1478468602 [Info] Table external_mounts has 1 entries
1478468602 [OK] Table external_applicable of app files_external exists
1478468602 [Info] Table external_applicable has 1 entries
1478468602 [OK] Table external_config of app files_external exists
1478468602 [Info] Table external_config has 1 entries
1478468602 [OK] Table external_options of app files_external exists
1478468602 [Info] Table external_options has 1 entries
1478468602 [OK] App files_sharing is installed and enabled
1478468602 [OK] Table share_external of app files_sharing exists
1478468602 [Info] Table share_external has 0 entries
1478468602 [OK] App files_trashbin is installed and enabled
1478468602 [OK] Table files_trash of app files_trashbin exists
1478468602 [Info] Table files_trash has 6 entries
1478468602 [OK] App twofactor_backupcodes is not found, not installed and disabled
1478468602 [OK] Table twofactor_backup_codes of app twofactor_backupcodes does not exists
1478468602 [OK] App user_ldap is not installed and disabled
1478468602 [OK] Table ldap_user_mapping of app user_ldap does not exists
1478468602 [OK] Table ldap_group_mapping of app user_ldap does not exists
1478468602 [OK] Table ldap_group_members of app user_ldap does not exists
1478468602 [OK] App workflowengine is not found, not installed and disabled
1478468602 [Error] Table flow_checks of app workflowengine exists although the app is not installed
1478468602 [Info] Table flow_checks has 0 entries
1478468602 [Error] Table flow_operations of app workflowengine exists although the app is not installed
1478468602 [Info] Table flow_operations has 0 entries
1478468602 [OK] App activity is installed but not enabled
1478468602 [OK] Table activity of app activity exists
1478468602 [Info] Table activity has 0 entries
1478468602 [OK] Table activity_mq of app activity exists
1478468602 [Info] Table activity_mq has 0 entries
1478468602 [OK] App notifications is not installed and disabled
1478468602 [Error] Table notifications of app notifications exists although the app is not installed
1478468602 [Info] Table notifications has 0 entries
1478468602 [OK] App files_retention is not found, not installed and disabled
1478468602 [OK] Table retention of app files_retention does not exists
1478468602 [OK] App user_saml is not found, not installed and disabled
1478468602 [OK] Table user_saml_users of app user_saml does not exists
1478468602 [OK] Table user_saml_auth_token of app user_saml does not exists
1478468602 [OK] App calendar is installed and enabled
1478468602 [OK] Table clndr_objects of app calendar exists
1478468602 [Info] Table clndr_objects has 137 entries
1478468602 [OK] Table clndr_calendars of app calendar exists
1478468602 [Info] Table clndr_calendars has 5 entries
1478468602 [OK] Table clndr_share_event of app calendar exists
1478468602 [Info] Table clndr_share_event has 0 entries
1478468602 [OK] Table clndr_share_calendar of app calendar exists
1478468602 [Info] Table clndr_share_calendar has 0 entries
1478468602 [OK] Table clndr_repeat of app calendar exists
1478468602 [Info] Table clndr_repeat has 0 entries
1478468602 [OK] App contacts is installed and enabled
1478468602 [Error] Table addressbook of app contacts does not exists
1478468602 [Error] Table addressbook_backend of app contacts does not exists
1478468602 [OK] Table contacts_addressbooks of app contacts exists
1478468602 [Info] Table contacts_addressbooks has 2 entries
1478468602 [OK] Table contacts_cards of app contacts exists
1478468602 [Info] Table contacts_cards has 43 entries
1478468602 [OK] Table contacts_cards_properties of app contacts exists
1478468602 [Info] Table contacts_cards_properties has 266 entries
So this is a ownCloud 8.2 instance you say. But the strange thing is, it also contains tables from 9.0 (which give you the error atm), e.g. the DAV app:
[Error] Table addressbooks of app dav exists although the app is not installed
[Error] Table cards of app dav exists although the app is not installed
[Error] Table addressbookchanges of app dav exists although the app is not installed
[Error] Table calendarobjects of app dav exists although the app is not installed
[Error] Table calendars of app dav exists although the app is not installed
[Error] Table calendarchanges of app dav exists although the app is not installed
[Error] Table calendarsubscriptions of app dav exists although the app is not installed
[Error] Table schedulingobjects of app dav exists although the app is not installed
[Error] Table cards_properties of app dav exists although the app is not installed
[Error] Table dav_shares of app dav exists although the app is not installed
The tables are not empty, because the conversion from the old calendar/contacts app already got executed before: Old:
[Info] Table clndr_objects has 137 entries
[Info] Table clndr_calendars has 5 entries
New:
[Info] Table calendarobjects has 137 entries
[Info] Table calendars has 11 entries
The additional new calendars are the default calendars for users.
New tables that are not filled with a migration exist and are empty:
[Info] Table federated_reshares has 0 entries
[Info] Table trusted_servers has 0 entries
[Info] Table notifications has 0 entries
The most interesting thing is, that there are already 2 tables, which was newly introduced in Nextcloud 10 in your database
[Error] Table flow_checks of app workflowengine exists although the app is not installed
[Error] Table flow_operations of app workflowengine exists although the app is not installed
Did you happen to try an update to 9 and 10 and then restored a backup/rolled back?
Since you are still on a 8.2 instance, it is save to delete the following tables, because they are not in use for an 8.2 instance. So I would propose the following steps:
DROP TABLE `addressbooks`;
DROP TABLE `cards`;
DROP TABLE `addressbookchanges`;
DROP TABLE `calendarobjects`;
DROP TABLE `calendars`;
DROP TABLE `calendarchanges`;
DROP TABLE `calendarsubscriptions`;
DROP TABLE `schedulingobjects`;
DROP TABLE `cards_properties`;
DROP TABLE `dav_shares`;
DROP TABLE `federated_reshares`;
DROP TABLE `trusted_servers`;
DROP TABLE `flow_checks`;
DROP TABLE `flow_operations`;
DROP TABLE `notifications`;
In case you ever restore a backup again, after a successful or failed update attempt, make sure to delete all tables from the database before restoring the backup.
@nickvergessen yes it is ownCloud 8.2.1. All of my work for these issues were on a test server. I have attempted updates/upgrades to Nextcloud 9 and 10. I try to be careful not to backup failed upgrade/updates. I will however do as you ask from above.
BTW I will be taking steps on restores as you recommend. I reviewed my backup sqldump file and noticed there is no drop database at the beginning of the file.
@nickvergessen Dude you nailed it!
I successfully upgraded from ownCloud 8.2.1 => Nextcloud 9.0.54 => Nextcloud 10.0.1
@karlitschek buy this young fellow a couple of beers!!!
Finally!
Can this script repair-appconfig.php also applied to a higher nextcloud version? We upgraded now to NC 14.09 and get
federation: Failed to enable app federation
notifications: Failed to enable app notifications
probably since a while, when we upgraded from OC 8.2 one year ago.
When activating Federation App we see:
An exception occurred while executing 'CREATE TABLEoc_trusted_servers(idINT AUTO_INCREMENT NOT NULL,urlVARCHAR(512) NOT NULL COMMENT 'Url of trusted server',url_hashVARCHAR(255) DEFAULT '' NOT NULL COMMENT 'sha1 hash of the url without the protocol',tokenVARCHAR(128) DEFAULT NULL COMMENT 'token used to exchange the shared secret',shared_secretVARCHAR(256) DEFAULT NULL COMMENT 'shared secret used to authenticate',statusINT DEFAULT 2 NOT NULL COMMENT 'current status of the connection',sync_tokenVARCHAR(512) DEFAULT NULL COMMENT 'cardDav sync token', UNIQUE INDEX url_hash (url_hash), PRIMARY KEY(id)) DEFAULT CHARACTER SET utf8 COLLATE utf8_bin ENGINE = InnoDB': SQLSTATE[42S01]: Base table or view already exists: 1050 Table 'oc_trusted_servers' already exists
Notification App
An exception occurred while executing 'CREATE TABLEoc_notifications(notification_idINT AUTO_INCREMENT NOT NULL,appVARCHAR(32) NOT NULL,userVARCHAR(64) NOT NULL,timestampINT DEFAULT 0 NOT NULL,object_typeVARCHAR(64) NOT NULL,object_idVARCHAR(64) NOT NULL,subjectVARCHAR(64) NOT NULL,subject_parametersLONGTEXT DEFAULT NULL,messageVARCHAR(64) DEFAULT NULL,message_parametersLONGTEXT DEFAULT NULL,linkVARCHAR(4000) DEFAULT NULL,iconVARCHAR(4000) DEFAULT NULL,actionsLONGTEXT DEFAULT NULL, INDEX oc_notifications_app (app), INDEX oc_notifications_user (user), INDEX oc_notifications_timestamp (timestamp), INDEX oc_notifications_object (object_type,object_id), PRIMARY KEY(notification_id)) DEFAULT CHARACTER SET utf8 COLLATE utf8_bin ENGINE = InnoDB': SQLSTATE[42S01]: Base table or view already exists: 1050 Table 'oc_notifications' already exists
The script should still be working fine. So just remove the tables in question manually.
Signing status
``` Login as admin user into your Nextcloud and access http://example.com/index.php/settings/integrity/failed paste the results here. ```App list
``` If you have access to your command line run e.g.: sudo -u www-data php occ app:list from within your Nextcloud installation folder ```Config report
``` If you have access to your command line run e.g.: sudo -u www-data php occ config:list system from within your Nextcloud installation folder or Insert your config.php content here (Without the database password, passwordsalt and secret) ```LDAP config
``` With access to your command line run e.g.: sudo -u www-data php occ ldap:show-config from within your Nextcloud installation folder Without access to your command line download the data/owncloud.db to your local computer or access your SQL server remotely and run the select query: SELECT * FROM `oc_appconfig` WHERE `appid` = 'user_ldap'; Eventually replace sensitive data as the name/IP-address of your LDAP server or groups. ```Web server error log
``` Insert your webserver log here ```Nextcloud log
``` Insert your Nextcloud log here ```Browser log
``` Insert your browser log here, this could for example include: a) The javascript console log b) The network log c) ... ```