One event could not be migrated: Invalid Mimedir file

[fossxplorer]

Steps to reproduce

1.Upgrade from oC stable8.2 to stable9 with updating contacts to current version 2.Follow the oC log

Expected behaviour

Contact gets updated to recent version

Actual behaviour

Lots of error as given in the subject field

Server configuration

Operating system: CentOS 7

Web server: Apache 2.4

Database: MariaDB 10.x

PHP version: 5.6.x

ownCloud version: (see ownCloud admin page)

• version: 9.0.8.2
• versionstring: 9.0.8

Contacts version: 1.3.3

Updated from an older ownCloud or fresh install: From stable8.2

Signing status (ownCloud 9.0 and above):

Login as admin user into your ownCloud and access 
http://example.com/index.php/settings/integrity/failed 
paste the results here.

List of activated apps: nabled:

• activity: 2.2.1
• bookmarks: 0.666666
• calendar: 1.3.3
• comments: 0.2
• contacts: 1.5.1
• dav: true
• documents: 0.12.1
• federatedfilesharing: true
• files: 1.4.4
• files_external: 0.5.2
• files_pdfviewer: 0.8.1
• files_sharing: 0.9.1
• files_texteditor: 2.1
• files_trashbin: 0.8.0
• files_versions: 1.2.0
• files_videoviewer: 0.1.3
• mozilla_sync: 1.4
• music: 0.3.13
• news: 8.0.0
• notes: 2.0.2
• ownnote: 1.08
• provisioning_api: 0.4.1

#### ownCloud log (data/owncloud.log)

Insert your ownCloud log here

{"reqId":"dF74gdnzmIns+V46ga5I","remoteAddr":"","app":"dav","message":"One event could not be migrated. (id: 1080927, addressbookid: 16705): {\"Exception\":\"Sabre\\\\VObject\\\\ParseException\",\"Message\":\"Invalid Mimedir file. Line starting at 18 did not follow iCalendar\\\/vCard conventions\",\"Code\":0,\"Trace\":\"#0 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/3rdparty\\\/sabre\\\/vobject\\\/lib\\\/Parser\\\/MimeDir.php(189): Sabre\\\\VObject\\\\Parser\\\\MimeDir->readProperty(';INITIALS=P.W.;...')\\n#1 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/3rdparty\\\/sabre\\\/vobject\\\/lib\\\/Parser\\\/MimeDir.php(137): Sabre\\\\VObject\\\\Parser\\\\MimeDir->parseLine(';INITIALS=P.W.;...')\\n#2 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/3rdparty\\\/sabre\\\/vobject\\\/lib\\\/Parser\\\/MimeDir.php(63): Sabre\\\\VObject\\\\Parser\\\\MimeDir->parseDocument()\\n#3 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/3rdparty\\\/sabre\\\/vobject\\\/lib\\\/Reader.php(44): Sabre\\\\VObject\\\\Parser\\\\MimeDir->parse('BEGIN:VCARD\\\\r\\\\nVE...', 0)\\n#4 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/public_html\\\/apps\\\/dav\\\/lib\\\/carddav\\\/carddavbackend.php(928): Sabre\\\\VObject\\\\Reader::read('BEGIN:VCARD\\\\r\\\\nVE...')\\n#5 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/public_html\\\/apps\\\/dav\\\/lib\\\/carddav\\\/carddavbackend.php(887): OCA\\\\DAV\\\\CardDAV\\\\CardDavBackend->readCard('BEGIN:VCARD\\\\r\\\\nVE...')\\n#6 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/public_html\\\/apps\\\/dav\\\/lib\\\/carddav\\\/carddavbackend.php(499): OCA\\\\DAV\\\\CardDAV\\\\CardDavBackend->updateProperties(3060, '1f79b8f0-7749-4...', 'BEGIN:VCARD\\\\r\\\\nVE...')\\n#7 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/public_html\\\/apps\\\/dav\\\/lib\\\/migration\\\/migrateaddressbooks.php(93): OCA\\\\DAV\\\\CardDAV\\\\CardDavBackend->createCard(3060, '1f79b8f0-7749-4...', 'BEGIN:VCARD\\\\r\\\\nVE...')\\n#8 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/public_html\\\/apps\\\/dav\\\/lib\\\/migration\\\/addressbookadapter.php(87): OCA\\\\Dav\\\\Migration\\\\MigrateAddressbooks->OCA\\\\Dav\\\\Migration\\\\{closure}(Array)\\n#9 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/public_html\\\/apps\\\/dav\\\/lib\\\/migration\\\/migrateaddressbooks.php(103): OCA\\\\Dav\\\\Migration\\\\AddressBookAdapter->foreachCard('16705', Object(Closure))\\n#10 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/public_html\\\/apps\\\/dav\\\/lib\\\/migration\\\/migrateaddressbooks.php(77): OCA\\\\Dav\\\\Migration\\\\MigrateAddressbooks->migrateBook('16705', 3060)\\n#11 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/public_html\\\/apps\\\/dav\\\/lib\\\/migration\\\/addressbookadapter.php(66): OCA\\\\Dav\\\\Migration\\\\MigrateAddressbooks->OCA\\\\Dav\\\\Migration\\\\{closure}(Array)\\n#12 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/public_html\\\/apps\\\/dav\\\/lib\\\/migration\\\/migrateaddressbooks.php(79): OCA\\\\Dav\\\\Migration\\\\AddressBookAdapter->foreachBook('cbimages', Object(Closure))\\n#13 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/public_html\\\/apps\\\/dav\\\/appinfo\\\/application.php(193): OCA\\\\Dav\\\\Migration\\\\MigrateAddressbooks->migrateForUser('cbimages')\\n#14 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/public_html\\\/lib\\\/private\\\/user\\\/manager.php(348): OCA\\\\Dav\\\\AppInfo\\\\Application->OCA\\\\Dav\\\\AppInfo\\\\{closure}(Object(OC\\\\User\\\\User))\\n#15 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/public_html\\\/apps\\\/dav\\\/appinfo\\\/application.php(194): OC\\\\User\\\\Manager->callForAllUsers(Object(Closure))\\n#16 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/public_html\\\/apps\\\/dav\\\/appinfo\\\/install.php(26): OCA\\\\Dav\\\\AppInfo\\\\Application->migrateAddressbooks()\\n#17 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/public_html\\\/lib\\\/private\\\/installer.php(640): include('\\\/var\\\/www\\\/vhosts...')\\n#18 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/public_html\\\/lib\\\/private\\\/installer.php(591): OC_Installer::includeAppScript('\\\/var\\\/www\\\/vhosts...')\\n#19 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/public_html\\\/lib\\\/private\\\/installer.php(556): OC_Installer::installShippedApp('dav')\\n#20 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/public_html\\\/lib\\\/private\\\/updater.php(334): OC_Installer::installShippedApps(true)\\n#21 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/public_html\\\/lib\\\/private\\\/updater.php(212): OC\\\\Updater->doUpgrade('9.0.8.2', '8.2.10.2')\\n#22 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/public_html\\\/core\\\/command\\\/upgrade.php(246): OC\\\\Updater->upgrade()\\n#23 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/3rdparty\\\/symfony\\\/console\\\/Command\\\/Command.php(259): OC\\\\Core\\\\Command\\\\Upgrade->execute(Object(Symfony\\\\Component\\\\Console\\\\Input\\\\ArgvInput), Object(Symfony\\\\Component\\\\Console\\\\Output\\\\ConsoleOutput))\\n#24 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/public_html\\\/core\\\/command\\\/base.php(158): Symfony\\\\Component\\\\Console\\\\Command\\\\Command->run(Object(Symfony\\\\Component\\\\Console\\\\Input\\\\ArgvInput), Object(Symfony\\\\Component\\\\Console\\\\Output\\\\ConsoleOutput))\\n#25 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/3rdparty\\\/symfony\\\/console\\\/Application.php(840): OC\\\\Core\\\\Command\\\\Base->run(Object(Symfony\\\\Component\\\\Console\\\\Input\\\\ArgvInput), Object(Symfony\\\\Component\\\\Console\\\\Output\\\\ConsoleOutput))\\n#26 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/3rdparty\\\/symfony\\\/console\\\/Application.php(192): Symfony\\\\Component\\\\Console\\\\Application->doRunCommand(Object(OC\\\\Core\\\\Command\\\\Upgrade), Object(Symfony\\\\Component\\\\Console\\\\Input\\\\ArgvInput), Object(Symfony\\\\Component\\\\Console\\\\Output\\\\ConsoleOutput))\\n#27 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/3rdparty\\\/symfony\\\/console\\\/Application.php(123): Symfony\\\\Component\\\\Console\\\\Application->doRun(Object(Symfony\\\\Component\\\\Console\\\\Input\\\\ArgvInput), Object(Symfony\\\\Component\\\\Console\\\\Output\\\\ConsoleOutput))\\n#28 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/public_html\\\/lib\\\/private\\\/console\\\/application.php(146): Symfony\\\\Component\\\\Console\\\\Application->run(NULL, NULL)\\n#29 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/public_html\\\/console.php(96): OC\\\\Console\\\\Application->run()\\n#30 \\\/var\\\/www\\\/vhosts\\\/owncloud\\\/public_html\\\/occ(11): require_once('\\\/var\\\/www\\\/vhosts...')\\n#31 {main}\",\"File\":\"\\\/var\\\/www\\\/vhosts\\\/owncloud\\\/3rdparty\\\/sabre\\\/vobject\\\/lib\\\/Parser\\\/MimeDir.php\",\"Line\":385}","level":3,"time":"2017-02-10T15:48:27+00:00","method":"--","url":"--","user":"--"}

Have many of the ^^ lines and i wonder if those events are lost after migration? I won't be able to confirm since it's still in the process!

Thanks.

[DeepDiver1975]

Data is not lost - please have a look at the old table and see if you can manually recover the event

[fossxplorer]

It's now 777 of such events. It's a pretty big instance we are talking about with thousands of users. But i'm running a pre-upgrade test environment to test the upgrade to oC 9 so the prod env isn't touched yet (still on stable8.2).

Is there a way to avoid this prior to the upgrade to oC 9 and contacts 1.3.3? Thanks @DeepDiver1975 :)

[DeepDiver1975]

Is there a way to avoid this prior to the upgrade to oC 9 and contacts 1.3.3?

the only option I see is to extract all events from the database and fix them. Pretty bad situation ....

[fossxplorer]

@DeepDiver1975 I'm running this again couple in advance of a real migration in couple of days. It seems the contacts get migrated good actually, but when that parts finishes, i get out of memory error. The samme happened a while back. How can i continue with calendar migration now?

$ time php -c /home/oc/php.ini ./occ upgrade --skip-migration-test --no-app-disable -vvv ownCloud or one of the apps require upgrade - only a limited number of commands are available You may use your browser or the occ upgrade command to do the upgrade

• Hint: You can speed up the upgrade by executing this SQL command manually: ALTER TABLE *PREFIX*filecache ADD COLUMN checksum varchar(255) DEFAULT NULL AFTER permissions; Continue with update (y/n) y 2017-03-08T01:34:03+00:00 Set log level to debug 2017-03-08T01:34:03+00:00 Repair step: Repair encryption app incompatibility 2017-03-08T01:34:03+00:00 Repair info: No repair necessary 2017-03-08T01:34:03+00:00 Repair step: Repair MySQL database engine 2017-03-08T01:34:03+00:00 Repair step: Repair MySQL collation 2017-03-08T01:34:03+00:00 Repair step: Repair SQLite autoincrement 2017-03-08T01:34:03+00:00 Repair step: Repair duplicate entries in oc_lucene_status 2017-03-08T01:34:03+00:00 Repair info: removing duplicate entries from lucene_status 2017-03-08T01:34:03+00:00 Repair step: Copy the rewrite base to the config file 2017-03-08T01:34:03+00:00 Updating database schema 2017-03-08T01:34:03+00:00 Updated database 2017-03-08T01:34:04+00:00 Disabled incompatible app: sharelinks 2017-03-08T01:34:04+00:00 Updating ... 2017-03-08T01:34:04+00:00 Updated to 1.3.3 2017-03-08T01:34:04+00:00 Updating ... 2017-03-08T01:34:04+00:00 Updated to 1.5.1 2017-03-08T01:34:04+00:00 Updating ... 2017-03-08T01:34:04+00:00 Updated to 0.12.1 2017-03-08T01:34:04+00:00 Updating ... 2017-03-08T01:34:04+00:00 Updated to 0.8.1 2017-03-08T01:34:04+00:00 Updating ... 2017-03-08T01:34:04+00:00 Updated to 2.1 2017-03-08T01:34:04+00:00 Updating ... 2017-03-08T01:34:04+00:00 Updated to 1.4.4 2017-03-08T01:34:04+00:00 Updating ... 2017-03-08T01:34:58+00:00 Updated to 2.2.1 2017-03-08T01:34:58+00:00 Updating ... 2017-03-08T01:34:59+00:00 Updated to 0.5.2 2017-03-08T01:34:59+00:00 Updating ... 2017-03-08T01:35:02+00:00 Updated to 0.9.1 2017-03-08T01:35:02+00:00 Updating ... 2017-03-08T01:35:02+00:00 Updated to 0.8.0 2017-03-08T01:35:02+00:00 Updating ... 2017-03-08T01:35:02+00:00 Updated to 1.2.0 PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 25690114 bytes) in /var/www/vhosts/oc/3rdparty/sabre/vobject/lib/Parser/MimeDir.php on line 272

real 1079m28.892s user 44m33.818s sys 2m40.499s

I've now changed the memory_limit to 4GB and trying to rerun: 2017-03-08T20:01:20+00:00 Set log level to debug 2017-03-08T20:01:20+00:00 Repair step: Repair encryption app incompatibility 2017-03-08T20:01:20+00:00 Repair info: No repair necessary 2017-03-08T20:01:20+00:00 Repair step: Repair MySQL database engine 2017-03-08T20:01:20+00:00 Repair step: Repair MySQL collation 2017-03-08T20:01:20+00:00 Repair step: Repair SQLite autoincrement 2017-03-08T20:01:20+00:00 Repair step: Repair duplicate entries in oc_lucene_status 2017-03-08T20:01:20+00:00 Repair info: removing duplicate entries from lucene_status 2017-03-08T20:01:20+00:00 Repair step: Copy the rewrite base to the config file 2017-03-08T20:01:20+00:00 Updating database schema 2017-03-08T20:01:20+00:00 Updated database 2017-03-08T20:01:21+00:00 dav: An exception occurred while executing 'CREATE TABLE oc_addressbooks (id BIGINT UNSIGNED AUTO_INCREMENT NOT NULL, principaluri VARCHAR(255) DEFAULT NULL, displayname VARCHAR(255) DEFAULT NULL, uri VARCHAR(255) DEFAULT NULL, description VARCHAR(255) DEFAULT NULL, synctoken INT UNSIGNED DEFAULT 1 NOT NULL, UNIQUE INDEX addressbook_index (principaluri, uri), PRIMARY KEY(id)) DEFAULT CHARACTER SET utf8 COLLATE utf8_bin ENGINE = InnoDB':

SQLSTATE[42S01]: Base table or view already exists: 1050 Table 'oc_addressbooks' already exists 2017-03-08T20:01:21+00:00 Repair step: Repair mime types

Naturally it's complaning oc_addressbooks is there, but it's continuing with the upgrade process it seems. But i didn't see any calendar upgrade at all. Is that also ignored due to contacts table error?

@DeepDiver1975