Migration service and script

[Elfangor93]

This PR adds the migration service as well as a new MVC routine called migration. A migration script called Jg3ToJg4 to migrate a JoomGallery v3.x to v4.x is already included. Additional migration scripts intended to be installed as an extension like in Jg3.

The migration view can be requestid with the following url: /administrator/index.php?option=com_joomgallery&view=migration

How to test this PR

Attention!! This PR is not compatible with Joomla v4.4.0, v4.4.1, v5.0.0, v5.0.1. Older and newer versions of Joomla are fully compatible.

Prerequirements

There has to be a source (JG3 updated to JG4) and a destination (JG4) available.

There are two possibilities:

1. Source and destination are within the same Joomla installation. For this you update a JG3 in a Joomla 3 to Joomla 4 and then update the JG3 to JG4 using this PR for the installation.
2. Source and destination are in two different Joomla installations, but on the same server. Here you prepare the source installation the same way as described in 1). Additionally you setup a second, fresh Joomla 4 installation. Within this Joomla 4 you install a fresh JG4. This second installation serves as your destination.

This way you end up having a source installation containing database tables looking something like #__joomgallery_XX_old. And a JoomGallery administration folder containing an xml file called joomgallery_old.xml. The source needs to have some categories and images.

Perform a migration

A migration consists of 4 steps.

1. Configuration
2. Pre-Check
3. Migration manager
4. Post-Check & Finish

You can go to the next step of the migration, if you successfully passed the prvious one.

Step 1: Configuration

Here you have to define where your source is located you want to use for the migration as well as how you want to write the records into the destination.

Source: JG3 updated to JG4 Destination: JG4

Step 2: Pre-Check

Your entire system and installation gets checked if everything is setted up correctly such that the migration will run without problems.

Step 3: Migration manager

Here you see a list of migration processes that have to be performed in a specific order. First the categories have to be migrated, then the images and at last the category thumbnails have to be reselected based on the new IDs created for the images during migration.

The start button starts the automatic execution of the migration process based on a queue. For each element in the queue an ajax request is started to execute the migration of this element. When the request returns from the server, the progress bar and the log output gets updated to inform you about the migration progress. The automatic execution of the migration can be stopped at any time with the "Stop migration" button.

As soon as all the migration processes are performed the button on the bottom "Check and finish migration" becomes clickable.

Step 4: Post-Check & Finish

The results of the post check tells you how well the migration was done. It checks if all the queues are processed and if there were errors in the migration which might need to be resolved. The button to remove the source data is not yet wokable. When you click it, you will always get a success message. But actually is not doing anything currently.

[AlexanderSupp]

I have prepared a test environment for this pull request. My steps are documented here: http://joomgallery4.fotofreunde-much.de/de/administration-de/migration-de I hope this works well. I'm looking forward. Likewise, I look forward to your progress.

[AlexanderSupp]

I performed my migration steps as defined in http://joomgallery4.fotofreunde-much.de/de/administration-de/migration-de I installed JG4-dev-migration-service.zip dated from Aug 29, 2023 16:43. The result was with warnings:

Then I started /administrator/index.php?option=com_joomgallery&view=migration The result was the following screen:

I see no field names. If i click to the button ...USE_SCRIPT.. I see the next screen without filed names and no choice to select anything. I tried the languages en-GB and de-DE. Where is my mistake?

[Elfangor93]

This PR is still work in progress. The language constants are acting currently only as placeholders since they still change all the time. As soon as I know what kind of text make sense at that point, I will add the them.

[rowi68]

My test was not successful, but I'm not sure if I've done everything right. I have execute as follow: ( PHP 8.1.26 | mysql 8.0.32 )

1. Update from J 3.10.12 to J 4.4.1 = :white_check_mark:
2. Upload PR "JG4-dev-migration-service.zip" = Error
3. Upload again PR "JG4-dev-migration-service.zip" = :white_check_mark:
4. Try to call " /administrator/index.php?option=com_joomgallery&view=migration " = Error
   • Same Error when calling "Control Panel"

[Elfangor93]

Yes, sorry. First we have to solve this error in PR #161 and then I have to update this PR to include the fixes of PR #161 in order this PR can be installed in Joomla using the installer.

[rowi68]

Have just same error as user inLIMITit yesterday...

My steps:

1. Fresh Joomla 4.3.4. installed ( PHP 8.1.26 , mysql 8.0.32 )
2. JG4-dev-4.0.0-alpha1
3. JG4-dev-migration-service.zip

[rowi68]

Now after "start script" appears error message:

[Elfangor93]

@rowi68 Can you please activate the debug mode and post the call stack of the php error? Additionally it would be helpful to have the missing info stated under testing in the ReadMe of this Repository. Otherwise its hard for me to reproduce and thus solve the issue...

[reni68]

@Elfangor93 I did that and the call stack is in the screenshot2 PHP 8.1.17 Database 10.4.28-MariaDB

[rowi68]

Call stack:

System Information:

I made 2 different installing attempts. All with same error.

1st try

• Fresh Joomla 4.3.4.
• Fresh JG4-dev-4.0.0-alpha1
• JG4-dev-migration-service.zip

2nd try

• Update from J3 to J4.3.4.
• JG4-dev-migration-service.zip

[Elfangor93]

Form::loadForm could not load file

@reni68 @rowi68 This error appears if the XML file for the migration form is not available. Please check if

administrator/components/com_joomgallery/src/Service/Migration/Scripts/Jg3ToJg4.xml

is available and readable. If this file is not available, it looks as if the installation of the PR was not successful.

[Elfangor93]

1st try

• Fresh Joomla 4.3.4.
• Fresh JG4-dev-4.0.0-alpha1
• JG4-dev-migration-service.zip

This defenitely will not work. You can not install this PR on top of the JG4-dev-4.0.0-alpha1. Since both versions of the component have the same version number, the Installer will mess up the installation.

[rowi68]

@Elfangor93

I found the XML file and should be readable:

I am not sure it is important or not. I found file not in your mentioned path above, but under: administrator/components/com_joomgallery/src/Service/Migration/Scripts/

Could this be the reason?

[Elfangor93]

administrator/components/com_joomgallery/src/Service/Migration/Scripts/

No, this one is correct. I made a typo in above comment. Sry.

[Elfangor93]

It seems to me as if this is something related to Linux. Since for @MrMusic and me it works and we are working on Windows and for you @rowi68 and @reni68 it does not work and it seems as if you are testing on Linux...

[rowi68]

@Elfangor93 :

Now I can open the script :+1:

Next issue is step 2 of migration with failed category path:

An advice how to adjust folder paths manually?

[Elfangor93]

Yes, this is known. Im working on it.

[reni68]

1, yes, this File is available and readable 2, yes, I'm testing with Linux

[reni68]

Now the script starts :+1:

But what do I have to do about this message? Step 2: Migration pre-check failed. Reason: Site offline (Website is currently online.)

Sorry :( page set offline => works

[reni68]

Migration without errors: But the Check & finish button remains deactivated.

EDIT: ok, after I used "Manuel repair" once, Step4 also works.

[reni68]

Wonderful. All categories and images migrated and visible in the new joomla (4.4.2 by the way).

Very nice. Thank you !

[Elfangor93]

Problem found

1. Migration of a nested set (e.g category) had an error and therefore migration stopped with one failed record.
2. Delete the wrong migrated record and reset the migration of this record using "Manual repair" 2.1 Change state to pending 2.2 Previously failed record will be added to migration queue again
3. Restart migration will end in a mess and lots of error messages

Reason Change state manually to pending will put the id of the record to the end of the queue. In nested sets the order of the migration and therefore the position of the specific id in the queue is very important. Adding a random id to the end of the queue is not possible and will mess up the migration.

Possible Solution The queue has to be recalculated after changing the state to pending manually. --> Currently I dont know how to do that...

[rowi68]

I have just tried a migration. Unfortunately not (yet) successful... I did the following steps:

1. Update J3 to J 4.4.2 ( PHP 8.1.27 | mysql 8.0.32 )
2. Installing this PR
3. Start migration script
4. Used the pre adjusted config in step 1
5. Step 2 'Migration pre-check' -> error message

6. Went back in step 1 and switched off 'Use new folder structure' and activated 'JG3 compatibility mode' in the JoomGallery configuration
7. Step 2 'Migration pre-check' again -> successful
8. Start migration manager
9. Start migration categories --> successful
10. Start migration images -- Error 'Field 'description' doesn't have a default value' after 4 images

and migration stops. If I start migration again, every 2 images more appears same error and migrations stops again...

[Elfangor93]

This error appears if your JG3 database is messed up... @rowi68 Is it a real life database or have you populated it manually for testing purposes? Can you post a category table row of one of the category ids that are marked in the precheck as incostistent?

[reni68]

I also get this error. My settings:

Migration:

Update J3 to J 4.4.2 ( PHP 8.1.17 | mysql 10.4.28-MariaDB )

[rowi68]

@Elfangor93 It is real life database populated with an Akeeba Backup from a J3 site.

Attached a table row:

[reni68]

No new directories for images have been created. Is that correct? Will this only be done in a later step?

[reni68]

a new attempt:

[Elfangor93]

Yes, this category record doesn't have a proper dataset. The catpath of the category must be derivable from the alias with the scheme catpath = parent-path/alias_cid in order to be migrated activating the "Use new folder structure". In your case the catpath would need to be bilder-vereinstouren_14 but it is vereinstouren_14. Therefore the failed precheck is correct here...

This topic was the main discussion of the last meeting at the 09.01.2024. So, if you want to know more details about that, join our next online meeting and we will explain it to you. The topic is more advanced and can not be explained in a comment.

[Elfangor93]

No new directories for images have been created. Is that correct? Will this only be done in a later step?

@reni68 Can you please give more context? I dont know what you mean...

[MrMusic]

Two attempts to migrate from the JG3. The result:

First test: 'Use new folder structure' is off. 'Use original ids' is off.

Before start: Pending: The number of categories is too high by '1' ('root' should not be migrated or counted). Pending: The number of images to be migrated is correct. Pending: The number of category thumbnails is always '1' regardless of how many category thumbnails are actually set.

Result of the migration: After the last step 'Adjustment: Category thumbnails' the button 'Check & finish migration' is not activated. After switching back to Step 2, the migration can be completed.

Categories and images have been successfully migrated to the DB. The original folders have been retained. The images are still saved there. Image title, alias and image date are not transferred but reset Images are not displayed. Obviously the static_path is not yet used if 'JG3 compatibility mode' is switched on in the configuration. Is this still missing? In the category thumbnails, the old image IDs have been used, but these are no longer correct.

Second test: 'Use new folder structure' is off. 'Use original ids' is on.

Before start: Pending: The number of categories is too high by '1' ('root' should not be migrated or counted). Pending: The number of images to be migrated is correct. Pending: The number of category thumbnails is always '1' no matter how many category thumbnails are actually set.

Result: Categories have been migrated successfully. The following error occurs when starting the image migration:

[Info] Error in server response. We will try again. (1/3)
[Error] Joomgallery\Component\Joomgallery\Administrator\Model\MigrationModel::insertDummyRecord(): Argument #2 ($key) must be of type int, null given, called in ...administrator\components\com_joomgallery\src\Model\MigrationModel.php on line 1040
[Info] Error in server response. We will try again. (2/3)
...

and abort

DE: Zwei Versuche einer Migration von der JG3:

Erster Test: 'Use new folder structure' ist off. 'Use original ids' ist off.

Vor dem Start: Pending: Die Zahl der Kategorien ist um '1' zu hoch ('root' sollte nicht migriert oder gezählt werden.). Pending: Die Zahl der zu migrierenden Bilder ist korrekt. Pending: Die Zahl der Kategorie-Thumbnails ist immer '1' egal wieviele Kategorie-Thumbnails tatsächlich gesetzt sind.

Ergebnis der Migration: Nach dem letzten Schritt 'Adjustment: Category thumbnails' wird der Button 'Check & finish migration' nicht aktiviert. Nach dem Zurückwechseln zu Step 2 kann die Migration beendet werden.

Kategorien und Bilder wurden erfolgreich in die DB migriert. Die Original-Ordner wurden beibehalten. Die Bilder sind weiterhin dort gespeichert. Bild-Titel, Alias und Bild-Datum werden nicht übernommen sondern neu gesetzt. Bilder werden nicht angezeigt. Offensichtlich wird noch nicht der static_path verwendet, wenn in der Konfiguration 'JG3 compatibility mode' eingeschaltet ist. Fehlt das noch? Bei den Kategorie Thumbnails wurden die alten IDs übernommen diese sind aber nicht mehr korrekt.

Zweiter Test: 'Use new folder structure' ist off. 'Use original ids' ist on.

Vor dem Start: Pending: Die Zahl der Kategorien ist um '1' zu hoch ('root' sollte nicht migriert oder gezählt werden.). Pending: Die Zahl der zu migrierenden Bilder ist korrekt. Pending: Die Zahl der Kategorie-Thumbnails ist immer '1' egal wieviele Kategorie-Thumbnails tatsächlich gesetzt sind.

Ergebnis: Kategorien wurden erfolgreich migriert. Beim Start der Bilder Migration folgender Fehler:

[Info] Error in server response. We will try again. (1/3)
[Error] Joomgallery\Component\Joomgallery\Administrator\Model\MigrationModel::insertDummyRecord(): Argument #2 ($key) must be of type int, null given, called in ...administrator\components\com_joomgallery\src\Model\MigrationModel.php on line 1040
[Info] Error in server response. We will try again. (2/3)
...

und Abbruch

[Elfangor93]

Images are not displayed. Obviously the static_path is not yet used if 'JG3 compatibility mode' is switched on in the configuration. Is this still missing?

Yes, the JG3 compatibility mode is not yet implemented. I just created the setting in the config manager such that I can implement the check that the compatibility mode is activated. The JG3 compatibility mode itself is a big change and will be done in a following PR.

[reni68]

No new directories for images have been created. Is that correct? Will this only be done in a later step?

@reni68 Can you please give more context? I dont know what you mean...

Sorry, my mistake. I had made a mistake in the settings.

[rowi68]

New test today ( similar result as MrMusic commented above )

Settings: 'Use new folder structure' --> off. 'Use source ID's' --> off.

Migration of Categories, Images, Adjustment: Category thumbnails --> successful After the last step 'Adjustment: Category thumbnails' the button 'Check & finish migration' is not activated. After switching back to Step 2, the migration can be completed.

Image title, alias and image date are not transferred. Thumbnails of images are not displayed. If trying to edit an image or category appears error message:

edited: 19.01.24 Remedy: do not use Standard Editor "CodeMirror" and turn off plugin 'Editor - CodeMirror'

[Elfangor93]

Image title, alias and image date are not transferred but reset

Yes, sorry. I forgot to adjust the fields mapping according to the new field names in the images table...

[Elfangor93]

[Error] Joomgallery\Component\Joomgallery\Administrator\Model\MigrationModel::insertDummyRecord(): Argument #2 ($key) must be of type int, null given, called in ...administrator\components\com_joomgallery\src\Model\MigrationModel.php on line 1040

I havent thought of this special case when you try to map a field name from the old db to the same field name in the new db. But this is exactly whats happen when migrating an image forcing to use the same id. ;-)

[Elfangor93]

The number of category thumbnails is always '1' regardless of how many category thumbnails are actually set.

The list of pending category thumbnails adjustments is currently created by reading out the list of available categories from #__joomgallery_categories (JG4) at that point in time when the site is loaded. If you are migrating into an empty JoomGallery there is always just one pending here namely the root category.

Therefore currently you have to reload the page after you migrated at least the categories successfully to recreate the list of pending category thumbnails adjustments. I guess we have to find here a smarter solution. So if you have a good idea how to solve that inconvenience, feel free to post it here... ;-)

[Elfangor93]

After the last step 'Adjustment: Category thumbnails' the button 'Check & finish migration' is not activated.

This is weird. I thought I already fixed that. Do you have any JavaScript errors in the browser concole? For a quickfix you can reload the page after you migrated everything, then the button should be active.

[Elfangor93]

If trying to edit an image or category appears error message:

Can you please give me here more info. I can not reproduce this issue. Are you editing a migrated record? Was the migration successful? Can you post a screenshot of the row of the record in the database JG4 table which is throwing the error?

[rowi68]

Yes, a migrated record. Migration was successful, but without correct image title, alias and image date. If i call 'Image' or 'category' (see arrow in screenshot) appears the error.

edited: 20.01.24 Resolved: reason was outdated german language pack

[Elfangor93]

Oke, then its a result of the previously failed migration. 👍

[rowi68]

Made just a new test and image title, alias and image date are correct now. But the error if calling 'Image' or 'category is still there...

edited: 20.01.24 Resolved: reason was outdated german language pack

[Elfangor93]

Can you please give me here more info. I can not reproduce this issue. Are you editing a migrated record? Was the migration successful? Can you post a screenshot of the row of the record in the database JG4 table which is throwing the error?

Can you please provide more information?

[rowi68]

Yes, a migrated record and migration was also successful. But I don't know which table throwing the error. I suppose 'j25_joomgallery' (saved pictures) and/or 'j25_joomgallery_categories' (saved categories) ? Following some example screenshots Error: Row of record from table 'j25_joomgallery'

System:

edited: 20.01.24 Resolved: reason was outdated german language pack

[Elfangor93]

number of category thumbnails is always '1'

Now when a content type which is needed before another type can be migrated, the page gets refreshed to refresh the queue of the dependent content type. With this fix migration "Adjustment: Category thumbnails" pending queue gets created after the category migration is completed.

[Elfangor93]

@rowi68 The error is thrown by the codemirror editor and I can't see any connection to JG4. Please check the following:

• Are you able to edit an image which is created within JG4 (not a migrated one)?
• Are you able to edit any article (com_content)?

My guess is that the error is coming from an incompatible editor button plugin. Is it a new Joomla 4 installation or updated from Joomla 3? If it is an updated one, please deactivate all 3rd party plugins especially the ones of type "editors-xtd".

[rowi68]

@Elfangor93 Thank you for support. :+1:

Resolved: Reason was outdated german language pack...

[rowi68]

I have tested this PR successful ( 49 categories with 2365 images )

Settings: Same Joomla! installation? --> yes Same database? --> yes Check owners --> no Use source ID's --> no Use new folder structure --> no Image usage --> direct usage

Open points:

• migrated images and thumbnails are not displayed
• after last step the button 'Check & finish migration' is not activated ( but with reloading the page is activated )

[Elfangor93]

migrated images and thumbnails are not displayed

Yes, that will be added later. See https://github.com/JoomGalleryfriends/JG4-dev/pull/123#issuecomment-1895745277

after last step the button 'Check & finish migration' is not activated

Is fixed now. Refreshing the page within the migration manager messed up the script activating the final button.

[reni68]

I can confirm rowi68. Migration successful with these settings and the 'Check & finish migration' button is activated :+1: