Migration service and script

[Elfangor93]

This PR adds the migration service as well as a new MVC routine called migration. A migration script called Jg3ToJg4 to migrate a JoomGallery v3.x to v4.x is already included. Additional migration scripts intended to be installed as an extension like in Jg3.

The migration view can be requestid with the following url: /administrator/index.php?option=com_joomgallery&view=migration

How to test this PR

Attention!! This PR is not compatible with Joomla v4.4.0, v4.4.1, v5.0.0, v5.0.1. Older and newer versions of Joomla are fully compatible.

Prerequirements

There has to be a source (JG3 updated to JG4) and a destination (JG4) available.

There are two possibilities:

1. Source and destination are within the same Joomla installation. For this you update a JG3 in a Joomla 3 to Joomla 4 and then update the JG3 to JG4 using this PR for the installation.
2. Source and destination are in two different Joomla installations, but on the same server. Here you prepare the source installation the same way as described in 1). Additionally you setup a second, fresh Joomla 4 installation. Within this Joomla 4 you install a fresh JG4. This second installation serves as your destination.

This way you end up having a source installation containing database tables looking something like #__joomgallery_XX_old. And a JoomGallery administration folder containing an xml file called joomgallery_old.xml. The source needs to have some categories and images.

Perform a migration

A migration consists of 4 steps.

1. Configuration
2. Pre-Check
3. Migration manager
4. Post-Check & Finish

You can go to the next step of the migration, if you successfully passed the prvious one.

Step 1: Configuration

Here you have to define where your source is located you want to use for the migration as well as how you want to write the records into the destination.

Source: JG3 updated to JG4 Destination: JG4

Step 2: Pre-Check

Your entire system and installation gets checked if everything is setted up correctly such that the migration will run without problems.

Step 3: Migration manager

Here you see a list of migration processes that have to be performed in a specific order. First the categories have to be migrated, then the images and at last the category thumbnails have to be reselected based on the new IDs created for the images during migration.

The start button starts the automatic execution of the migration process based on a queue. For each element in the queue an ajax request is started to execute the migration of this element. When the request returns from the server, the progress bar and the log output gets updated to inform you about the migration progress. The automatic execution of the migration can be stopped at any time with the "Stop migration" button.

As soon as all the migration processes are performed the button on the bottom "Check and finish migration" becomes clickable.

Step 4: Post-Check & Finish

The results of the post check tells you how well the migration was done. It checks if all the queues are processed and if there were errors in the migration which might need to be resolved. The button to remove the source data is not yet wokable. When you click it, you will always get a success message. But actually is not doing anything currently.

[reni68]

Does it make sense to test the migration with other settings? For example with Use new folder structure --> yes

[rowi68]

after last step the button 'Check & finish migration' is not activated

Is fixed now.

Confirmed. 👍

[MrMusic]

Does it make sense to test the migration with other settings? For example with Use new folder structure --> yes

That would be the optimum if all (or some) combinations were tested

[rowi68]

Does it make sense to test the migration with other settings? For example with Use new folder structure --> yes

I have already done this, but then I got trouble with 'inconsistent category folder paths'.

[Elfangor93]

The categories in the source db (JG3) have to be consistent according to catpath = parent-path/alias_cid: https://docs.google.com/presentation/d/1kXGfGRrHswU0M3yh0zUvOwW1fYqB07cksHzNxzcEyxI/edit#slide=id.g2ab648b0a2c_0_0

Therefore if you get this error inconsistent category folder paths then you would need to use other source data.

[rowi68]

Is there a recommandation what would be best way to migrate or does it make no difference at all ?

I see a lot of combinations. For example:

Check owners --> yes/no Use source ID's --> yes/no Use new folder structure --> yes/no Image usage --> direct usage / copy / recraete / move

Are there advantages or disadvantages of different combination? Is e.g. 'Use new folder structure' --> Yes better then --> No?

[reni68]

My Settings: Same Joomla! installation? --> yes Same database? --> yes Check owners --> no Use source ID's --> no Use new folder structure --> yes Image usage --> direct usage

Migration: Categories --> successful Migration: Images --> successful Adjustment: Category thumbnails --> failed

Log output: [Fehler] [Migrator.js] Migration of catimage with id = undefined failed.

Joomla 4.4.2 PHP 8.1.17 Database 10.4.28-MariaDB

EDIT: The same result with: Use new folder structure --> no

[AlexanderSupp]

What happens if a category ID is less than the parent ID? Maybe this is a reason for the Cat ID 3 error, as the parent ID 5 is not processed at this point? And consequently, any category ID that references a bug ID as a parent is also a bug? Category ID 39 and 40 point to ID 3 as the parent.

[Success] [Migrator.js] Migration of category with id = 1 successful. [Fehler] [Migrator.js] Migration of category with id = 3 failed. [Fehler] Invalid parent_id [0] in Joomgallery\Component\Joomgallery\Administrator\Table\CategoryTable::check() [Fehler] Data could not be inserted into destination database. See log file for more details. [Success] [Migrator.js] Migration of category with id = 5 successful. [Success] [Migrator.js] Migration of category with id = 6 successful. [Success] [Migrator.js] Migration of category with id = 8 successful. [Success] [Migrator.js] Migration of category with id = 11 successful. [Success] [Migrator.js] Migration of category with id = 38 successful. [Fehler] [Migrator.js] Migration of category with id = 39 failed. [Fehler] Invalid parent_id [0] in Joomgallery\Component\Joomgallery\Administrator\Table\CategoryTable::check() [Fehler] Data could not be inserted into destination database. See log file for more details. [Fehler] [Migrator.js] Migration of category with id = 40 failed. [Fehler] Invalid parent_id [0] in Joomgallery\Component\Joomgallery\Administrator\Table\CategoryTable::check() [Fehler] Data could not be inserted into destination database. See log file for more details. [Success] [Migrator.js] Migration of category with id = 42 successful. [Success] [Migrator.js] Migration of category with id = 43 successful.

[Elfangor93]

What happens if a category ID is less than the parent ID?

What are your migration settings?

You mean for example, parent category id = 4 and and category id = 2? From my point of view this should not be a problem.

But if there is an error in a parent category then its child will also enter an error. Thats logical.

Ideally the migration should be stopped if a child of an erroreous parent wants to be migrated. Then you eould have time to fix the parent manually before continue...

[Elfangor93]

[Fehler] [Migrator.js] Migration of catimage with id = undefined failed.

@reni68 I guess this error happened if the queue for the "Adjustment: Category thumbnails" was at zero (Pending: 0) from the beginning. This should be fixed now.

[reni68]

This should be fixed now.

😃 yes, migration successful now. Thanks!

EDIT: But shouldn't there actually be new directories with the Use new folder structure --> yes setting?

[Elfangor93]

But shouldn't there actually be new directories with the Use new folder structure --> yes setting?

Depending on your migration settings. If you coose direct usage, the available folder gets renamed to the new folder naming logic.

[Elfangor93]

@AlexanderSupp Does the last commit fix your issue?

[AlexanderSupp]

Congratulations. Looks very good at first glance. My settings: Default, then No new folder structure, JG3 compatibly mode, direct usage warnings ignored.

But I can't see the images in "Images". Example: Migrated: Id 1636 Thumbnail: as_-wolken-_sam_7130_201901311698750108.jpg Originals: as-wolken-_sam_7130_201901311698750108.jpg Details: as-wolken-_sam_7130_20190131_1698750108.jpg

DB-Table Filename: as_-wolken-_sam_7130_20190131_1698750108.jpg Title: AS - Wolken - SAM_7130 Alias: as-wolken-sam-7130

No thumbnail is displayed in images. Does "images" manipulate the filename? Joomla Media as_-wolken-_sam_7130_20190131_1698750108.jpg shows everything perfect.

In JG 3.6.2. Thumbs: as_-wolken-_sam_7130_201901311698750108.jpg DB-Table: Id 3208 Alias: as-wolken-sam-7130-3208 File-/image: as-wolken-_sam_7130_20190131_1698750108.jpg

I check other results later and continue my tests.

[AlexanderSupp]

The reason. The real directory: C:\wamp64\www\FotofreundeMuch\images\joomgallery\thumbnails\fotofreunde_much_5\alexander_supp_3 Path in the categories table: fotofreunde-much\alexander-supp

[Elfangor93]

@AlexanderSupp Yes its correct. Images are currently not shown if setting Use new folder structure = No. See https://github.com/JoomGalleryfriends/JG4-dev/pull/123#issuecomment-1895745277

[rowi68]

I have done another test with difference this time check owners button is activated.

My settings: Same Joomla! installation? --> yes Same database? --> yes Check owners --> yes Use source ID's --> no Use new folder structure --> no Image usage --> direct usage

In this case I got the following error:

When I reset Check owners --> no, then migration is successful...

[Elfangor93]

@rowi68 Your error should be fixed now.

[rowi68]

@rowi68 Your error should be fixed now.

Yes, error has gone. Thank you.

[AlexanderSupp]

If I "x interrupt migration" from Step1, first page, An error has occurred. 0 Joomla\Database\DatabaseFactory::getDriver(): Argument #1 ($name) must be of type string, null given, called in C:\wamp64\www\FotofreundeMuch\administrator\components\com_joomgallery\src\Service\Migration\Migration.php on line 609

But without any consequence, because I can simply restart.

[rowi68]

I made further tests with "check owner" and got another result as in following description:

My result at the moment: If a user ID cannot be found, the owner is not set to 0 (no one), but the owner is myself ( same if before in JG3 a category or image was without owner ).

[AlexanderSupp]

I tried to use a source from 3.6.2 in another Joomla 3.10.x at the same server. Source check failed Source extension compatibility The XML file of your source extension could not be found. Please make sure the XML is available and readable. Table: #__joomgallery_categories Table does not exist.

Are the parameters correct?

[MrMusic]

Source check failed Source extension compatibility The XML file of your source extension could not be found. Please make sure the XML is available and readable.

I can't get that further. When trying to migrate from an external installation (with otherwise identical settings), the following error appears when clicking on 'Check migration capability' the following error occurs: _An error has occurred. 0 in_array(): Argument #2 ($haystack) must be of type array, string given
in JROOT\administrator\components\comjoomgallery\src\Service\Migration\Type.php:209

DE: Soweit komme ich gar nicht. Beim Versuch einer Migration von einer ext. Installation (bei sonst gleichen Einstellungen) kommt bereits beim Klicken auf 'Check migration capability' folgender Fehler: _An error has occurred. 0 in_array(): Argument #2 ($haystack) must be of type array, string given
in JROOT\administrator\components\comjoomgallery\src\Service\Migration\Type.php:209

[Elfangor93]

@AlexanderSupp and @MrMusic: The two issues you have detected should be fixed now...

[AlexanderSupp]

Das verstehe ich nicht. Weder auf Englisch noch auf Deutsch. I do not understand that. Neither in English nor in German. Im Folgenden meine Ausgangsbasis: Below is my starting point:

[AlexanderSupp]

Was sind erlaubte Kombinationen? What are allowed combinations?

[AlexanderSupp]

Field names: "JG3 compatibility mode" and "Use new folder structure", is this the same?

[AlexanderSupp]

Conflict with every category?

Table: #__joomgallery_categories failed Table does not exist. In source???? Source extension compatibility failed Extension not supported (Extension: )

[AlexanderSupp]

"Image usage" and Image "mapping" is this the same?

[AlexanderSupp]

New attempt:

Result:

[MrMusic]

The two issues you have detected should be fixed now... Thanks.

New Tests: Same Joomla! installation? --> no Same database? --> no Check owners --> no Use source ID's --> yes Use new folder structure --> no Image usage --> copy

In step 2: danger Step 2: Migration pre-check failed. Failed check: Source extension compatibility Cause: Missing definition of 'administrator' in the path to the old joomgallery.xml in Jg3ToJg4.php at line 133, 135, 149. After a manual fix, migration is possible!

[AlexanderSupp]

@MrMusic I can confirm these results with Joomla 5.0.2. Unfortunately, I don't have the knowledge to make changes.

[MrMusic]

I don't have the knowledge to make changes.

Do you mean the proposed code change? For example change line 133 from if(\file_exists(Path::clean($joomla_root . '/components/com_joomgallery/joomgallery.xml'))) to if(\file_exists(Path::clean($joomla_root . '/administrator/components/com_joomgallery/joomgallery.xml')))

[MrMusic]

Table: #__joomgallery_categories failed Table does not exist. In source????

Confirmed. It's a separate Bug when setting is Use source ID's --> no Possible fix: change line 205 from catimage' => array(_JOOM_TABLE_CATEGORIES, 'cid', false, false, false) to catimage' => array('#__joomgallery_catg', 'cid', false, false, false)

[AlexanderSupp]

@MrMusic I tried to change it. First, I am looking for Jg3ToJg4.php, and I find it. Then I made the changes, where line 205 was line 198.

The result: Categories migration works fine. Images migrations stop as shown.

[AlexanderSupp]

Continue previous: Categories db is fine. Images db has 1 image with correct data but files in detail, original and thumbnails missing.

[MrMusic]

'#__joomgallery_migration' is missing in the uninstall.mysql.utf8.sql

[Elfangor93]

Categories db is fine. Images db has 1 image with correct data but files in detail, original and thumbnails missing.

@AlexanderSupp Can you please provide me more information how and when this issue arises?

[Elfangor93]

array_diff_key(): Argument #1 ($array) must be of type array, null given

@AlexanderSupp Can you please provide me a call stack, where this php error is thrown? You can find it in the browser console in the network tab. There you will see the failed request with the full php error reporting.

[Elfangor93]

What are allowed combinations?

@AlexanderSupp I have rewritten the error message appearing when using impossible combinations. Is this now clear, written is this way?

Move, Copy or Recreate within the same filesystem (same joomla, same filesystem) and by keeping the old folder structure is impossible. You have chosen an impossible combination of migration options. Please adjust your migration options. Impossible combination: [Image usage: Move|Copy|Recreate, Same Joomla! installation: Yes, Use new folder structure: No, (JG4 config)Filesystem: Local]

[AlexanderSupp]

Thank you for your changes. I will test all procedures from scratch in the near future. And then I use the current status of this PR.

[AlexanderSupp]

PR 123 - migration - version from 18.02.2024 11:32 DB 8.0.31 PHP 8.1.13 Apa 2.4.54 Joomla 5.0.2 Site offline "Uncategorized" in destination deleted Compatibility mode = yes (no with new folder struct does not work) Orig filename = yes ImageMagick by height 1080 and 250 Migration process: Source: Same = no DB = no Categories in old database checked and cleared with 3.6.2 standard (clear alias value and then save to get a new alias) Check owner = no Use source ID = no Use new folder struct = no (because yes with compa mode no does not work) Image usage = recreate

Categories runs quick (135 records) Images runs 1 hr, 10 min with 3003 images. (Win 11 pro, main Storage 32 GB, i7-12700k)

System runs without any errors. First view shows, everything is fine.

Differences: The alias in the old JG contains the hole path without cat ID. The alias in the new JG4 contains only the name of the category. Categories table export included here. joomgallery_catg-3-6-2.csv joomgallery_categories-4-0-0.csv

If I try to use the new folder structure, an error messages occur. Why? What is wrong in the old structure?

In the new cat tab the "static path" contains the name and the ID for all levels. In "path" the last name has no ID. Is this correct?

For all categories at level 1 in addition an empty directory with the category name (without ID) is created.

[AlexanderSupp]

PR 123 - migration - version from 18.02.2024 17:49 Same game, other parameters. Now image usage copy. As before, main functions OK. No additional directories for first level categories with name only ID as empty directories found. Run time 15 minutes. Runs without any problem.

[AlexanderSupp]

And like copy, move also works. But if the source is outside my system, is there a difference between copy and move?

[AlexanderSupp]

But if the source is outside my system, is there a difference between copy and move?

Yes it is. All images are moved and not more on the old place. Without any warning. Be happy if you have a backup. I restore now my test system, and then I try a direct upgrade in the same system. This would complete my test series.

[Elfangor93]

In the new cat tab the "static path" contains the name and the ID for all levels. In "path" the last name has no ID. Is this correct?

No is not. Will look into that tomorrow. Thanks a lot for your testing efforts!

[AlexanderSupp]

Same Joomla, same database. Check owners = yes Use Source id = no New folder = no Direct usage. 3003 images in 8 minutes. No error messages. Everything's seams fine. But it's a pain to control the results. We need a frontend at this moment.

[MrMusic]

If I try to use the new folder structure, an error messages occur. Why? What is wrong in the old structure?

First you have to clean up the JG3 with this PR: https://github.com/JoomGalleryfriends/JoomGallery/pull/129 After that you can use the new folder structure.

For all categories at level 1 in addition an empty directory with the category name (without ID) is created.

I assume this should be fixed by now.

But if the source is outside my system, is there a difference between copy and move? Yes it is. All images are moved and not more on the old place. Without any warning. Be happy if you have a backup.

I would expect this. Copy means that the images are available in both: the source and the destination. Move means that the images are only available in the destination and no longer in the source.

But it's a pain to control the results. We need a frontend at this moment.

If the categories and images are displayed in the backend, everything should be ok. The menu item 'Category view' also works.

I have now carried out several tests with the migration to the new category structure. With Joomla 4.4 everything works fine (recreate/copy/move). With Joomla 5.0.2 it works too.

Config: compatibility mode -> no Same Joomla! installation? --> no Same database? --> no Check owners --> yes Use source ID's --> yes Use new folder structure --> yes Image usage --> recreate

[Elfangor93]

With Joomla 5.0.2 no images are migrated/created if image usage is set to 'recreate'

Whats the error you get in this case?

[MrMusic]

All folders are empty. But no error message.

Am 19. Februar 2024 18:03:08 MEZ schrieb Elfangor @.***>:

With Joomla 5.0.2 no images are migrated/created if image usage is set to 'recreate'

Whats the error you get in this case?

-- Reply to this email directly or view it on GitHub: https://github.com/JoomGalleryfriends/JG4-dev/pull/123#issuecomment-1952888713 You are receiving this because you were mentioned.

Message ID: @.***> -- Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.