search

MichaelBelgium / mybb_to_flarum

A Flarum extension to convert a mybb forum to a fresh flarum install
MIT License
26 stars 15 forks source link

Make migration able to resume after erroring out #22

Open Javier-Rotelli opened 3 years ago

Javier-Rotelli commented 3 years ago

so that, one big way to add reliability to a tool like this one is to save the progress somewhere, and give the option of resuming from there. that way even if you have trouble making it to the finish line, you don't lose all you progress.

for example, I have a big forum, and half way it breaks due to a lack of memory, that could be easily worked around by just starting from the last thing migrated.

MichaelBelgium commented 3 years ago

Good idea

EDIT: how the hell is it possible to detect where u left off? And to detect "the breaking"?

Javier-Rotelli commented 3 years ago

well, Vanilla forums migrator registered in a text file its current progress. if there's no file, we start a new migration. otherwise, we take from last know good state. this requires making all the queries able to accept an offset. and to make it more resilient, I'm adding a "step" to run it in batches.

Javier-Rotelli commented 3 years ago

I'm taking a stab on this. as it turns out, it's harder than I initially thought. you can take a look here: https://github.com/Javier-Rotelli/mybb_to_flarum/commits/make-migration-resumable so far, I got all the mybb queries extracted and added a offset and limit params to make them run in batches, and to have the option to fetch from a certain offset. I need to code the second part, that is: soemhow register the current progress somewhere (i'm thinking a json file in the storage folder) and make the migrator functions able to resume from the last known good state (IE, the count on the file)

I'd love to sya a date to finish this but time and concentration are two things I´m finding hard to get lately =/

Javier-Rotelli commented 2 years ago

so, a year and a half later, but I have a working proof of concept in #31 now, it is nowhere near production ready, as it has a lot of other changes I'm making in order to be able to migrate a big forum I admin. and it will probably benefit a lot from someone who knows a little bit more about flarum than I do

But if you like it, I can try to clean it up and make it ready to merge.

I tried to comment the PR in order to explain what I did. summarized, I made all the queries able to take an offset, and saved the count object into a json file so we can read it, and restart where we stopped the last time