Manual migration instruction

[soleny894]

Hi,

The script returns me that I need to manually migrate a lot of VODs (some streamer changed their name). I've seen that files that are created by Ganymede use a random(?) string unlike ceres with the VOD ID so how can I manually migrate? Do I need a specific ID by VODs? Also, how do I insert correctly those VODs in the DB?

Thanks you.

[Zibbp]

Hi,

Have you enabled the SHOULD_RENAME env variable or were you just running without that and testing if VODs would get entered. If you haven't ran the script with SHOULD_RENAME I would go back to the README and go through that. Make sure you delete the VODs that are in Ganymede generated from the script.

Once you run the script with SHOULD_RENAME you can manually import the old Ceres VODs by doing the following:

1. Ensure the channel is created in Ganymede
2. Go to admin/vods and click New.
3. In the dialog that appears, there is a greyed-out ID field at the top. This is the UUID used for files.
4. Fill in the VOD details and start renaming files.

The new Ganymede foldername for VODs is now ID_UUID Where UUID is the ID in the create VOD dialog, and ID is the Twitch VOD ID which is the old Ceres's name.

Ceres VOD Foldername: /channel/1546578032

Ganymede VOD Foldername: /channel/1546578032_c8f907d3-0f9a-11ed-a394-0242ac1d0004

Some of the filenames within the VOD folder also got changed. Here is a screenshot.

As this is manual and will take some time here is another thought. If there are a lot of VODs. You can manually create the old channel's name in Ganymede and the import should work. You still have to create the new channel name in Ganymede for archiving any new VODs. This will result in two channels OLD and NEW.

If you have any questions let me know as this migration process is a little wonky.

[soleny894]

Hi again,

Sorry for the delay, I was busy with work. I retried the migration steps but I still gets errors like this with the renamed streamer (it tries to move the VODs from old name to the new name but of course the VODID doesn't exist in it):

failed to rename video in vod: 14786XXXXX with error: rename /vods//OLDNAME/14786XXXXX/14786XXXXX_video.mp4 /vods/NEWNAME/14786XXXXX/14786XXXXX-video.mp4: no such file or directory

Do you have any idea of what I can do? I have 120 VODs from this streamer so doing everything manually is not really an option since you have to multiple by 6/7 for all files per VOD.

[Zibbp]

Can you provide the full path to one of the Vods having issues. Is it still in the format of /streamer/1235678/12345678_video.mp4 or does it have a UUID in the path like so /streamer/23455-43589354-35983450-359538/12345678_video.mp4.

You might have to look into a program that allows powerful mass renaming of files.

[soleny894]

Is it still in the format of /streamer/1235678/12345678_video.mp4 or does it have a UUID in the path like so /streamer/23455-43589354-35983450-359538/12345678_video.mp4

It's still using the old format /streamer/1235678/12345678_video.mp4 for every single one with errors. Also, I found out that for "normal" streamer my entry are duplicated. For example I have a channel with 5 VOD (no name change) and I have 2 entry per VOD (1 working, the other don't and have a grey thumbnail). Edit: I didn't delete the VODs before running with the rename variable, my bad.

You might have to look into a program that allows powerful mass renaming of files.

I don't think I can use any renaming program since it needs to use specific IDs coming from the ganymede DB. Do you have any way to support the legacy names? It would allow my non working VOD to be used without renaming hundreds of them. My idea is: if any error occur while renaming -> stick to older ID and use that in DB instead of the new long UUID.

Edit 2: All the DB entry for the renamed streamer appear in the new channel name, the other one say "no vod".

[Zibbp]

It is impossible to support the old file system in Ganymede.

So all other VODs are imported and working in Ganymede except for this streamer correct?

I can probably figure a way to get them to import but will need some more information. Before doing anything below, please backup both databases either dumping the database or copying the mounted database volume to another location.

If all VODs except for the failed ones are in Ganymede and working. Then ensure the failed VODs do not exist in Ganymede either. Once that has been done, run the migration container again and post the full log here. I would prefer if you did not redact anything as that helps with troubleshooting.

[soleny894]

Hi again,

I made sure to follow everything. I did the db test first, it added all the VODs to their respective channel, only the renamed streamer had issues. The VODs that have been saved before his name change are added to his new name and not his old one.

Old channel name in ganymede:

New channel name in ganymede (with example VODs saved under his old username):

Then, I started the container again with the rename variable enabled and by making sure I deleted all the VODs in ganymede.

• Streamer with no name change works, no double entry like I used to have before.
• Renamed streamer has his VODs with new name working,
  • VODs under his old username didn't rename and are unavailable in ganymede (old channel in ganymede still empty).

Full errors log: https://privatebin.net/?ccca9bfde583fdfb#FWwoSsUbsZTacFiykhyPUnfto9xsGrra2eTdczDEpN8r For some reason, it output errors for the new streamer name saved VODs but after checking my VOD folder those have been renamed correctly and are working. Sometime it tries to rename already renamed files it seems.

Hope all of this can help you.

Edit: Now that I think about it, my only issue is the fact that the VOD folder doesn't exist in the new streamer folder when it try to rename it right? If the renaming script create the folder in the new streamer name and move the old file to the new path, it will work since the DB entries all point to the new name?

[Zibbp]

To confirm, the files actually exist that are in the migrate log such as /vods/newname/1577768369/1577768369-chat.json? Because the log is saying these files do not exist. The migrate script is getting the location from the Ceres database.

If possible can you provide a screenshot of a few VOD folders in each the OLD and NEW channel folders?

[soleny894]

To confirm, the files actually exist that are in the migrate log such as /vods/newname/1577768369/1577768369-chat.json?

This file doesn't exist, the VOD had no chat and ceres used to crash for this reason (I think I reported that behavior to you before) so it's normal.

For example rename /vods//oldname/45992376476/45992376476_video.mp4 /vods/newname/45992376476/45992376476-video.mp4 is strange because VOD "45992376476" was saved under the old name but on the other part it use the new name with the same VOD ID which doesn't exist obviously.

Also, in ceres all VODs are working under the new name, they are not separated in two (old and new). So when I import, everything is going to the new name (like on my second screenshot earlier). But what's important here is that on the disk side, all vod are separated like /vods/oldname/VODID and /vods/newname/VODID so I think it tries to rename them to the new name (based on what's in DB) but fail because the VODID doesn't exist in the new channel name folder. If the script create it and rename/move the files inside, maybe it can fix my issue.

Edit: Here's an example of my idea

# What's currently happening
❯ mv /vods/oldname/1314217371/1314217371_chat.mp4  /vods/newname/1314217371/1314217371-chat.mp4
mv: cannot move '/vods/oldname/1314217371/1314217371_chat.mp4' to '/vods/newname/1314217371/1314217371-chat.mp4': No such file or directory

# What needs to be done
❯ mkdir /vods/newname/1314217371 && mv /vods/oldname/1314217371/1314217371_chat.mp4 /vods/newname/1314217371/1314217371-chat.mp4
-> No error (maybe needs to update the DB for VOD path)

If possible can you provide a screenshot of a few VOD folders in each the OLD and NEW channel folders?

Old name is completely unchanged (old ID and files are still using the old name with _), on the new channel, I have folders with the new UUID but they have been saved under the new name, no trace of other VOD coming from older name.

[Zibbp]

Also, in ceres all VODs are working under the new name, they are not separated in two (old and new). So when I import, everything is going to the new name (like on my second screenshot earlier). But what's important here is that on the disk side, all vod are separated like /vods/oldname/VODID and /vods/newname/VODID so I think it tries to rename them to the new name (based on what's in DB) but fail because the VODID doesn't exist in the new channel name folder. If the script create it and rename/move the files inside, maybe it can fix my issue.

This is what is probably causing the issue. In Ganymede and Ceres each path to the video, chat, etc is saved in the DB as CHAT_PATH. So it is trying to look into a different directory but is getting confused because the old VODs alongside the new VODs in Ceres but different in the filesystem.

I can take a look at the migration script and see if your idea can be implemented. I don't think it should be an issue implementing it.

[Zibbp]

I've made a new branch named "fix". In the rename function I added some code to create the VOD folder using the new Ganymede channel name. Is this what you were thinking of?

newVodDir := fmt.Sprintf("/vods/%s/%s_%s", channel.Name, vod.ID, vID)
    err := os.MkdirAll(newVodDir, 0755)
    if err != nil {
        return fmt.Errorf("failed to create new vod directory: %v", err)
    }

I have no way to test this issue and it's specific to you so you'll have to run and see if it works. To build this container image you will have to:

I would back up your databases just incase.

1. Clone the fix branch. git clone --branch fix https://github.com/Zibbp/ganymede-migrate
2. CD to the ganymede-migrate directory cd ganymede-migrate
3. Build the docker image docker build -t zibbp/ganymede-migrate-fix .
4. Bring down the current ganymede-migrate container with docker-compose down (if it isn't already).
5. Update your docker-compose.yml file to use the above image name, such as zibbp/ganymede-migrate-fix.
6. Re-create the compose stack with docker-compose up.

Let me know if this helps.

[soleny894]

Yes this is exactly what I was thinking about, thanks you for that. I have to do a fresh backup of my VODs to try it, it will take a few hours. I'll get back to you with the result.

[soleny894]

I'm back with some results:

• All VODs stopped working once it's renamed. The script don't move the files into the new name scheme, for example for a non renamed streamer, I have 2 folders for the same VOD one with the old name (ex: 123456789) and one with 123456789_UUID (last one is empty). To make it work again, I needed to move the files from the old ID folder to the new one and it started working again.
• The issue with the renamed streamer is still there, the folders have been created but the script want to move them to the older id (ex: 45989940876) instead of the new one with the UUID in the name giving the same "no such file or directory" error . If you check the logs I provided earlier, the script don't try to move to the new UUID but to the old one like this: /vods/newname/45989940876/45989940876-chat.mp4 when it should be something like this: /vods/newname/45989940876_32d2da36-2c6e-11ed-870c-0242c0a82104/45989940876-chat.mp4

Edit for renamed streamer:

• I manually moved the files from the old streamer name to the new streamer UUID folder that was created, renamed files with "-" instead of "_" and VOD worked. If script can move to the correct folder and rename the file, it will fix this issue.

[Zibbp]

I manually moved the files from the old streamer name to the new streamer UUID folder that was created, renamed files with "-" instead of "_" and VOD worked. If script can move to the correct folder and rename the file, it will fix this issue.

This is something you are going to have to do either manually, with a program, or write some code to do it. I'm not going to change this migration script anymore as I don't want it deviating from it's original purpose.

[soleny894]

I manually moved the files from the old streamer name to the new streamer UUID folder that was created, renamed files with "-" instead of "_" and VOD worked. If script can move to the correct folder and rename the file, it will fix this issue.

This is something you are going to have to do either manually, with a program, or write some code to do it. I'm not going to change this migration script anymore as I don't want it deviating from it's original purpose.

This edit was just to say that if I do it manually (move file from /vods/oldname/45989940876/ to /vods/newname/45989940876_32d2da36-2c6e-11ed-870c-0242c0a82104/) it works in ganymede. The script still try to move everything to a wrong folder (ex: /vods/newname/45989940876) even if it creates the new folder (ex: 45989940876_32d2da36-2c6e-11ed-870c-0242c0a82104).

I'm not going to change this migration script anymore as I don't want it deviating from it's original purpose.

I'm sure that my issue would resolve the readme part about "xqcow" changing to "xqc". Everyone using the migration script will be able to concatenate all vods from renamed streamers into their new name instead of getting errors while doing so.

[soleny894]

After some tweaking, even if I'm not familiar with go, I finally been able to migrate everything, I even fixed the issue with any renamed streamer. If you want me to do a PR tell me, I'll do one so you can check if anything need to be changed.

[Zibbp]

After some tweaking, even if I'm not familiar with go, I finally been able to migrate everything, I even fixed the issue with any renamed streamer.

Glad to hear you got it working.

If you want me to do a PR tell me, I'll do one so you can check if anything need to be changed.

If you would like to open a PR I would appreciate it, although I'm not sure how many people will be using the migrate script anymore as Ceres has been put on archive for awhile and Ganymede has been out for a while.

[soleny894]

If you would like to open a PR I would appreciate it

I'll do one soon. I changed the script logic from renaming to create and delete which include a new variable SHOULD_DELETE but I still need to figure out how to run that efficiently since you can't run the script 2 times in a row without deleting VOD which will break everything since renaming already happened at that point. Maybe a few if statements need to be done to stop the injection when the deletion is asked. I decided to go that way because with the renaming behavior I couldn't create a new folder without changing the whole function and it works perfectly now for my case and for any normal streamer name.

although I'm not sure how many people will be using the migrate script anymore as Ceres has been put on archive for awhile and Ganymede has been out for a while.

I'm sure some people are still using it (I was until now), some people don't check Github and may still use it without knowing it's deprecated. The script can be useful just in case someone need it.