search

hetznercloud / cli

A command-line interface for Hetzner Cloud
MIT License
1.09k stars 79 forks source link

Check snapshot status before server start #290

Closed omexlu closed 3 years ago

omexlu commented 3 years ago

Hello,

I am programming a bash script that automatically creates snapshots for me. Since it is advised to shutdown the server before the snapshot I want to take this into account:

# create the snapshot
/home/USER/hcloud/hcloud server create-image --type snapshot --description TEST-SERVER-LAST-RUNNING-AUTO SERVER-ID >> /home/USER/hcloud/$(date +"%d-%m-%Y").log

# power-on the 'TEST-SERVER'
/home/USER/hcloud/hcloud server poweron SERVER-ID >> /home/USER/hcloud/$(date +"%d-%m-%Y").log

How can I check that the snapshot has reached at least 5% before executing the next command: # power-on the 'TEST-SERVER

Because I don't want to wait for the snapshot to finish I have to add a '&' to the end of the first command.

Thanks in advance.

LKaemmerling commented 3 years ago

Hey @omexlu,

the CLI always waits until the action is completely done (successful or failed). There is no way how to only wait for 5% of actions progress. I would recommend always waiting on the action to be successful (or failed) otherwise, you might delete your (oldest) snapshot even if the new one was not completely created.

omexlu commented 3 years ago

Hey @omexlu,

the CLI always waits until the action is completely done (successful or failed). There is no way how to only wait for 5% of actions progress. I would recommend always waiting on the action to be successful (or failed) otherwise, you might delete your (oldest) snapshot even if the new one was not completely created.

Yes this is a big problem that makes the cli not very attractive, because we need to wait until completely done and in this case the server (recommendation from hetzner to shutdown the server while doing the snapshot until 5% reached) are down. This cause a longer downtime.

This might be min 20-30 min. in case of a server with many data.

If there is no way (maybe check with a curl request) if 5% reached before starting the server the problem is solved.

But until now I can't use the CLI and its better to continue with the hetzner interface manually and start the server after 5% reached.

Is there really no other correct way to automatic a snapshot by Cronjob?

Référence to my script until now can be found here: https://forum.hetzner.com/index.php?thread/27378-automatische-snapshots/

Maybe someone here have a practical way to automatic this proceed.

Thanks in advance.

LKaemmerling commented 3 years ago

@omexlu you can always use our backup provided mechanism. The CLI always produces the same calls as the API. On a CLI script in a normal situation, you would like to wait until the specific action is finished.

As a personal note, I never had a case where I run into inconsistencies because of a snapshot/backup restoring. The snapshot is taken in such a small time period, that it shouldn't cause you problems like inconsistency. I have a few DB replication clusters with a lot of concurrent writes, I always make a fail-over training with these snapshots on a monthly basis and it always worked without any issues. I guess what you try there is shooting with canons on birds, it shouldn't be necessary to stop/shutdown your system, but of course, this can not be guaranteed.

For your wish, checking the action status (or better the progress) should be done via a curl command, within the CLI it is only possible to wait on actions within the specific command. You can not get easily the action ID or the action progress. We consider having this information not as critical for CLI users, as they are mostly not interested in the overall progress/action, they just want to know when the action is complete.

omexlu commented 3 years ago

Hello,

So I guess it's better to throw my whole idea overboard or?

I just thought that because Hetzner recommends shutting down the server before the snapshot to ensure consistency, this would have been a better option.

So you think it's not necessary to shut down the server before and I could take snapshots on the fly?

I use several forums as there is a lot of activity and there might be problems with consistency or not?

If not I could also use the automatic backup function of hetzner but here there is also a problem that qemu-agent makes a freeze for a short time.

What do you think is the best way to make regular snapshot (maybe automated)?

I do backups on rsync every night, but it is very useful to make snapshots to restore the whole project or to transfer it to another server :)

Please share your ideas and tips on how best to proceed now?

Thanks in advance.

LKaemmerling commented 3 years ago

@omexlu The process of "taking a snapshot" and our backup mechanism is identical. The only difference is that you can always trigger a snapshot and a backup can only be created from our backup scheduling system, both are images/snapshots of your server. On my personal servers I use the following backup mechanism:

  1. Database backup per mysqldump twice daily, store them for one week; the one from every Monday will be stored for 1 month, the MySQL dumps are compressed & uploaded to a storage box
  2. Same goes for important files (but only once per day)
  3. Every server has the daily automatic backups enabled

If a server break for whatever reason, i mostly rebuild it with a few ubuntu & let ansible install the server, i import everything needed and that's it. The snapshots/backups are just a fallback if something breaks that hard and reinstalling the server is nothing i can do or something like that. Backup only the data you need is the best solution.

omexlu commented 3 years ago

Hi,

Thanks for your help and suggests:

On the time this backups are created I have all websites and services in maintenance mode so that all is "blocked" in this time the backups are taken.

After finished the backups are rsyned to my local homeserver. Until now I had never a bad case but I want to be sure sure 😊

Until now I had make a Snapshot of my server every Monday (turn of, make snapshot, after 5% reached, turn on server again), this I have made manually.

Maybe it's a good option to enable the automatic backup function on the hetzner panel as they are also snapshots if I understand you right?

But with the automatic backup I have some trouble:

My backup strategy is good for this time the only trouble is the snapshot/backup of the whole server I want to automated.

Thank you.

LKaemmerling commented 3 years ago

You won't notice the freezing of the qemu-agent. This is done in a couple of ms/s. I really think you try to solve "theoretical" problems, that might appear but the chance is really high that they don't appear. So my personal opinion and recommendation is: Don't turn off the server, just make the snapshot per API/CLI/Manually and/or let the cloud backend to it (Automatic Backups) and as i said, you should focus on your data, not the whole server :) Servers should be disposable, you should be able to set up/rebuild your server right away (using ansible as a sample) and then just restore the data.

I will close the issue here now, as it is not directly CLI related. Feel free to continue writing here or open up another issue if you encounter other problems.

omexlu commented 3 years ago

Hello,

Thank you very much for all your ideas.

I think will do it this way:

So I create a simple script that create Snapshots all Monday and only keep the last 3 if a new is created.

Maybe you can help me to create this easy script for a Cronjob?

Thanks you very much and have nice holidays.

omexlu commented 3 years ago

What do you think about this to do snapshot and rotate them (maybe some little modifications are needed): https://github.com/omerh/hetzner-hcloud-snapshots/blob/main/backup.sh

But I want use them without docker.