search

Corsinvest / cv4pve-autosnap

Automatic snapshot tool for Proxmox VE
https://www.corsinvest.it/cv4pve
GNU General Public License v3.0
413 stars 51 forks source link

Skip snapshot when disk going full #51

Closed bds-congnguyen closed 2 years ago

bds-congnguyen commented 3 years ago

Hi there,

Thank for awesome tool, but in proxmox when a disk got full, all QMEMU/LXC instance will get Pause due to I/O error so to avoid when snapshot cause disk getting full, can we have a feature like skip snapshot when disk is over 95% capacity?

franklupo commented 3 years ago

Hi, All disks attached to the virtual machine should be checked. Now I think how this functionality can be implemented.

best regards

franklupo commented 2 years ago

Hi, I want to add a parameter --max-perc-storage to specify any percentage for statrage other than the default 95%. For each VM I check the disks on which storage it is supported, from the storage I check used if lower percentage. If one of the disks is on a storage out of range, the snapshot of the VM will not be taken and an error will be reported.

What do you think?

best regards

adrianvg commented 2 years ago

@franklupo

Is this the error you mention will be reported? This autosnap is being run on a proxmox cluster node that is 90% full.

ACTION Snap
VMs:              all
Label:            daily
Keep:             1
State:            False
Timeout:          30000
Timestamp format: yyMMddHHmmss
Max % Storage :   95%
================ EXCEPTION ================ 
System.TypeInitializationException
The type initializer for 'Humanizer.Configuration.Configurator' threw an exception.
   at Humanizer.Configuration.Configurator.get_Formatters() in /_/src/Humanizer/Configuration/Configurator.cs:line 37
   at Humanizer.Configuration.Configurator.GetFormatter(CultureInfo culture) in /_/src/Humanizer/Configuration/Configurator.cs:line 101
   at Humanizer.Bytes.ByteSize.GetLargestWholeNumberSymbol(IFormatProvider provider) in /_/src/Humanizer/Bytes/ByteSize.cs:line 72
   at Humanizer.Bytes.ByteSize.ToString(IFormatProvider provider) in /_/src/Humanizer/Bytes/ByteSize.cs:line 231
   at Humanizer.Bytes.ByteSize.ToString() in /_/src/Humanizer/Bytes/ByteSize.cs:line 223
   at BetterConsoleTables.Table.GetColumnLengths()
   at BetterConsoleTables.Table.ToString()
   at Corsinvest.ProxmoxVE.Api.Extension.Helpers.TableHelper.Create(IEnumerable`1 columns, IEnumerable`1 rows, TableOutputType outputType, Boolean hasInnerRows)
   at Corsinvest.ProxmoxVE.Api.Extension.Helpers.TableHelper.Create(IEnumerable`1 columns, IEnumerable`1 rows, TableOutputType outputType, Boolean hasInnerRows)
   at Corsinvest.ProxmoxVE.AutoSnap.Api.Application.Snap(String vmIdsOrNames, String label, Int32 keep, Boolean state, Int64 timeout, String timestampFormat, Int32 optMaxPercentageStorage)
   at Corsinvest.ProxmoxVE.AutoSnap.ShellCommands.<>c__DisplayClass9_1.<Snap>b__1()
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.<>c__DisplayClass144_0.<OnExecute>b__0(CancellationToken _)
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.ExecuteAsync(String[] args, CancellationToken cancellationToken)
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
   at Corsinvest.ProxmoxVE.Api.Shell.Helpers.ShellHelper.ExecuteConsoleApp(CommandLineApplication app, String[] args)
bds-congnguyen commented 2 years ago

yes, I've tried with storage usage is 91,5% and --max-perc-storage=90 and return same error 


VMs:              @all-pve5
Label:            15min
Keep:             1
State:            False
Timeout:          30000
Timestamp format: yyMMddHHmmss
Max % Storage :   90%
================ EXCEPTION ================ 
System.TypeInitializationException
The type initializer for 'Humanizer.Configuration.Configurator' threw an exception.
   at Humanizer.Configuration.Configurator.get_Formatters() in /_/src/Humanizer/Configuration/Configurator.cs:line 37
   at Humanizer.Configuration.Configurator.GetFormatter(CultureInfo culture) in /_/src/Humanizer/Configuration/Configurator.cs:line 101
   at Humanizer.Bytes.ByteSize.GetLargestWholeNumberSymbol(IFormatProvider provider) in /_/src/Humanizer/Bytes/ByteSize.cs:line 72
   at Humanizer.Bytes.ByteSize.ToString(IFormatProvider provider) in /_/src/Humanizer/Bytes/ByteSize.cs:line 231
   at Humanizer.Bytes.ByteSize.ToString() in /_/src/Humanizer/Bytes/ByteSize.cs:line 223
   at BetterConsoleTables.Table.GetColumnLengths()
   at BetterConsoleTables.Table.ToString()
   at Corsinvest.ProxmoxVE.Api.Extension.Helpers.TableHelper.Create(IEnumerable`1 columns, IEnumerable`1 rows, TableOutputType outputType, Boolean hasInnerRows)
   at Corsinvest.ProxmoxVE.Api.Extension.Helpers.TableHelper.Create(IEnumerable`1 columns, IEnumerable`1 rows, TableOutputType outputType, Boolean hasInnerRows)
   at Corsinvest.ProxmoxVE.AutoSnap.Api.Application.Snap(String vmIdsOrNames, String label, Int32 keep, Boolean state, Int64 timeout, String timestampFormat, Int32 optMaxPercentageStorage)
   at Corsinvest.ProxmoxVE.AutoSnap.ShellCommands.<>c__DisplayClass9_1.<Snap>b__1()
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.<>c__DisplayClass144_0.<OnExecute>b__0(CancellationToken _)
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.ExecuteAsync(String[] args, CancellationToken cancellationToken)
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
   at Corsinvest.ProxmoxVE.Api.Shell.Helpers.ShellHelper.ExecuteConsoleApp(CommandLineApplication app, String[] args)```
franklupo commented 2 years ago

Hi, Which version do you use of Proxmox?

franklupo commented 2 years ago

@franklupo

Is this the error you mention will be reported? This autosnap is being run on a proxmox cluster node that is 90% full.

ACTION Snap
VMs:              all
Label:            daily
Keep:             1
State:            False
Timeout:          30000
Timestamp format: yyMMddHHmmss
Max % Storage :   95%
================ EXCEPTION ================ 
System.TypeInitializationException
The type initializer for 'Humanizer.Configuration.Configurator' threw an exception.
   at Humanizer.Configuration.Configurator.get_Formatters() in /_/src/Humanizer/Configuration/Configurator.cs:line 37
   at Humanizer.Configuration.Configurator.GetFormatter(CultureInfo culture) in /_/src/Humanizer/Configuration/Configurator.cs:line 101
   at Humanizer.Bytes.ByteSize.GetLargestWholeNumberSymbol(IFormatProvider provider) in /_/src/Humanizer/Bytes/ByteSize.cs:line 72
   at Humanizer.Bytes.ByteSize.ToString(IFormatProvider provider) in /_/src/Humanizer/Bytes/ByteSize.cs:line 231
   at Humanizer.Bytes.ByteSize.ToString() in /_/src/Humanizer/Bytes/ByteSize.cs:line 223
   at BetterConsoleTables.Table.GetColumnLengths()
   at BetterConsoleTables.Table.ToString()
   at Corsinvest.ProxmoxVE.Api.Extension.Helpers.TableHelper.Create(IEnumerable`1 columns, IEnumerable`1 rows, TableOutputType outputType, Boolean hasInnerRows)
   at Corsinvest.ProxmoxVE.Api.Extension.Helpers.TableHelper.Create(IEnumerable`1 columns, IEnumerable`1 rows, TableOutputType outputType, Boolean hasInnerRows)
   at Corsinvest.ProxmoxVE.AutoSnap.Api.Application.Snap(String vmIdsOrNames, String label, Int32 keep, Boolean state, Int64 timeout, String timestampFormat, Int32 optMaxPercentageStorage)
   at Corsinvest.ProxmoxVE.AutoSnap.ShellCommands.<>c__DisplayClass9_1.<Snap>b__1()
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.<>c__DisplayClass144_0.<OnExecute>b__0(CancellationToken _)
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.ExecuteAsync(String[] args, CancellationToken cancellationToken)
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
   at Corsinvest.ProxmoxVE.Api.Shell.Helpers.ShellHelper.ExecuteConsoleApp(CommandLineApplication app, String[] args)

No, this is an error. Can execute with --debug optional attach log.

Remember remove password

Best regards

adrianvg commented 2 years ago

No, this is an error. Can execute with --debug optional attach log.

Remember remove password

Best regards

@franklupo

Attached. Thanks for any insights!

autosnap.log

franklupo commented 2 years ago

The error is fixed in new version. Please download last release.

Best rrgards

adrianvg commented 2 years ago

The error is fixed in new version. Please download last release.

Best rrgards

Just noticed it. Trying it out. Seems okay so far. Thanks for the quick fix!

Update: It's done. No problems. Thank you very much!

bds-congnguyen commented 2 years ago

Hi there and @franklupo I think the table of Storage status is not right, since I'm running snapshot only for pve5 but table also show storages of other hosts also, we should skip check storage if this type of storage is not contains disk images, container like pbs_ohan_backup as below because it's storage of proxmox backup server and contain only VZDump backup file


VMs:              @all-pve5
Label:            15min
Keep:             1
State:            False
Timeout:          30000
Timestamp format: yyMMddHHmmss
Max % Storage :   50%
-----------------------------------------------------------------------                                                                                                           
| Storage               | Valid | Used %  | Max Disk (GB) | Disk (GB) |                                                                                                           
-----------------------------------------------------------------------                                                                                                           
| pve1/backup_ohan      | Ok    | 1.4     | 241200        | 3493      |                                                                                                           
| pve1/data             | Ko    | 72.5    | 3689          | 2676      |                                                                                                           
| pve1/local            | Ok    | 5.6     | 176           | 9         |                                                                                                           
| pve1/pbs_ohan_backup  | Ok    | 13.7    | 275446        | 37738     |                                                                                                                                                                                                               
| pve5/backup_ohan     | Ok    | 1.4     | 241200        | 3493      |                                                                                                           
| pve5/data            | Ko    | 81.9    | 7003          | 5737      |                                                                                                           
| pve5/local           | Ok    | 3.4     | 152           | 5         | ```
franklupo commented 2 years ago

Ok, in the next release i check the content type of storage and the nodes used for Autosnap.

Best regards