SystemVMs won't start with error "no usable volumes found for the VM" after upgrading to 4.17.2

rajujith commented 1 year ago

ISSUE TYPE

Bug Report

COMPONENT NAME

Upgrade, SystemVM

CLOUDSTACK VERSION

4.17.2

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

Once you destroy the SystemsVMs after upgrading CloudStack from an older version to 4.17.1/2 the SystemVMs fails to get created with an error “no usable volumes found”. This is appearing since the newly added SystemVM template entries in the vm_template are missing 'size' , it would be null. The same would be seen in the template_store_ref as well. This appears if you choose not to register the systemVM template and use the new "automatic registration of systemVM templates".

While allocating the volumes of SSVM and CPVM it hits a NPE and skips allocating the volume. This VM remains and the next run of the secstorage-1 or consoleproxy-1 thread detects these as stopped SystemVMs then tries to start them with error "no usable volumes found for the VM"

A workaround would be to set a size on both vm_template and template_store_ref tables for the new systemVM template, we can get the value from the templates.properties in the secondary storage for the template.

java.lang.NullPointerException
        at org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO.getSize(TemplateDataStoreVO.java:280)
        at com.cloud.template.TemplateManagerImpl.getTemplateSize(TemplateManagerImpl.java:2007)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

STEPS TO REPRODUCE

1. Upgrade CloudStack (I used 4.14) to 4.17.1 using "automatic registration of systemVM templates"
2. Destroy SystemVMs

EXPECTED RESULTS

SystemVMs should come up after destroying

ACTUAL RESULTS

SystemVMs didn't come up

shwstppr commented 1 year ago

@rajujith I'm not able to reproduce this. I tried with a Xen env and 4.15.2 to 4.17.2. Template entries were correctly updated with size. If you still have the env, can you please check the contents of template.properties on the secondary store for the automatic registered template?

rajujith commented 1 year ago

@shwstppr The env is upgraded and it had the size in the templates.properties. I copied the same value to update the DB.

DaanHoogland commented 1 year ago

@rajujith , i try to fix if the Long size is null and retrieved as long in https://github.com/apache/cloudstack/pull/7691 I am not sure if this is the whole issue though.

xuanyuanaosheng commented 1 year ago

I meet the similar problem:

But the management-server.log does not have TemplateManagerImpl.getTemplateSize key words.

# grep -ri "TemplateManagerImpl.getTemplateSize" management-server.log

The issue is: https://github.com/apache/cloudstack/issues/7720

weizhouapache commented 1 year ago

fixed via #7691

apache / cloudstack