Open nxsbi opened 1 month ago
@nxsbi I gather this is a Xen environment. Are there a chain of snapshots present for the given volume? If yes, have all the snapshots belonging to the chain been migrated to the newly added secondary store? the destination storage where the new volume snapshot is backed up to is decided based on the location of the previous snapshot: https://github.com/apache/cloudstack/blob/4.19.1.0/engine/storage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/DefaultSnapshotStrategy.java#L151.
@nxsbi I gather this is a Xen environment. Are there a chain of snapshots present for the given volume? If yes, have all the snapshots belonging to the chain been migrated to the newly added secondary store? the destination storage where the new volume snapshot is backed up to is decided based on the location of the previous snapshot: https://github.com/apache/cloudstack/blob/4.19.1.0/engine/storage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/DefaultSnapshotStrategy.java#L151.
Hello, Yes, all the snapshots have moved over - there is no more folder for volume Id under the snapshots/account/ in the old storage. All of them are in the new storage.
One thing I noticed is that the records in the snapshot_store_ref for these volumes show parent_snapshot_id as 0 - which is usually only when it’s the first full snapshot, and second these records show a Physical_size =0.
I went through the code, but I couldn’t determine what query is actually getting executed to get the store id. Can you provide what query is actually getting run ? I can try running that and figure out what records are coming back and see if I can troubleshoot this
Based on the observation you made, it seems like the following piece of code didn't run: https://github.com/apache/cloudstack/blob/4.19.1.0/engine/storage/image/src/main/java/org/apache/cloudstack/storage/image/SecondaryStorageServiceImpl.java#L223-L227 . Did the migration operation complete successfully? Have the original records of the snapshot (in snapshot_store_ref) been deleted? The store chosen during creation of a snapshot is determined from the parent snapshot - this is the code snip that helps in getting the secondary store for backing up the snapshot https://github.com/apache/cloudstack/blob/4.19.1.0/engine/storage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/SnapshotServiceImpl.java#L298
ISSUE TYPE
COMPONENT NAME
CLOUDSTACK VERSION
CONFIGURATION
XCP-NG
OS / ENVIRONMENT
NA
SUMMARY
After adding new secondary store, I moved (balanced) from existing secondary store to new secondary store. The move completed successfully. However, new snapshots are all failing for the snapshots that moved over to the new secondary store. In logs, It appears Cloudstack is trying to mount the old NFS server's snapshot folder path. However the cloud.snapshot_store_ref shows the correct new path for all snapshots (on new NFS server).
Note that the new NFS share is from Truenas, while old one was NFS share from CentOS 7 linux - not that it should make any difference.
STEPS TO REPRODUCE
EXPECTED RESULTS
ACTUAL RESULTS