typesafehub / conductr-cli

CLI for Lightbend ConductR
Other
16 stars 21 forks source link

Add support for restoring a backup #532

Closed ashutoshraina closed 7 years ago

ashutoshraina commented 7 years ago
ashutoshraina commented 7 years ago

Manual Testing :

Backup a Cluster Stop A Bundle Restore From Backup

Restore scales the stopped bundle (eslite) again

$ conduct restore local.zip

Restoring bundle : continuous-delivery.
[##################################################] 100%
Loaded continuous-delivery with bundleId : 870ee7d6a4f5853229275cca14b604b8
Bundle 870ee7d6a4f5853229275cca14b604b8 expected scale 1 is met
Scaled 870ee7d6a4f5853229275cca14b604b8 to : 1.

Restoring bundle : eslite.
[##################################################] 100%
Loaded eslite with bundleId : 57e432d0c647be2bbc83fa8e59ee469b
Bundle 57e432d0c647be2bbc83fa8e59ee469b expected scale 1 is met
Scaled 57e432d0c647be2bbc83fa8e59ee469b to : 1.

Restoring bundle : reactive-maps-frontend.
[##################################################] 100%
Loaded reactive-maps-frontend with bundleId : 69aa58f00909e36a3cb4229414698b64
Bundle 69aa58f00909e36a3cb4229414698b64 expected scale 0 is met
Scaled 69aa58f00909e36a3cb4229414698b64 to : 0.

Restoring bundle : reactive-maps-backend-summary.
[##################################################] 100%
Loaded reactive-maps-backend-summary with bundleId : abf60451c6af18adcc851d67b369b7f5
Bundle abf60451c6af18adcc851d67b369b7f5 expected scale 0 is met
Scaled abf60451c6af18adcc851d67b369b7f5 to : 0.

Restoring bundle : reactive-maps-backend-region.
[##################################################] 100%
Loaded reactive-maps-backend-region with bundleId : 6273d7a5b059d0e978c6d69ee1a5d7b4
Bundle 6273d7a5b059d0e978c6d69ee1a5d7b4 expected scale 0 is met
Scaled 6273d7a5b059d0e978c6d69ee1a5d7b4 to : 0.

# ashutosh @ ashutoshs-MacBook-Pro in ~/github/conductr-cli on git:restore_backup x [17:03:40] C:1
$ conduct stop eslite
Bundle stop request sent.
Bundle 57e432d0c647be2bbc83fa8e59ee469b waiting to reach expected scale 0
Bundle 57e432d0c647be2bbc83fa8e59ee469b has scale 1, expected 0
Bundle 57e432d0c647be2bbc83fa8e59ee469b expected scale 0 is met
Unload bundle with:       conduct unload 57e432d
Print ConductR info with: conduct info
Print bundle info with:   conduct info 57e432d

# ashutosh @ ashutoshs-MacBook-Pro in ~/github/conductr-cli on git:restore_backup x [17:03:48]
$ conduct restore local.zip

Restoring bundle : continuous-delivery.
[##################################################] 100%
Loaded continuous-delivery with bundleId : 870ee7d6a4f5853229275cca14b604b8
Bundle 870ee7d6a4f5853229275cca14b604b8 expected scale 1 is met
Scaled 870ee7d6a4f5853229275cca14b604b8 to : 1.

Restoring bundle : eslite.
[##################################################] 100%
Loaded eslite with bundleId : 57e432d0c647be2bbc83fa8e59ee469b
Bundle 57e432d0c647be2bbc83fa8e59ee469b waiting to reach expected scale 1
Bundle 57e432d0c647be2bbc83fa8e59ee469b has scale 0, expected 1
Bundle 57e432d0c647be2bbc83fa8e59ee469b expected scale 1 is met
Scaled 57e432d0c647be2bbc83fa8e59ee469b to : 1.

Restoring bundle : reactive-maps-frontend.
[##################################################] 100%
Loaded reactive-maps-frontend with bundleId : 69aa58f00909e36a3cb4229414698b64
Bundle 69aa58f00909e36a3cb4229414698b64 expected scale 0 is met
Scaled 69aa58f00909e36a3cb4229414698b64 to : 0.

Restoring bundle : reactive-maps-backend-summary.
[##################################################] 100%
Loaded reactive-maps-backend-summary with bundleId : abf60451c6af18adcc851d67b369b7f5
Bundle abf60451c6af18adcc851d67b369b7f5 expected scale 0 is met
Scaled abf60451c6af18adcc851d67b369b7f5 to : 0.

Restoring bundle : reactive-maps-backend-region.
[##################################################] 100%
Loaded reactive-maps-backend-region with bundleId : 6273d7a5b059d0e978c6d69ee1a5d7b4
Bundle 6273d7a5b059d0e978c6d69ee1a5d7b4 expected scale 0 is met
Scaled 6273d7a5b059d0e978c6d69ee1a5d7b4 to : 0.
ashutoshraina commented 7 years ago

Manual Test : Backup from a Cluster where the Bundle is stopped Run the Bundle Restore the backup

# ashutosh @ ashutoshs-MacBook-Pro in ~/github/conductr-cli on git:restore_backup x [17:42:58] C:127
$ conduct run reactive-maps-backend-region
Bundle run request sent.
Bundle 6273d7a5b059d0e978c6d69ee1a5d7b4 waiting to reach expected scale 1
Bundle 6273d7a5b059d0e978c6d69ee1a5d7b4 has scale 0, expected 1
Bundle 6273d7a5b059d0e978c6d69ee1a5d7b4 expected scale 1 is met
Stop bundle with:         conduct stop 6273d7a
Print ConductR info with: conduct info
Print bundle info with:   conduct info 6273d7a

# ashutosh @ ashutoshs-MacBook-Pro in ~/github/conductr-cli on git:restore_backup x [17:43:33]
$ conduct restore local.zip

Restoring bundle : continuous-delivery.
[##################################################] 100%
Loaded continuous-delivery with bundleId : 870ee7d6a4f5853229275cca14b604b8
Bundle 870ee7d6a4f5853229275cca14b604b8 expected scale 1 is met
Scaled 870ee7d6a4f5853229275cca14b604b8 to : 1.

Restoring bundle : eslite.
[##################################################] 100%
Loaded eslite with bundleId : 57e432d0c647be2bbc83fa8e59ee469b
Bundle 57e432d0c647be2bbc83fa8e59ee469b expected scale 1 is met
Scaled 57e432d0c647be2bbc83fa8e59ee469b to : 1.

Restoring bundle : reactive-maps-frontend.
[##################################################] 100%
Loaded reactive-maps-frontend with bundleId : 69aa58f00909e36a3cb4229414698b64
Bundle 69aa58f00909e36a3cb4229414698b64 expected scale 0 is met
Scaled 69aa58f00909e36a3cb4229414698b64 to : 0.

Restoring bundle : reactive-maps-backend-summary.
[##################################################] 100%
Loaded reactive-maps-backend-summary with bundleId : abf60451c6af18adcc851d67b369b7f5
Bundle abf60451c6af18adcc851d67b369b7f5 expected scale 0 is met
Scaled abf60451c6af18adcc851d67b369b7f5 to : 0.

Restoring bundle : reactive-maps-backend-region.
[##################################################] 100%
Loaded reactive-maps-backend-region with bundleId : 6273d7a5b059d0e978c6d69ee1a5d7b4
Bundle 6273d7a5b059d0e978c6d69ee1a5d7b4 waiting to reach expected scale 0
Bundle 6273d7a5b059d0e978c6d69ee1a5d7b4 expected scale 0 is met
Scaled 6273d7a5b059d0e978c6d69ee1a5d7b4 to : 0.

# ashutosh @ ashutoshs-MacBook-Pro in ~/github/conductr-cli on git:restore_backup x [17:43:54] C:1
$ conduct info
Licensed To: 2e8b3724-d81e-4ba5-94b1-47bd9da02775
Max ConductR agents: 3
Expires In: 44 days (Wed 20 Sep 2017 14:41PM)
ConductR Version(s): 2.1.*, 2.1.0-beta.1
Grants: akka-sbr, cinnamon, conductr

ID       NAME                                    TAG  #REP  #STR  #RUN  ROLES
69aa58f  reactive-maps-frontend         1.0-SNAPSHOT     1     0     0  dmz
abf6045  reactive-maps-backend-summary                   1     0     0  intranet
57e432d  eslite                                2.1.0     1     0     1  elasticsearch
6273d7a  reactive-maps-backend-region                    1     0     0  intranet
870ee7d  continuous-delivery                   2.1.0     1     0     1  continuous-delivery
ashutoshraina commented 7 years ago

Manual Testing A Failed Scale in Restore

# ashutosh @ ashutoshs-MacBook-Pro in ~/github/conductr-cli on git:restore_backup x [14:41:28] C:1
$ conduct restore elastic.zip

Restoring bundle : elasticsearch.
[##################################################] 100%
Loaded elasticsearch with bundleId : 85dd2657bf86e5ed817c6cbe9d4c18e3-15dbf1287911386ba0b179a616775161
Bundle 85dd2657bf86e5ed817c6cbe9d4c18e3-15dbf1287911386ba0b179a616775161 waiting to reach expected scale 4
Bundle 85dd2657bf86e5ed817c6cbe9d4c18e3-15dbf1287911386ba0b179a616775161 has scale 0, expected 4..
Bundle 85dd2657bf86e5ed817c6cbe9d4c18e3-15dbf1287911386ba0b179a616775161 has scale 1, expected 4...
Error: Failure to scale bundle 85dd2657bf86e5ed817c6cbe9d4c18e3-15dbf1287911386ba0b179a616775161

Check latest bundle events with:
  conduct events 85dd2657bf86e5ed817c6cbe9d4c18e3-15dbf1287911386ba0b179a616775161
Current bundle events:
TIME                          EVENT                                         DESC
Tue 2017-08-08T14:41:38-0700  conductr.loadExecutor.bundleWritten           Bundle written
Tue 2017-08-08T14:41:38-0700  conductr.scaleScheduler.scaleBundleRequested  Scale bundle requested: scale=4
Tue 2017-08-08T14:41:39-0700  conductr.launcher.bundleStarted               Bundle started: host=192.168.10.1

Check latest bundle logs with:
  conduct logs 85dd2657bf86e5ed817c6cbe9d4c18e3-15dbf1287911386ba0b179a616775161
Current bundle logs:
TIME                          HOST                         LOG
Tue 2017-08-08T14:41:42-0700  ashutoshs-MacBook-Pro.local  [2017-08-08 14:41:42,432][INFO ][node                     ] [Amiko Kobayashi] initialized
Tue 2017-08-08T14:41:42-0700  ashutoshs-MacBook-Pro.local  [2017-08-08 14:41:42,433][INFO ][node                     ] [Amiko Kobayashi] starting ...
Tue 2017-08-08T14:41:42-0700  ashutoshs-MacBook-Pro.local  [2017-08-08 14:41:42,672][INFO ][transport                ] [Amiko Kobayashi] bound_address {inet[/192.168.10.1:10647]}, publish_address {inet[/192.168.10.1:10647]}
Tue 2017-08-08T14:41:42-0700  ashutoshs-MacBook-Pro.local  [2017-08-08 14:41:42,692][INFO ][discovery                ] [Amiko Kobayashi] conductr-elasticsearch/nWQ6vhy0Q-idbIDFuD1_vQ
Tue 2017-08-08T14:41:45-0700  ashutoshs-MacBook-Pro.local  [2017-08-08 14:41:45,721][INFO ][cluster.service          ] [Amiko Kobayashi] new_master [Amiko Kobayashi][nWQ6vhy0Q-idbIDFuD1_vQ][ashutoshs-MacBook-Pro.local][inet[/192.168.10.1:10647]], reason: zen-disco-join (elected_as_master)
Tue 2017-08-08T14:41:45-0700  ashutoshs-MacBook-Pro.local  [2017-08-08 14:41:45,782][INFO ][http                     ] [Amiko Kobayashi] bound_address {inet[/192.168.10.1:10410]}, publish_address {inet[/192.168.10.1:10410]}
Tue 2017-08-08T14:41:45-0700  ashutoshs-MacBook-Pro.local  [2017-08-08 14:41:45,782][INFO ][node                     ] [Amiko Kobayashi] started
Tue 2017-08-08T14:41:45-0700  ashutoshs-MacBook-Pro.local  [2017-08-08 14:41:45,787][INFO ][gateway                  ] [Amiko Kobayashi] recovered [0] indices into cluster_state
Tue 2017-08-08T14:41:46-0700  ashutoshs-MacBook-Pro.local  Bundle status successfully reported.
Tue 2017-08-08T14:41:46-0700  ashutoshs-MacBook-Pro.local  Component conductr-elasticsearch-status exited with 0

Error: Bundle 85dd2657bf86e5ed817c6cbe9d4c18e3-15dbf1287911386ba0b179a616775161 has error

Inspect the latest bundle events and logs using:
  conduct events 85dd2657bf86e5ed817c6cbe9d4c18e3-15dbf1287911386ba0b179a616775161
  conduct logs 85dd2657bf86e5ed817c6cbe9d4c18e3-15dbf1287911386ba0b179a616775161
Error: elasticsearch could not be scaled.

# ashutosh @ ashutoshs-MacBook-Pro in ~/github/conductr-cli on git:restore_backup x [14:42:07] C:1
$   conduct events 85dd2657bf86e5ed817c6cbe9d4c18e3-15dbf1287911386ba0b179a616775161
TIME                          EVENT                                                          DESC
Tue 2017-08-08T14:41:38-0700  conductr.loadExecutor.bundleWritten                            Bundle written
Tue 2017-08-08T14:41:38-0700  conductr.scaleScheduler.scaleBundleRequested                   Scale bundle requested: scale=4
Tue 2017-08-08T14:41:39-0700  conductr.launcher.bundleStarted                                Bundle started: host=192.168.10.1
Tue 2017-08-08T14:42:06-0700  conductr.bundleScaler.signalScaleBundleFailedDueToNoResources  Scale bundle failed due to no resources available in the cluster.
Bundle scheduling requirement
cpus=2.0, memory=3.8MiB, disk=10.0MB, roles=[elasticsearch]

Recently declined offers:

Conduct Info Response

$ conduct info
Licensed To: 2e8b3724-d81e-4ba5-94b1-47bd9da02775
Max ConductR agents: 3
Expires In: 43 days (Wed 20 Sep 2017 14:41PM)
ConductR Version(s): 2.1.*, 2.1.0-beta.1
Grants: akka-sbr, cinnamon, conductr

ID                 NAME                   TAG  #REP  #STR  #RUN  ROLES
870ee7d            continuous-delivery  2.1.0     1     0     1  continuous-delivery
57e432d            eslite               2.1.0     1     0     1  elasticsearch
! 85dd265-15dbf12  elasticsearch                  1     0     1  elasticsearch
There are errors: use `conduct events` or `conduct logs` for further information
ashutoshraina commented 7 years ago

@longshorej I think we are good to go now. Can you also look at guides post these merges https://github.com/typesafehub/conductr-doc/pull/417 and https://github.com/typesafehub/conductr-doc/pull/416 to see if the guides are in sync and give the user a good start.

ashutoshraina commented 7 years ago

Did manual tests again post the fixes. We should be good to go now.