search

pshima / consul-snapshot

consul-snapshot is a backup and restore utility for Consul (https://www.consul.io). This is slightly different than some other utilities out there as this runs as a daemon for backups and ships them to S3. Also has integrated monitoring and backup health checks.
Apache License 2.0
116 stars 35 forks source link

Unable to create local restore temp file!: no such file or directory #4

Closed PurrBiscuit closed 8 years ago

PurrBiscuit commented 8 years ago

Been having trouble getting the consul-snapshot restore command to work with the nested directories that the backup creates in the S3 bucket. Here's the result of the consul-snapshot backup command:

root@81cdbbf43a72:/tmp# consul-snapshot backup
[INFO] v0.1.3: Starting Consul Snapshot
2016/06/21 23:56:04 [DEBUG] Backup starting on interval: 15s
2016/06/21 23:56:19 [INFO] Starting Backup At: 1466553379
2016/06/21 23:56:19 [INFO] Listing keys from consul
2016/06/21 23:56:19 [INFO] Converting 17 keys to JSON
2016/06/21 23:56:19 [INFO] Writing Local Backup File
2016/06/21 23:56:19 [DEBUG] Wrote 5850 bytes to file, /tmp/consul.backup.1466553379.gz
2016/06/21 23:56:19 [INFO] Writing Backup to Remote File
2016/06/21 23:56:19 [INFO] Uploading example-consul-snapshots/backups/2016/6/21/consul.backup.1466553379.gz to S3 in us-east-1
2016/06/21 23:56:19 [INFO] Running post processing
2016/06/21 23:56:19 [INFO] Backup completed successfully

Everything works great there with the full path in the S3 bucket becoming example-consul-snapshots/backups/2016/6/21/consul.backup.1466553379.gz. I can see the file there in the bucket.

On restores I get an error because the directory structure isn't in place when the restore attempts to create the temp file locally. Here's the output from the consul-snapshot restore command:

root@81cdbbf43a72:/tmp# consul-snapshot restore backups/2016/6/21/consul.backup.1466553379.gz
[INFO] v0.1.3: Starting Consul Snapshot
2016/06/22 00:05:23 [DEBUG] Starting restore of example-consul-snapshots/backups/2016/6/21/consul.backup.1466553379.gz
2016/06/22 00:05:23 [ERR] Unable to create local restore temp file!: open /tmp/backups/2016/6/21/consul.backup.1466553379.gz: no such file or directory

It seems that it's trying to create the consul.backup.1466553379.gz file in the /tmp/backups/2016/6/21/ path, even though it doesn't exist yet. If I created that /backups/2016/6/21/ path before attempting the restore it works.

root@81cdbbf43a72:/tmp# consul-snapshot restore backups/2016/6/21/consul.backup.1466553379.gz
[INFO] v0.1.3: Starting Consul Snapshot
2016/06/22 00:06:24 [DEBUG] Starting restore of articulate-consul-snapshots/backups/2016/6/21/consul.backup.1466553379.gz
2016/06/22 00:06:24 [INFO] Downloading articulate-consul-snapshots/tmp/backups/2016/6/21/consul.backup.1466553379.gz from S3 in us-east-1
2016/06/22 00:06:25 [INFO] Download completed
2016/06/22 00:06:25 [INFO] Extracting Backup File
2016/06/22 00:06:25 [INFO] Extracted 17 keys to restore
2016/06/22 00:06:25 [INFO] Restored 17 keys with 0 errors
2016/06/22 00:06:25 [INFO] Restore completed.
root@81cdbbf43a72:/tmp/backups/2016/6/21# ls -al
total 4
drwxr-xr-x 2 root root   40 Jun 22 00:06 .
drwxr-xr-x 3 root root   15 Jun 22 00:06 ..
-rw-r--r-- 1 root root 3249 Jun 22 00:06 consul.backup.1466553379.gz

Seems like an os.MkdirAll is needed before the os.Create to create all the nested directories before trying to create the file here: https://github.com/pshima/consul-snapshot/blob/5e2cc24e0970ec224fbc42355cd1a6cde2d61b08/restore/restore.go#L66

pshima commented 8 years ago

Thanks @PurrBiscuit for the report and sorry for the delay in responding. I will take a closer look at this issue.

pshima commented 8 years ago

Fixed in https://github.com/pshima/consul-snapshot/pull/5