scylladb / scylla-manager

The Scylla Manager
https://manager.docs.scylladb.com/stable/
Other
51 stars 33 forks source link

Backup task creation failed: the path must be DNS compliant #3086

Open ShlomiBalalis opened 2 years ago

ShlomiBalalis commented 2 years ago

Client version: 3.0-dev-0.20220402.bd72c550f37 Server version: 3.0-dev-0.20220402.bd72c550f37 Multi DC cluster (two nodes in us-east-1, one node in us-west-2)

When I execute a default backup, there are no issues. But if I execute a backup with a specified path as part of the location list, the command fails:

sctool backup -c 308764ad-4379-4cc7-9ffa-04ebded6d9cf --location s3:manager-backup-tests-us-east-1/path_testing/
Exit status: 1
Error: malformed request: evaluate properties: extract retention for task 97ffa869-04e9-4fc4-9642-e8a13ec62f9d: invalid location "s3:manager-backup-tests-us-east-1/path_testing/", the format is [dc:]<provider>:<path> ex. s3:my-bucket, the path must be DNS compliant
Trace ID: srMX_eVjRsCMQ0kyHJi3lw (grep in scylla-manager logs)

Fom the manager server log:

Apr 03 13:38:49 manager-regression-manager--monitor-node-2e9be272-1 scylla-manager[20741]: {"L":"INFO","T":"2022-04-03T13:38:49.303Z","N":"backup","M":"Generating backup target","cluster_id":"308764ad-4379-4cc7-9ffa-04ebded6d9cf","_trace_id":"srMX_eVjRsCMQ0kyHJi3lw"}
Apr 03 13:38:49 manager-regression-manager--monitor-node-2e9be272-1 scylla-manager[20741]: {"L":"INFO","T":"2022-04-03T13:38:49.303Z","N":"http","M":"POST /api/v1/cluster/308764ad-4379-4cc7-9ffa-04ebded6d9cf/tasks","from":"127.0.0.1:37992","status":400,"bytes":432,"duration":"3ms","error":"create backup target: unexpected end of JSON input","_trace_id":"srMX_eVjRsCMQ0kyHJi3lw"}

From what I understand about the error, I need to specify the dc name and can no longer backup everything to a single bucket like we used to? But if so, why does it only fails when specifying a path inside the bucket?

Logs: db-cluster | https://cloudius-jenkins-test.s3.amazonaws.com/2e9be272-548a-4a17-9767-ff527e8efe33/20220403_141534/db-cluster-2e9be272.tar.gz loader-set | https://cloudius-jenkins-test.s3.amazonaws.com/2e9be272-548a-4a17-9767-ff527e8efe33/20220403_141534/loader-set-2e9be272.tar.gz monitor-set | https://cloudius-jenkins-test.s3.amazonaws.com/2e9be272-548a-4a17-9767-ff527e8efe33/20220403_141534/monitor-set-2e9be272.tar.gz sct | https://cloudius-jenkins-test.s3.amazonaws.com/2e9be272-548a-4a17-9767-ff527e8efe33/20220403_141534/sct-runner-2e9be272.tar.gz (The manager server log is on the monitor-set. The agents' is on the db-cluster)

kevinbarbour commented 2 years ago

@ShlomiBalalis

Reopening to track an issue with underscores in paths:

 Command: 'sudo sctool backup -c b25d296b-52ea-4e01-9c7c-5761b00aaaa9 --location s3:manager-backup-tests-us-east-1/path_testing/ '

Exit code: 1

Stdout:

Stderr:

Error: malformed request: evaluate properties: extract retention for task b31aadce-b2f2-4432-8562-ab258361082d: invalid location "s3:manager-backup-tests-us-east-1/path_testing/", the format is [dc:]<provider>:<path> ex. s3:my-bucket, the path must be DNS compliant
Trace ID: CBTa_n9hQE2dCA-SWvNKvw (grep in scylla-manager logs)
juliayakovlev commented 2 years ago

This issue looks like was re-created on the cloud.

Manager version: Client version: 2.6.3-0.20220307.89b222a1a68 Server version: 2.6.3-0.20220307.89b222a1a68

Command: '/opt/scylla/scylla-cloud -c /opt/scylla/scylla-dbaas.yml cluster manager sctool --cluster-id 17392 --  backup  --location AWS_EU_WEST_1 '
Exit code: 1
Stdout:
Error: malformed request: evaluate properties: extract retention for task d7df2d72-90e6-4b28-8415-4b15761f70f3: invalid location "AWS_EU_WEST_1", the format is [dc:]<provider>:<path> ex. s3:my-bucket, the path must be DNS compliant
Trace ID: AOWZiC14TMGrDFhyMt6AYw (grep in scylla-manager logs)
Stderr:
{"level":"info","ts":1650603467.2875085,"caller":"scylla-cloud/main.go:66","msg":"Command failed","error":"m.Handler.Cluster.ManagerSctoolRun: ssh.Output: command: SCYLLA_MANAGER_CLUSTER=#17392 sctool --api-cert-file /home/centos/manager_client.crt --api-key-file /home/centos/manager_client.key backup --location AWS_EU_WEST_1: Process exited with status 1","errorVerbose":"Process exited with status 1\[ngithub.com/scylladb/siren/cluster/ssh.wrapError\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/cluster/ssh/error.go:66\ngithub.com/scylladb/siren/cluster/ssh.(*sshClientSessionRunner).Session\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/cluster/ssh/connection.go:251\ngithub.com/scylladb/siren/cluster/ssh.connection.Session\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/cluster/ssh/connection.go:259\ngithub.com/scylladb/siren/cluster/ssh.(*client).Session\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/cluster/ssh/client.go:137\ngithub.com/scylladb/siren/cluster/ssh.Output\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/cluster/ssh/connection.go:149\ngithub.com/scylladb/siren/cluster/ssh.(*client).Output\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/cluster/ssh/client.go:155\ngithub.com/scylladb/siren/cluster.(*Service).ManagerSctoolRun\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/cluster/manager.go:91\ngithub.com/scylladb/siren/command/cluster/manager.(*sctoolCmd).run\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/cluster/manager/sctool.go:51\ngithub.com/scylladb/siren/command/cluster/context.BuildContextMiddleware.func1.1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/cluster/context/context.go:53\ngithub.com/scylladb/siren/command.Use.func1.1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36\ngithub.com/scylladb/siren/command.Use.func1.1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36\ngithub.com/scylladb/siren/command.Use.func1.1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36\ngithub.com/scylladb/siren/command.Use.func1.1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36\ngithub.com/scylladb/siren/command.(*App).TreeMiddleware.func1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/tree.go:20\ngithub.com/scylladb/siren/command.Use.func1.1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/vendor/github.com/spf13/cobra/command.go:850\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/vendor/github.com/spf13/cobra/command.go:958\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/vendor/github.com/spf13/cobra/command.go:895\nmain.main\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/cmd/scylla-cloud/main.go:42\nruntime.main\n\t/opt/hostedtoolcache/go/1.16.5/x64/src/runtime/proc.go:225\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.16.5/x64/src/runtime/asm_amd64.s:1371\ncommand](http://ngithub.com/scylladb/siren/cluster/ssh.wrapError%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/cluster/ssh/error.go:66%5Cngithub.com/scylladb/siren/cluster/ssh.(*sshClientSessionRunner).Session%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/cluster/ssh/connection.go:251%5Cngithub.com/scylladb/siren/cluster/ssh.connection.Session%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/cluster/ssh/connection.go:259%5Cngithub.com/scylladb/siren/cluster/ssh.(*client).Session%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/cluster/ssh/client.go:137%5Cngithub.com/scylladb/siren/cluster/ssh.Output%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/cluster/ssh/connection.go:149%5Cngithub.com/scylladb/siren/cluster/ssh.(*client).Output%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/cluster/ssh/client.go:155%5Cngithub.com/scylladb/siren/cluster.(*Service).ManagerSctoolRun%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/cluster/manager.go:91%5Cngithub.com/scylladb/siren/command/cluster/manager.(*sctoolCmd).run%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/cluster/manager/sctool.go:51%5Cngithub.com/scylladb/siren/command/cluster/context.BuildContextMiddleware.func1.1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/cluster/context/context.go:53%5Cngithub.com/scylladb/siren/command.Use.func1.1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36%5Cngithub.com/scylladb/siren/command.Use.func1.1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36%5Cngithub.com/scylladb/siren/command.Use.func1.1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36%5Cngithub.com/scylladb/siren/command.Use.func1.1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36%5Cngithub.com/scylladb/siren/command.(*App).TreeMiddleware.func1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/tree.go:20%5Cngithub.com/scylladb/siren/command.Use.func1.1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36%5Cngithub.com/spf13/cobra.(*Command).execute%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/vendor/github.com/spf13/cobra/command.go:850%5Cngithub.com/spf13/cobra.(*Command).ExecuteC%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/vendor/github.com/spf13/cobra/command.go:958%5Cngithub.com/spf13/cobra.(*Command).Execute%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/vendor/github.com/spf13/cobra/command.go:895%5Cnmain.main%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/cmd/scylla-cloud/main.go:42%5Cnruntime.main%5Cn%5Ct/opt/hostedtoolcache/go/1.16.5/x64/src/runtime/proc.go:225%5Cnruntime.goexit%5Cn%5Ct/opt/hostedtoolcache/go/1.16.5/x64/src/runtime/asm_amd64.s:1371%5Cncommand): SCYLLA_MANAGER_CLUSTER=#17392 sctool --api-cert-file /home/centos/manager_client.crt --api-key-file /home/centos/manager_client.key backup --location AWS_EU_WEST_1\[ngithub.com/scylladb/siren/cluster/ssh.Output\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/cluster/ssh/connection.go:151\ngithub.com/scylladb/siren/cluster/ssh.(*client).Output\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/cluster/ssh/client.go:155\ngithub.com/scylladb/siren/cluster.(*Service).ManagerSctoolRun\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/cluster/manager.go:91\ngithub.com/scylladb/siren/command/cluster/manager.(*sctoolCmd).run\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/cluster/manager/sctool.go:51\ngithub.com/scylladb/siren/command/cluster/context.BuildContextMiddleware.func1.1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/cluster/context/context.go:53\ngithub.com/scylladb/siren/command.Use.func1.1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36\ngithub.com/scylladb/siren/command.Use.func1.1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36\ngithub.com/scylladb/siren/command.Use.func1.1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36\ngithub.com/scylladb/siren/command.Use.func1.1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36\ngithub.com/scylladb/siren/command.(*App).TreeMiddleware.func1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/tree.go:20\ngithub.com/scylladb/siren/command.Use.func1.1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/vendor/github.com/spf13/cobra/command.go:850\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/vendor/github.com/spf13/cobra/command.go:958\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/vendor/github.com/spf13/cobra/command.go:895\nmain.main\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/cmd/scylla-cloud/main.go:42\nruntime.main\n\t/opt/hostedtoolcache/go/1.16.5/x64/src/runtime/proc.go:225\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.16.5/x64/src/runtime/asm_amd64.s:1371\nssh.Output\ngithub.com/scylladb/siren/cluster.(*Service).ManagerSctoolRun\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/cluster/manager.go:93\ngithub.com/scylladb/siren/command/cluster/manager.(*sctoolCmd).run\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/cluster/manager/sctool.go:51\ngithub.com/scylladb/siren/command/cluster/context.BuildContextMiddleware.func1.1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/cluster/context/context.go:53\ngithub.com/scylladb/siren/command.Use.func1.1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36\ngithub.com/scylladb/siren/command.Use.func1.1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36\ngithub.com/scylladb/siren/command.Use.func1.1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36\ngithub.com/scylladb/siren/command.Use.func1.1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36\ngithub.com/scylladb/siren/command.(*App).TreeMiddleware.func1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/tree.go:20\ngithub.com/scylladb/siren/command.Use.func1.1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/vendor/github.com/spf13/cobra/command.go:850\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/vendor/github.com/spf13/cobra/command.go:958\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/vendor/github.com/spf13/cobra/command.go:895\nmain.main\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/cmd/scylla-cloud/main.go:42\nruntime.main\n\t/opt/hostedtoolcache/go/1.16.5/x64/src/runtime/proc.go:225\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.16.5/x64/src/runtime/asm_amd64.s:1371\nm.Handler.Cluster.ManagerSctoolRun\ngithub.com/scylladb/siren/command/cluster/manager.(*sctoolCmd).run\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/cluster/manager/sctool.go:55\ngithub.com/scylladb/siren/command/cluster/context.BuildContextMiddleware.func1.1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/cluster/context/context.go:53\ngithub.com/scylladb/siren/command.Use.func1.1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36\ngithub.com/scylladb/siren/command.Use.func1.1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36\ngithub.com/scylladb/siren/command.Use.func1.1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36\ngithub.com/scylladb/siren/command.Use.func1.1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36\ngithub.com/scylladb/siren/command.(*App).TreeMiddleware.func1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/tree.go:20\ngithub.com/scylladb/siren/command.Use.func1.1\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/vendor/github.com/spf13/cobra/command.go:850\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/vendor/github.com/spf13/cobra/command.go:958\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/vendor/github.com/spf13/cobra/command.go:895\nmain.main\n\t/home/runner/work/siren/siren/src/github.com/scylladb/siren/cmd/scylla-cloud/main.go:42\nruntime.main\n\t/opt/hostedtoolcache/go/1.16.5/x64/src/runtime/proc.go:225\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.16.5/x64/src/runtime/asm_amd64.s:1371](http://ngithub.com/scylladb/siren/cluster/ssh.Output%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/cluster/ssh/connection.go:151%5Cngithub.com/scylladb/siren/cluster/ssh.(*client).Output%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/cluster/ssh/client.go:155%5Cngithub.com/scylladb/siren/cluster.(*Service).ManagerSctoolRun%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/cluster/manager.go:91%5Cngithub.com/scylladb/siren/command/cluster/manager.(*sctoolCmd).run%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/cluster/manager/sctool.go:51%5Cngithub.com/scylladb/siren/command/cluster/context.BuildContextMiddleware.func1.1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/cluster/context/context.go:53%5Cngithub.com/scylladb/siren/command.Use.func1.1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36%5Cngithub.com/scylladb/siren/command.Use.func1.1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36%5Cngithub.com/scylladb/siren/command.Use.func1.1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36%5Cngithub.com/scylladb/siren/command.Use.func1.1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36%5Cngithub.com/scylladb/siren/command.(*App).TreeMiddleware.func1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/tree.go:20%5Cngithub.com/scylladb/siren/command.Use.func1.1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36%5Cngithub.com/spf13/cobra.(*Command).execute%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/vendor/github.com/spf13/cobra/command.go:850%5Cngithub.com/spf13/cobra.(*Command).ExecuteC%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/vendor/github.com/spf13/cobra/command.go:958%5Cngithub.com/spf13/cobra.(*Command).Execute%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/vendor/github.com/spf13/cobra/command.go:895%5Cnmain.main%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/cmd/scylla-cloud/main.go:42%5Cnruntime.main%5Cn%5Ct/opt/hostedtoolcache/go/1.16.5/x64/src/runtime/proc.go:225%5Cnruntime.goexit%5Cn%5Ct/opt/hostedtoolcache/go/1.16.5/x64/src/runtime/asm_amd64.s:1371%5Cnssh.Output%5Cngithub.com/scylladb/siren/cluster.(*Service).ManagerSctoolRun%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/cluster/manager.go:93%5Cngithub.com/scylladb/siren/command/cluster/manager.(*sctoolCmd).run%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/cluster/manager/sctool.go:51%5Cngithub.com/scylladb/siren/command/cluster/context.BuildContextMiddleware.func1.1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/cluster/context/context.go:53%5Cngithub.com/scylladb/siren/command.Use.func1.1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36%5Cngithub.com/scylladb/siren/command.Use.func1.1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36%5Cngithub.com/scylladb/siren/command.Use.func1.1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36%5Cngithub.com/scylladb/siren/command.Use.func1.1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36%5Cngithub.com/scylladb/siren/command.(*App).TreeMiddleware.func1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/tree.go:20%5Cngithub.com/scylladb/siren/command.Use.func1.1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36%5Cngithub.com/spf13/cobra.(*Command).execute%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/vendor/github.com/spf13/cobra/command.go:850%5Cngithub.com/spf13/cobra.(*Command).ExecuteC%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/vendor/github.com/spf13/cobra/command.go:958%5Cngithub.com/spf13/cobra.(*Command).Execute%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/vendor/github.com/spf13/cobra/command.go:895%5Cnmain.main%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/cmd/scylla-cloud/main.go:42%5Cnruntime.main%5Cn%5Ct/opt/hostedtoolcache/go/1.16.5/x64/src/runtime/proc.go:225%5Cnruntime.goexit%5Cn%5Ct/opt/hostedtoolcache/go/1.16.5/x64/src/runtime/asm_amd64.s:1371%5Cnm.Handler.Cluster.ManagerSctoolRun%5Cngithub.com/scylladb/siren/command/cluster/manager.(*sctoolCmd).run%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/cluster/manager/sctool.go:55%5Cngithub.com/scylladb/siren/command/cluster/context.BuildContextMiddleware.func1.1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/cluster/context/context.go:53%5Cngithub.com/scylladb/siren/command.Use.func1.1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36%5Cngithub.com/scylladb/siren/command.Use.func1.1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36%5Cngithub.com/scylladb/siren/command.Use.func1.1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36%5Cngithub.com/scylladb/siren/command.Use.func1.1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36%5Cngithub.com/scylladb/siren/command.(*App).TreeMiddleware.func1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/tree.go:20%5Cngithub.com/scylladb/siren/command.Use.func1.1%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/command/middleware.go:36%5Cngithub.com/spf13/cobra.(*Command).execute%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/vendor/github.com/spf13/cobra/command.go:850%5Cngithub.com/spf13/cobra.(*Command).ExecuteC%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/vendor/github.com/spf13/cobra/command.go:958%5Cngithub.com/spf13/cobra.(*Command).Execute%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/vendor/github.com/spf13/cobra/command.go:895%5Cnmain.main%5Cn%5Ct/home/runner/work/siren/siren/src/github.com/scylladb/siren/cmd/scylla-cloud/main.go:42%5Cnruntime.main%5Cn%5Ct/opt/hostedtoolcache/go/1.16.5/x64/src/runtime/proc.go:225%5Cnruntime.goexit%5Cn%5Ct/opt/hostedtoolcache/go/1.16.5/x64/src/runtime/asm_amd64.s:1371)"}
Error: m.Handler.Cluster.ManagerSctoolRun: ssh.Output: command: SCYLLA_MANAGER_CLUSTER=#17392 sctool --api-cert-file /home/centos/manager_client.crt --api-key-file /home/centos/manager_client.key backup --location AWS_EU_WEST_1: Process exited with status 1

Installation details Kernel version: 5.13.0-1021-aws Scylla version (or git commit hash): 2021.1.10-0.20220410.e8e681dee with build-id f407f6d42e2570c781e916292747e651dd7f50da Cluster size: 6 nodes (i3.2xlarge) Scylla running with shards number (live nodes): longevity-100gb-48h-cloud-master-db-node-535764e0-7 (54.72.239.239 | 172.23.105.182): 8 shards Scylla running with shards number (terminated nodes): longevity-100gb-48h-cloud-master-db-node-535764e0-5 (63.35.74.191 | 172.23.105.22): None shards longevity-100gb-48h-cloud-master-db-node-535764e0-6 (176.34.228.239 | 172.23.106.237): None shards longevity-100gb-48h-cloud-master-db-node-535764e0-2 (52.210.87.210 | 172.23.105.224): None shards longevity-100gb-48h-cloud-master-db-node-535764e0-3 (52.214.16.183 | 172.23.104.33): None shards longevity-100gb-48h-cloud-master-db-node-535764e0-1 (52.49.117.62 | 172.23.106.245): None shards longevity-100gb-48h-cloud-master-db-node-535764e0-8 (52.213.72.142 | 172.23.106.238): None shards longevity-100gb-48h-cloud-master-db-node-535764e0-4 (99.81.127.167 | 172.23.104.60): None shards longevity-100gb-48h-cloud-master-db-node-535764e0-9 (63.35.143.176 | 172.23.104.92): None shards OS (RHEL/CentOS/Ubuntu/AWS AMI): ami-02baaf20135f94bac (aws: eu-west-1)

Test: cloud-longevity-100gb-3d Test name: longevity_test.LongevityTest.test_custom_time Test config file(s):

Restore Monitor Stack command: $ hydra investigate show-monitor 535764e0-8eb3-414d-86c0-00d44ccd7f00 Restore monitor on AWS instance using Jenkins job Show all stored logs command: $ hydra investigate show-logs 535764e0-8eb3-414d-86c0-00d44ccd7f00

Test id: 535764e0-8eb3-414d-86c0-00d44ccd7f00

Logs:

db-cluster - https://cloudius-jenkins-test.s3.amazonaws.com/535764e0-8eb3-414d-86c0-00d44ccd7f00/20220422_121147/db-cluster-535764e0.tar.gz

sct - https://cloudius-jenkins-test.s3.amazonaws.com/535764e0-8eb3-414d-86c0-00d44ccd7f00/20220422_121147/sct-runner-535764e0.tar.gz

siren-manager-set - https://cloudius-jenkins-test.s3.amazonaws.com/535764e0-8eb3-414d-86c0-00d44ccd7f00/20220422_121147/siren-manager-set-535764e0.tar.gz

Jenkins job URL

kevinbarbour commented 2 years ago

@juliayakovlev I don't believe that is related to this issue. That is just an incorrectly formatted location without a provider. The error message given contains the correct formatting required:

The format is [dc:]<provider>:<path> ex. s3:my-bucket

kevinbarbour commented 2 years ago

Reopening and reverting this in 2.6/3.0 - to be added as a new feature in 3.1.

From @ShlomiBalalis

"Good morning! I seem to have made a small mistake. In one of our tests we try to create a backup with a location of "/certain_path" which is suppose to fail. I misread the test result, and because of it I thought the backup is suppose to succeed. Kevin changed the code and now the backup does succeed, but deleting it fails, since the manager cannot find the manifest. Should we revert the option of specifing a path in the location, or alter the delete function to be able to find the manifest even inside a folder?"