rlmcpherson / s3gof3r

Fast, concurrent, streaming access to Amazon S3, including gof3r, a CLI. http://godoc.org/github.com/rlmcpherson/s3gof3r
MIT License
1.15k stars 181 forks source link

doesn't work in AWS china region #96

Open tj13 opened 8 years ago

tj13 commented 8 years ago

Hi all,

per the china region is isolated. the URL pattern of S3 endpoint is diferent, this tool failed with following error:

Attempt 1: passing endpoint via CLI parameter

echo a | gof3r put -b caijiajia -k mysql/a --debug --endpoint "s3.cn-north-1.amazonaws.com.cn"

panic: can't find endpoint region

goroutine 1 [running]: github.com/rlmcpherson/s3gof3r.(_S3).Region(0xc820064640, 0x0, 0x0) /root/src/github.com/rlmcpherson/s3gof3r/s3gof3r.go:41 +0x1ce github.com/rlmcpherson/s3gof3r.(_Bucket).Sign(0xc820092a20, 0xc8200dc1c0) /root/src/github.com/rlmcpherson/s3gof3r/sign.go:56 +0x164 github.com/rlmcpherson/s3gof3r.(_putter).retryRequest(0xc820062600, 0x7fa698, 0x4, 0xc820064700, 0x40, 0x0, 0x0, 0xc82006dce0, 0x0, 0x0, ...) /root/src/github.com/rlmcpherson/s3gof3r/putter.go:382 +0x37e github.com/rlmcpherson/s3gof3r.newPutter(0x7fc570, 0x5, 0x0, 0x0, 0x0, 0xc8200f81b0, 0x28, 0xc8200fa200, 0x8, 0x0, ...) /root/src/github.com/rlmcpherson/s3gof3r/putter.go:88 +0x349 github.com/rlmcpherson/s3gof3r.(_Bucket).PutWriter(0xc82004b9e0, 0x7fff7c996754, 0x7, 0xc82006dce0, 0xc82004ba90, 0x0, 0x0, 0x0, 0x0) /root/src/github.com/rlmcpherson/s3gof3r/s3gof3r.go:139 +0x126 main.(_putOpts).Execute(0x9d6ac0, 0xc8200a6400, 0x0, 0x8, 0x7fbf37c48568, 0xc8200f8150) /root/src/github.com/rlmcpherson/s3gof3r/gof3r/put.go:53 +0x65f github.com/rlmcpherson/s3gof3r/gof3r/vendor/github.com/jessevdk/go-flags.(_Parser).ParseArgs(0xc8200764b0, 0xc820074010, 0x8, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0) /root/src/github.com/rlmcpherson/s3gof3r/gof3r/vendor/github.com/jessevdk/go-flags/parser.go:301 +0xa16 github.com/rlmcpherson/s3gof3r/gof3r/vendor/github.com/jessevdk/go-flags.(*Parser).Parse(0xc8200764b0, 0x0, 0x0, 0x0, 0x0, 0x0) /root/src/github.com/rlmcpherson/s3gof3r/gof3r/vendor/github.com/jessevdk/go-flags/parser.go:175 +0x9b main.main() /root/src/github.com/rlmcpherson/s3gof3r/gof3r/main.go:61 +0x191

goroutine 17 [syscall, locked to thread]: runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1696 +0x1

Attempt 2: set AWS_REGION environment variable.

export AWS_REGION='cn-north-1';echo a | gof3r put -b caijiajia -k mysql/a --debug

gof3r error: 400: "The provided token is malformed or otherwise invalid."

rlmcpherson commented 8 years ago

This is an issue with the current code that assumes all endpoints end with amazonaws.com. I've updated the regexp matcher in a pr here https://github.com/rlmcpherson/s3gof3r/pull/107 I don't have a way to test in the China region, so if you can test it and confirm whether the issue is resolved, that would be helpful @tj13