How to configure to communicate with S3 clones?

[akiradeveloper]

If I understand correctly your goal is to access filesystems and S3 storages transparently. Good tool. I want to use mc instead of s3cmd.

I am considering If I could pass destination address, port number, the version of authentication scheme (v2 or v4), auth key, secret key and so on that's required to communicate with S3 clones like Riak CS.

Can we write config file like s3cmd and set up for S3 clones?

[akiradeveloper]

sorry. I missed this sentence in README "Update AccessKeyID and SecretAccessKey fields in your ~/.mc/config.json configuration file"

[akiradeveloper]

what's milkyway...? mc uses unusual region so this tool is only for minio, not for other clones. did you test with AWS S3? it should be sending with v2 authentication that's region irrelevant so to talk with any clones.

[harshavardhana]

what's milkyway...? mc uses unusual region so this tool is only for minio, not for other clones. did you test with AWS S3? it should be sending with v2 authentication that's region irrelevant so to talk with any clones.

mc only supports AWS S3 Signature v4, We do not support signature v2.

For now mc tool is only tested for minio and AWS S3 not other clones. We are not planning on supporting signature V2 any time soon.

If RiakCS supports signature V4 then mc can be made to work with RiakCS with few changes.

[harshavardhana]

Easiest way to add your keys for s3 is simple as shown below.

$ mc config host help
....
....
   1. Add host configuration for a URL. For security reasons turn off bash history
      $ set +o history
      $ mc config host add s3.amazonaws.com BKIKJAA5BMMU2RHO6IBB V7f1C\wQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr1
      $ set -o history

   2. List all hosts.
      $ mc config host list

   3. Remove host config.
      $ mc config host remove s3.amazonaws.com

[akiradeveloper]

Thanks.

Heard that you aren't going to make v2 signature available soon but you think it's possible to implement v2? I don't know why you didn't implement v2 that's even easier than v4

[harshavardhana]

Heard that you aren't going to make v2 signature available soon but you think it's possible to implement v2? I don't know why you didn't implement v2 that's even easier than v4

It is our belief to keep things minimal that we only do one way to do one thing. Given a choice if it is signatureV2 v/s signatureV4 - it was signatureV4.

mc is designed to work for S3 first for all regions, so it does what Amazon S3 recommends. Even minio server had to implement signature V4 to be compatible with mc. We choose "milkyway" as the default region for minio.

[akiradeveloper]

I understand your belief but still I want v2 signature. The biggest reason is to use mc for S3 storage software that I am implementing now (which only supports v2). I tried s3cmd and s4cmd but they weren't good enough.

Do you think it is possible to support v2 signature by generalizing signing process? (like defining Singer like in AWS SDK http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/Signer.html). I think the code below is signing the request.

// Do - start the request
func (r *request) Do() (resp *http.Response, err error) {
        if r.config.AccessKeyID != "" && r.config.SecretAccessKey != "" {
                r.SignV4()
        }

[harshavardhana]

I understand your belief but still I want v2 signature. The biggest reason is to use mc for S3 storage software that I am implementing now (which only supports v2). I tried s3cmd and s4cmd but they weren't good enough.

Okay let's see - if we can make it work. I will need some time to get this working..

// Do - start the request func (r request) Do() (resp http.Response, err error) { if r.config.AccessKeyID != "" && r.config.SecretAccessKey != "" { r.SignV4() }

Signing process can be generalized.

[akiradeveloper]

Thanks. I am looking forward to it.

[akiradeveloper]

Oh, Good. I will try

[harshavardhana]

Thanks. I am looking forward to it.

Okay so the Signature V2 support is working fine now with Amazon S3.

Oh, Good. I will try

Okay so here is how you use it.. by default all hosts are going to do signature v4 ie. minio and s3 which already do.

But for new hosts i.e s3 clones you can add them in following manner.

 $ set +o history
 $ mc config host add <your_s3clone_host_port> BKIKJAA5BMMU2RHO6IBB V7f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12 v2
 $ set -o history

May be even add an alias to make it simpler to remember

$ mc config alias add s3clone <your_s3clone_host_port>

[harshavardhana]

Once done do simply

$ mc ls s3clone/

You should be able to see all the buckets, let me know if that is not the case since i do not have a s3clone cluster which supports Signature V2 other than Amazon. I need your help to validate this fix for s3clones.

Thanks for your patience.. I hope this works great for you :-)

[harshavardhana]

To update to latest if you have go installed use

$ go get -u github.com/minio/mc

[akiradeveloper]

Hi, facing a problem in setup.

$ mc config host add <your_s3clone_host_port> BKIKJAA5BMMU2RHO6IBB V7f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12 v2

c: <ERROR> Unrecognized version name provided, supported inputs are ‘S3v4’, ‘S3v2’ Invalid arguments provided, cannot proceed.

it's not v2. but S3v2 I guess.

I tried to create a host with S3v2 (the id and key are from your example) but .mc/config.json is like this. I suspect something is wrong in the code in writing the json file.

                        "accessKeyId": "BKIKJAA5BMMU2RHO6IBB",
                        "secretAccessKey": "V7f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12",
                        "api": "S3v4"

[harshavardhana]

$ mc config host add BKIKJAA5BMMU2RHO6IBB V7f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12 v2

There is no problem, i just changed the code a bit so you got the latest change after i posted the comment previously..

So S3v2 is in-fact correct.

For example

$ mc config host add test.s3.amazonaws.com BKIKJAA5BMMU2RHO6IBB V7f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12 S3v2
Added host ‘test.s3.amazonaws.com’ successfully.

$ mc config host list
[dl.minio.io:9000]
[test.s3.amazonaws.com] <- BKIKJAA5BMMU2RHO6IBB, V7f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12, S3v4
....

[harshavardhana]

Looks like there is a problem.. its not writing S3v2 properly into the file.. can you try editing the file directly until i fix this?

[harshavardhana]

Looks like there is a problem.. its not writing S3v2 properly into the file.. can you try editing the file directly until i fix this?

https://github.com/minio/mc/pull/1116 - fixed it.

[harshavardhana]

Can you again?

$ go get -u github.com/minio/mc

This time should write the API name properly.

[akiradeveloper]

Writing json works well and signature seems to correct but another problem reveals.

Your mc is too unstable.

My server is now listening to localhost:8080 and there is no bucket yet.

I repeatedly try to list the buckets by mc ls. Although this is idempotent operation, the result changes on every attempt.

I will look deeper with debugger on how the request is broken.

hayakawaakira-no-MacBook-Pro:~ akira$ mc ls mys3
mc: <ERROR> Unable to list target ‘http://localhost:8080/’. We encountered an internal error. Please try again.
hayakawaakira-no-MacBook-Pro:~ akira$ mc ls mys3
hayakawaakira-no-MacBook-Pro:~ akira$ mc ls mys3
hayakawaakira-no-MacBook-Pro:~ akira$ mc ls mys3
hayakawaakira-no-MacBook-Pro:~ akira$ mc ls mys3
mc: <ERROR> Unable to list target ‘http://localhost:8080/’. We encountered an internal error. Please try again.

[harshavardhana]

hayakawaakira-no-MacBook-Pro:~ akira$ mc ls mys3 mc: Unable to list target ‘http://localhost:8080/’. We encountered an internal error. Please try again. hayakawaakira-no-MacBook-Pro:~ akira$ mc ls mys3 hayakawaakira-no-MacBook-Pro:~ akira$ mc ls mys3 hayakawaakira-no-MacBook-Pro:~ akira$ mc ls mys3 hayakawaakira-no-MacBook-Pro:~ akira$ mc ls mys3 mc: Unable to list target ‘http://localhost:8080/’. We encountered an internal error. Please try again.

Can you do --debug so that i can see the HTTP trace? - this might be an interaction issue. That it works fine for few times but not for others.

"We encountered an internal error. Please try again." is coming from server it is HTTPStatus 500.

Your mc is too unstable.

I am sorry about that, but i am here to help so feel free to post questions here.

[harshavardhana]

"We encountered an internal error. Please try again." is coming from server it is HTTPStatus 500.

When you do --debug mc will post the whole call trace from the point where the error originated, if its a library problem i can fix it quickly.

You can also join us at https://gitter.im/minio/minio , so that we can work towards getting this working for you.

[akiradeveloper]

Just ran with --debug. The first one is success and the second one is failure.

I wonder why it is going through the s3v4 path? (0) .../src/github.com/minio/mc/pkg/client/s3v4/s3.go:162 s3v4.(*s3Client).Stat(..)

hayakawaakira-no-MacBook-Pro:~ akira$ mc --debug ls mys3
mc: <DEBUG> GET http://localhost:8080/ HTTP/1.1
Host: localhost:8080
User-Agent:  Minio/2015-09-22T04:02:16.008862161Z (mc; darwin; amd64)
Authorization: AWS BCDEFGHIJKLMNOPQRSTA:t+g5gWpmZFzpoW0zn7zHUHmalsE=
Date: Tue, 22 Sep 2015 08:38:15 GMT
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 263
Content-Type: text/xml; charset=UTF-8
Date: Tue, 22 Sep 2015 08:38:15 GMT
Server: spray-can/1.3.3
X-Amz-Request-Id: 87HBwztOXXR0gVlp

mc: <DEBUG> Response Time:  50.501416ms

mc: <DEBUG> GET http://localhost:8080/ HTTP/1.1
Host: localhost:8080
User-Agent:  Minio/2015-09-22T04:02:16.008862161Z (mc; darwin; amd64)
Authorization: AWS BCDEFGHIJKLMNOPQRSTA:t+g5gWpmZFzpoW0zn7zHUHmalsE=
Date: Tue, 22 Sep 2015 08:38:15 GMT
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 263
Content-Type: text/xml; charset=UTF-8
Date: Tue, 22 Sep 2015 08:38:15 GMT
Server: spray-can/1.3.3
X-Amz-Request-Id: yAJF03KbJxIsXc8f

mc: <DEBUG> Response Time:  46.714105ms

hayakawaakira-no-MacBook-Pro:~ akira$ mc --debug ls mys3
mc: <DEBUG> HTTP/1.1 500 Internal Server Error
Content-Length: 217
Content-Type: text/xml; charset=UTF-8
Date: Tue, 22 Sep 2015 08:38:20 GMT
Server: spray-can/1.3.3

mc: <DEBUG> Response Time:  4.8186ms

mc: <ERROR> Unable to list target ‘http://localhost:8080/’. We encountered an internal error. Please try again.
 (2) .../src/github.com/minio/mc/ls-main.go:118 main.mainList(..) Tags: [http://localhost:8080/]
 (1) .../src/github.com/minio/mc/ls.go:111 main.doList(..) Tags: [http://localhost:8080/]
 (0) .../src/github.com/minio/mc/pkg/client/s3v4/s3.go:162 s3v4.(*s3Client).Stat(..)
 Host:hayakawaakira-no-MacBook-Pro.local | OS:darwin | Arch:amd64 | Lang:go1.5.1 | Mem:438kB/4.5MB | Heap:438kB/1.7MB

[harshavardhana]

I wonder why it is going through the s3v4 path? (0) .../src/github.com/minio/mc/pkg/client/s3v4/s3.go:162 s3v4.(*s3Client).Stat(..)

I think i know the problem, do you have two entries of "localhost:" ? in ~/.mc/config.json ?

[akiradeveloper]

Yes.

The one on localhost:8080 and the another on localhost:* which is set by default.

I will erase the latter and try again

[harshavardhana]

I think i know the problem, do you have two entries of "localhost:" ? in ~/.mc/config.json ?

This is our bug, i think the problem is in the regular expression matching two entries, and it picks one randomly.

[harshavardhana]

The one on localhost:8080 and the another on localhost:* which is set by default.

Thanks for reproducing, i will fix it during hostConfig verification.

[akiradeveloper]

I erased the hosts other than localhost:8080 which I am using but still reproduces.

Do I need to do something after updating .mc/config.json?

Here is the trace and the config

hayakawaakira-no-MacBook-Pro:~ akira$ mc --debug ls mys3
mc: <DEBUG> HTTP/1.1 500 Internal Server Error
Content-Length: 217
Content-Type: text/xml; charset=UTF-8
Date: Tue, 22 Sep 2015 08:56:35 GMT
Server: spray-can/1.3.3

mc: <DEBUG> Response Time:  3.18741ms

mc: <ERROR> Unable to list target ‘http://localhost:8080/’. We encountered an internal error. Please try again.
 (2) .../src/github.com/minio/mc/ls-main.go:118 main.mainList(..) Tags: [http://localhost:8080/]
 (1) .../src/github.com/minio/mc/ls.go:111 main.doList(..) Tags: [http://localhost:8080/]
 (0) .../src/github.com/minio/mc/pkg/client/s3v4/s3.go:162 s3v4.(*s3Client).Stat(..)
 Host:hayakawaakira-no-MacBook-Pro.local | OS:darwin | Arch:amd64 | Lang:go1.5.1 | Mem:404kB/4.5MB | Heap:404kB/1.7MB

hayakawaakira-no-MacBook-Pro:~ akira$ cat .mc/config.json
{
        "version": "5",
        "alias": {
                "mys3": "http://localhost:8080"
        },
        "hosts": {
                "localhost:8080": {
                        "accessKeyId": "BCDEFGHIJKLMNOPQRSTA",
                        "secretAccessKey": "bcdefghijklmnopqrstuvwxyzabcdefghijklmna",
                        "api": "S3v2"
                }
        }
}

[harshavardhana]

I erased the hosts other than localhost:8080 which I am using but still reproduces.

Okay, i am going to reproduce this issue locally. Will reply in a bit.

[harshavardhana]

Okay, i am going to reproduce this issue locally. Will reply in a bit.

Reproduced it, fixing it.. Give me few mins.

[harshavardhana]

Do I need to do something after updating .mc/config.json?

Can you update and try again?

[akiradeveloper]

I think it's fixed. Great work!

[harshavardhana]

I think it's fixed. Great work!

Thanks. Let me know if you face any more issues. Feel free to open as many bugs as you can. This is the only way to harden this tool :-)

[harshavardhana]

I think it's fixed. Great work!

Going to close this for now, if you see more issues. Please re-open. Thanks for your patience.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.