Question about Using AWS S3 in region "cn-northwest-1"(China)

[zhangsimingshannonai]

My S3 bucket is in AWS-CHINA, region is "cn-northwest-1", but I didn't find anything about this in the src, Is ice support china-region?

I used ice in kubernetes way. Here is my configmap-ice.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: ice
data:
  ice.properties: |
    ice.processor=true
    ice.reader=true
    ice.reservationCapacityPoller=false
    ice.reservationPeriod=oneyear
    ice.reservationUtilization=HEAVY
    ice.highstockUrl=https://code.highcharts.com/stock/4.2.1/highstock.js
    ice.urlPrefix=aws.shannonai
    ice.fromEmail=xxxxxxxxxxxxxxxxxxxxxxxx
    ice.ondemandCostAlertThreshold=250
    ice.s3SecretKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    ice.billing_s3bucketname=xxxxxxxxxxxx
    ice.billing_s3bucketregion=cn-northwest-1
    ice.billing_accessRoleName=xxxxxxxxxxxxxxxxx
    ice.startmillis=1364774400000
    ice.companyName=shannonai
    ice.work_s3bucketname=xxxxxxxxxxxxxxxxxx
    ice.work_s3bucketprefix=ice/
    ice.work_s3bucketregion=cn-northwest-1
    ice.processor.localDir=/mnt/ice_processor
    ice.reader.localDir=/mnt/ice_reader
    ice.monthlycachesize=12

Here is my deployment.yaml's env part

  - name: ice
          image: harbor.shannonai.com/test/ice:latest
          ports:
            - containerPort: 8080
          imagePullPolicy: IfNotPresent
          env: 
            - name: GRAILS_OPTS 
              value: "-server -Xmx2G -Xms1G -Dfile.encoding=UTF-8"
            - name: EC2_REGION
              value: cn-northwest-1
            - name: AWS_DEFAULT_REGION
              value: cn-northwest-1
          command:
            - /opt/ice/grailsw
            - -Djava.net.preferIPv4Stack=true
            - -Djava.net.preferIPv4Addresses
            - -Duser.timezone=Asia/Shanghai
            - -Dice.s3AccessKeyId=xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
            - -Dice.s3SecretKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
            - run-app

The error I met is

2019-10-15 18:17:32,210 [localhost-startStop-1] INFO  BootStrap  - Starting ice...
| Error com.amazonaws.services.s3.model.AmazonS3Exception: The authorization header is malformed; the authorization component "Credential=xxxxxxxxxxxxxx/20191015/us-east-1/s3/aws4_request" is malformed. (Service: Amazon S3; Status Code: 400; Error Code: AuthorizationHeaderMalformed; Request ID: 194A470802DB493B), S3 Extended Request ID: U0DPCCZy6ZswY91v4lav0jjMX7SEtAn6S+OHsCTMIU3Z7BGo9Q1mRNJF/eUk1NyPlm91vKRH0Ak=

It seems that ice is still connect to the default s3 region "us-east-1", where did I misconfig? Please help me to figure it out, thanks very much!

[nfonrose]

Hello,

Changing the region for the bucket is not enough for AWS China. You need to change the endpoints used to connect to the AWS China endpoints (they end with amazonaws.com.cn instead of amazonaws.com). You can find their list here: https://docs.amazonaws.cn/en_us/general/latest/gr/rande.html

This involves changing code as the endpoints are not configurable via the configuration file.

Nicolas

-- Nicolas Fonrose | Teevity | Founder +33.6.61.35.43.31 https://teevity.com - Cloud Costs Analytics for AWS, GCP and Azure powered by NetflixOSS Ice twitter - @nfonrose / @teevity

On Tue, Oct 15, 2019 at 12:43 PM zhangsimingshannonai < notifications@github.com> wrote:

My S3 bucket is in AWS-CHINA, region is "cn-northwest-1", but I didn't find anything about this in the src, Is ice support china-region?

I'm started ice in kubernetes way. Here is my configmap-ice.yaml

---

apiVersion: v1 kind: ConfigMap metadata: name: ice data: ice.properties: | ice.processor=true ice.reader=true ice.reservationCapacityPoller=false ice.reservationPeriod=oneyear ice.reservationUtilization=HEAVY ice.highstockUrl=https://code.highcharts.com/stock/4.2.1/highstock.js ice.urlPrefix=aws.shannonai ice.fromEmail=xxxxxxxxxxxxxxxxxxxxxxxx ice.ondemandCostAlertThreshold=250 ice.s3SecretKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ice.billing_s3bucketname=xxxxxxxxxxxx ice.billing_s3bucketregion=cn-northwest-1 ice.billing_accessRoleName=xxxxxxxxxxxxxxxxx ice.startmillis=1364774400000 ice.companyName=shannonai ice.work_s3bucketname=xxxxxxxxxxxxxxxxxx ice.work_s3bucketprefix=ice/ ice.work_s3bucketregion=cn-northwest-1 ice.processor.localDir=/mnt/ice_processor ice.reader.localDir=/mnt/ice_reader ice.monthlycachesize=12

Here is my deployment.yaml's env part

• name: ice image: harbor.shannonai.com/test/ice:latest ports:
  • containerPort: 8080 imagePullPolicy: IfNotPresent env:
  • name: GRAILS_OPTS value: "-server -Xmx2G -Xms1G -Dfile.encoding=UTF-8"
  • name: EC2_REGION value: cn-northwest-1
  • name: AWS_DEFAULT_REGION value: cn-northwest-1 command:
  • /opt/ice/grailsw
  • -Djava.net.preferIPv4Stack=true
  • -Djava.net.preferIPv4Addresses
  • -Duser.timezone=Asia/Shanghai
  • -Dice.s3AccessKeyId=xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  • -Dice.s3SecretKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  • run-app

The error I met is

2019-10-15 18:17:32,210 [localhost-startStop-1] INFO BootStrap - Starting ice...| Error com.amazonaws.services.s3.model.AmazonS3Exception: The authorization header is malformed; the authorization component "Credential=xxxxxxxxxxxxxx/20191015/us-east-1/s3/aws4_request" is malformed. (Service: Amazon S3; Status Code: 400; Error Code: AuthorizationHeaderMalformed; Request ID: 194A470802DB493B), S3 Extended Request ID: U0DPCCZy6ZswY91v4lav0jjMX7SEtAn6S+OHsCTMIU3Z7BGo9Q1mRNJF/eUk1NyPlm91vKRH0Ak=

It seems that ice is still connect to the default s3 region "us-east-1", where did I misconfig? Please help me to figure it out, thanks very much!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Teevity/ice/issues/294?email_source=notifications&email_token=AAAJJJMK24IFKLKEMXAFPPTQOWNDTA5CNFSM4JA2UKOKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HR2VLTQ, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAJJJMSFC65VOA6LYRV6TDQOWNDTANCNFSM4JA2UKOA .

[zhangsimingshannonai]

@nfonrose

Thanks for your reply, but I still not know where should I change, what is you mean "code"? I need to rebuild a dockerimage for china? Please tell me what file should I modify? Thanks

[zhangsimingshannonai]

You said "Changing the region for the bucket is not enough for AWS China", but I even not change region successfully. my error is

2019-10-15 18:17:32,210 [localhost-startStop-1] INFO  BootStrap  - Starting ice...
| Error com.amazonaws.services.s3.model.AmazonS3Exception: The authorization header is malformed; the authorization component "Credential=xxxxxxxxxxxxxx/20191015/us-east-1/s3/aws4_request" is malformed. (Service: Amazon S3; Status Code: 400; Error Code: AuthorizationHeaderMalformed; Request ID: 194A470802DB493B), S3 Extended Request ID: U0DPCCZy6ZswY91v4lav0jjMX7SEtAn6S+OHsCTMIU3Z7BGo9Q1mRNJF/eUk1NyPlm91vKRH0Ak=

It is still connect to "us-east-1", but I set "ice.billing_s3bucketregion=cn-north-1" in ice.properties. I don't know why, please help me

[zhangsimingshannonai]

Finally change AwsUtils.java file's endpoint, nothing happend.

$ ls
AwsUtils.java  Dockerfile

$ ack AwsUtils.java
Dockerfile
6:COPY AwsUtils.java src/java/com/netflix/ice/common/AwsUtils.java

$ ack Endpoint
AwsUtils.java
100:                s3Client.setEndpoint("s3.amazonaws.com");
103:                s3Client.setEndpoint("s3-" + System.getProperty("EC2_REGION") + ".amazonaws.com.cn");
123:                    simpleDBClient.setEndpoint("sdb.amazonaws.com");
126:                    simpleDBClient.setEndpoint("sdb." + System.getProperty("EC2_REGION") + ".amazonaws.com.cn");
276:                s3Client.setEndpoint("s3-" + bucketFileRegion + ".amazonaws.com.cn");

Could anyone answer my question? I would be greatful appreciate.