search

ing-bank / rokku

Rokku project. This project acts as a proxy on top of any S3 storage solution providing services like authentication, authorization, short-term tokens, and lineage.
Apache License 2.0
66 stars 17 forks source link

Issues with docker-compose #131

Closed DurgaChill closed 4 years ago

DurgaChill commented 4 years ago

Hi All,

After running docker-compose up, I am not able to see ranger up and running at port 6080, whereas I can see atlas up and running at 21000.

But I can see the ranger running at port 6080 as part of docker ps result.

Any thoughts on why could this happen ?? Please let me know if I need to provide anything.

Thanks in advance.

arempter commented 4 years ago

Hey, Ranger is applying quite few sql updates, first time it starts. Depending on your HW, it may need some more time to come up. Check container logs, if it is still applying them, just leave it running. It should eventually come up. Worst case, restart Ranger container once more. Just do not remove container with postgres.

DurgaChill commented 4 years ago

Hi,

Thanks for the info. Now I can see ranger up and running, but for some reason I can see the ceph image exits in 2 minutes in docker ps status, whereas other services are up and running continuously.

Find the attached result below: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6db06055fd07 wbaa/rokku-dev-apache-atlas:0.0.8 "sh -c /entrypoint.sh" 39 hours ago Up 8 hours 0.0.0.0:21000->21000/tcp rokku_atlas_1 aabd5b5545f4 confluentinc/cp-kafka:5.4.0 "/etc/confluent/dock…" 39 hours ago Up 8 hours 0.0.0.0:9092->9092/tcp kafka b46a05f41825 wbaa/rokku-dev-apache-ranger:0.0.21 "sh -c /tmp/entrypoi…" 39 hours ago Up 8 hours 0.0.0.0:6080->6080/tcp rokku_ranger-admin_1 a021ee7941c7 wbaa/rokku-sts:0.3.4 "/opt/docker/bin/rok…" 39 hours ago Up 8 hours 0.0.0.0:12345->12345/tcp rokku_rokku-sts_1 efd14918eb0d hazelcast/management-center "bash /mc-start.sh" 39 hours ago Up 8 hours 8081/tcp, 8443/tcp, 0.0.0.0:8081->8080/tcp rokku_cache-management_1 a69a5fb22bde wbaa/rokku-dev-mariadb:0.0.8 "docker-entrypoint.s…" 39 hours ago Up 8 hours 0.0.0.0:3307->3306/tcp rokku_mariadb_1 5711021516ce confluentinc/cp-zookeeper:5.4.0 "/etc/confluent/dock…" 39 hours ago Up 8 hours 2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp zookeeper 1a7494d8b3c9 wbaa/rokku-dev-keycloak:0.0.8 "/opt/jboss/tools/do…" 39 hours ago Up 8 hours 0.0.0.0:8080->8080/tcp, 8443/tcp rokku_keycloak_1 bc1fd486abcc ceph/daemon:v3.0.5-stable-3.0-luminous-centos-7 "/entrypoint.sh demo" 39 hours ago Exited (1) 2 minutes ago rokku_ceph_1 067fe2973f64 wbaa/rokku-dev-apache-ranger-postgres:0.0.21 "docker-entrypoint.s…" 39 hours ago Up 8 hours 5432/tcp rokku_postgres-server_1 cd1eabb67d64 cassandra:3 "docker-entrypoint.s…" 39 hours ago Up 8 hours 7000-7001/tcp, 0.0.0.0:9042->9042/tcp, 7199/tcp, 0.0.0.0:9160->9160/tcp cassandra

Note: I have started the proxy also and I can see --> ....... [info] {"timestamp":"2020-04-22T21:54:56.485Z","level":"INFO","thread":"rokku-akka.actor.default-dispatcher-3","logger":"com.ing.wbaa.rokku.proxy.Server$$anon$1","message":"Proxy service started listening: /127.0.0.1:8987","context":"default","application_name":"rokku-proxy"}

Any help would be much appreciated. Thanks in advance.

arempter commented 4 years ago

if you do docker ps -a do you see any previous ceph instances? If some please remove them all, before starting new ceph ... If this not helps check logs for ceph containter which exited.

DurgaChill commented 4 years ago

Hi, Thanks I had a one old ceph instance. When I re ran docker compose after removing the old ceph instance, it worked for me :)

And when proceeding further, I can see the below errors:

(rokku) [DURGACHILLAKURU@localhost rokku]$ aws s3api list-objects --bucket demobucket

An error occurred (NoSuchBucket) when calling the ListObjects operation: Unknown

I tried to create a bucket , assuming demo bucket wasn't there. Then I got this unauthorized error.

(rokku) [DURGACHILLAKURU@localhost rokku]$ aws s3api create-bucket --bucket my-bucket --region us-east-1

An error occurred (Unauthorized) when calling the CreateBucket operation: Unauthorized

Then, I used list-buckets to confirm.

(rokku) [DURGACHILLAKURU@localhost rokku]$ (rokku) [DURGACHILLAKURU@localhost rokku]$ aws s3api list-buckets --query "Buckets[].Name" [ "demobucket" ] (rokku) [DURGACHILLAKURU@localhost rokku]$ aws s3api list-objects --bucket demobucket

An error occurred (NoSuchBucket) when calling the ListObjects operation: Unknown

Can you please let me know what has to be done. Thanks in advance.

arempter commented 4 years ago

ok, that could be related to permission. Just make sure you do run setupS3Env.sh. If not ok, restart rokku afterwards...

arempter commented 4 years ago

Also one more tip. Add following to hosts file: 127.0.0.1 localhost s3.localhost

it simulates AWS virtual bucket access for local env

DurgaChill commented 4 years ago

Hi, I have added the above entry in localhost.

I am not able to run setupS3Env.sh because of permission issues. Any hints on how could I resolve those ? Thanks in advance.

DurgaChill commented 4 years ago

And also, I have added all config params too -

Got Keycloak token and then have set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN. Then I am trying to run the setupS3Env.sh, I am getting permission denied errors.

arempter commented 4 years ago

Above ENV vars are not needed to run setupS3Env... Script itself only connects to ceph docker and sets ACL and bucket polices ... there should be no high privileges required to run it.

What error are you getting when running script?

arempter commented 4 years ago

btw. make sure that you add --endpoint-url (for aws s3api) and --endpoint (aws s3) with local rokku proxy eg.

aws s3api list-objects --bucket demobucket --endpoint-url http://localhost:8987

DurgaChill commented 4 years ago

Hi, I am not able to set ACL. I am suspecting the validity of the token. Steps: Once I generate the token(keycloak) and get the session details, I would replace them in the credentails file under /.aws directory --> [rokku] aws_access_key_id = YOUR_ACCESS_KEY_ID aws_secret_access_key= YOUR_SECRET_ACCESS_KEY aws_session_token =YOUR_SESSION_TOKEN.

In general what is the default expiry for these credentials?

Then I ran the setup script (previously it wasn't an executable file, that was the problem)

Image : image

Manually I created the buckets that are mentioned in the script setupS3Env.sh.

[durgachillakuru@localhost rokku]$ workon rokku (rokku) [durgachillakuru@localhost rokku]$ aws s3api list-objects --bucket demobucket

An error occurred (NoSuchBucket) when calling the ListObjects operation: Unknown (rokku) [durgachillakuru@localhost rokku]$ aws s3 ls 2017-12-31 19:00:00 rokku_hc_bucket 2017-12-31 19:00:00 home 2017-12-31 19:00:00 demobucket 2017-12-31 19:00:00 shared

But when I am trying to run the below command,

(rokku) [durgachillakuru@localhost rokku]$ aws s3api list-objects --bucket demobucket --endpoint-url http://localhost:8987

An error occurred (NoSuchBucket) when calling the ListObjects operation: Unknown (rokku) [durgachillakuru@localhost rokku]$ aws --profile rokku s3api list-objects --bucket demobucket --endpoint-url http://localhost:8987

An error occurred (NoSuchBucket) when calling the ListObjects operation: Unknown

Can you please let me know what I am missing over here ? I did try to export aws credentials in terminal and also, I tried to change the params in setup script for aws credentials in line 13 and 14 in the script. When I do so, I can see..

upload: '/etc/issue' -> 's3://demobucket/subdir/issue' [1 of 1] 23 of 23 100% in 0s 592.57 B/s done Bucket 's3://home/' created Bucket 's3://rokku_hc_bucket/' created upload: '/etc/issue' -> 's3://home/testuser/issue' [1 of 1] 23 of 23 100% in 0s 658.37 B/s done upload: '/etc/issue' -> 's3://home/testuser1/issue' [1 of 1] 23 of 23 100% in 0s 721.64 B/s done upload: '/etc/issue' -> 's3://home/userone/issue' [1 of 1] 23 of 23 100% in 0s 754.25 B/s done Bucket 's3://shared/' created

An error occurred (AccessDenied) when calling the PutBucketAcl operation: Unknown

An error occurred (AccessDenied) when calling the PutBucketAcl operation: Unknown

An error occurred (AccessDenied) when calling the PutBucketAcl operation: Unknown

An error occurred (AccessDenied) when calling the PutBucketPolicy operation: Unknown

An error occurred (AccessDenied) when calling the PutBucketPolicy operation: Unknown

An error occurred (AccessDenied) when calling the PutBucketPolicy operation: Unknown

Kindly let me know what can be done here. Thanks in advance.

arempter commented 4 years ago

ok, so lets start simple first. Make sure you run setup script in new terminal window, and AWS credentials are not set via /.aws.

you do not need them, script directly talks to ceph container. So run this step first.

And once you have no errors after running it, lets proceed with rest.

BTW. In general what is the default expiry for these credentials?

once you have run setupEnv, to get credentials use:

$ source scripts/dev_sts_get_credentials.sh - this will setup credentials correctly

DurgaChill commented 4 years ago

Hi, (Have removed my previous credentials in the /.aws) 1.Have ran the setup script in a new terminal window. - No errors

(rokku) [durgachillakuru@localhost rokku]$ ./setupS3Env.sh { "user_id": "ceph-admin", "display_name": "Ceph demo user", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [], "keys": [ { "user": "ceph-admin", "access_key": "accesskey", "secret_key": "secretkey" } ], "swift_keys": [], "caps": [ { "type": "buckets", "perm": "" }, { "type": "metadata", "perm": "" }, { "type": "usage", "perm": "" }, { "type": "users", "perm": "" } ], "op_mask": "read, write, delete", "system": "true", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw" }

upload: '/etc/issue' -> 's3://demobucket/subdir/issue' [1 of 1] 23 of 23 100% in 0s 624.00 B/s done Bucket 's3://home/' created Bucket 's3://rokku_hc_bucket/' created upload: '/etc/issue' -> 's3://home/testuser/issue' [1 of 1] 23 of 23 100% in 0s 400.01 B/s done upload: '/etc/issue' -> 's3://home/testuser1/issue' [1 of 1] 23 of 23 100% in 0s 473.20 B/s done upload: '/etc/issue' -> 's3://home/userone/issue' [1 of 1] 23 of 23 100% in 0s 769.93 B/s done Bucket 's3://shared/' created (rokku) [durgachillakuru@localhost rokku]$ cd scripts/

2.Then ran source scripts/dev_sts_get_credentials.sh

Read keycloak token: xxx Aws Json recieved: { "Credentials": { "AccessKeyId": "xxxx", "SecretAccessKey": "xxxx", "SessionToken": "xxxx", "Expiration": "2020-04-27T21:48:59.885Z" } }

After this, have even checked echo $AWS_ACCESS_KEY_ID --> correct value has been printed, the one I got when I ran the dev_sts_get_credentials.sh script.

But still the issue stays the same,

(rokku) [durgachillakuru@localhost rokku]$ aws s3api list-objects --bucket demobucket

An error occurred (NoSuchBucket) when calling the ListObjects operation: Unknown (rokku) [durgachillakuru@localhost rokku]$ aws s3 ls 2017-12-31 19:00:00 rokku_hc_bucket 2017-12-31 19:00:00 home 2017-12-31 19:00:00 demobucket 2017-12-31 19:00:00 shared (rokku) [durgachillakuru@localhost rokku]$ aws s3api list-objects --bucket demobucket --endpoint-url http://localhost:8987

An error occurred (NoSuchBucket) when calling the ListObjects operation:

Do you want me to restart the proxy or can you please let me know what else I am missing over here.? Thanks in advance.

DurgaChill commented 4 years ago

I even tried by restarting the proxy and repeated the steps, but no luck.

If I give aws s3 ls -> I can see all buckets. But when I list the contents, no such bucket exception is been thrown.

(rokku) [durgachillakuru@localhost rokku]$ aws s3 ls s3://demobucket

An error occurred (NoSuchBucket) when calling the ListObjectsV2 operation: Unknown (rokku) [durgachillakuru@localhost rokku]$ aws s3 ls s3://home

An error occurred (NoSuchBucket) when calling the ListObjectsV2 operation: Unknown (rokku) [durgachillakuru@localhost rokku]$ aws s3 ls s3://rokku_hc_bucket

An error occurred (Unauthorized) when calling the ListObjectsV2 operation: Unauthorized (rokku) [durgachillakuru@localhost rokku]$ aws s3 ls s3://demobucket

An error occurred (NoSuchBucket) when calling the ListObjectsV2 operation: Unknown

(rokku) [durgachillakuru@localhost rokku]$ aws s3api list-objects --bucket demobucket --endpoint-url http://localhost:8987

An error occurred (NoSuchBucket) when calling the ListObjects operation: Unknown

Do you think any ranger policy is being applied ? Do you want me to investigate from that perspective ?

Can you please let me know what I could be missing over here ? Thanks in advance.

DurgaChill commented 4 years ago

I suspected Ranger policies to be in action, so I performed the step that you have mentioned.

Well, your policy in Ranger only allows you to read objects from the demobucket. So we'll need to allow a write as well.

Go to Ranger on http://localhost:6080 and login with admin:admin.
Go to the testservice under the S3 header.
Edit the one existing policy. You'll have to allow the testuser to write, but also don't forget to remove the deny condition!
Save the policy at the bottom of the page.

Still no luck.

(rokku) [durgachillakuru@localhost rokku]$ aws s3api list-objects --bucket demobucket --endpoint-url http://localhost:8987

An error occurred (NoSuchBucket) when calling the ListObjects operation: Unknown (rokku) [durgachillakuru@localhost rokku]$

A quick question, how do I check if my credentials are still active ?

arempter commented 4 years ago

ok, since you mentioned ranger, one thing to confirm. Do you see any errors when starting proxy? for instance this one: https://github.com/ing-bank/rokku/issues/74

Ranger plugin needs config file to be able to talk to ranger admin server

arempter commented 4 years ago

Also please have a look at logs in rokku-sts container for possible errors with authentication. If there is problem with token it will be reported in sts console

DurgaChill commented 4 years ago

ok, since you mentioned ranger, one thing to confirm. Do you see any errors when starting proxy? for instance this one:

74

Ranger plugin needs config file to be able to talk to ranger admin server

No I didn't see any errors like this. And I have placed the config file too. I can login to ranger and can even edit the policy.

DurgaChill commented 4 years ago

Also please have a look at logs in rokku-sts container for possible errors with authentication. If there is problem with token it will be reported in sts console

I did check in sts logs before, I can even see the authorization is success -

[info] {"timestamp":"2020-04-28T03:10:00.329Z","level":"INFO","thread":"rokku-akka.actor.default-dispatcher-228","mdc":{"request.id":"a0c11b48-d8c3-4ced-9811-043f4294a525"},"logger":"com.ing.wbaa.rokku.proxy.handler.LoggerHandlerWithId","message":"User (UserName(testuser)) successfully authorized for request: S3Request(AwsRequestCredential(AwsAccessKey(NgPS4OlnAblRKKUpI12jGs3kaNRid42w),Some(AwsSessionToken([jmYuH...]))),Some(/demobucket),None,Read(GET),127.0.0.1:58010,HeaderIPs(None,None,None),none/none)","context":"default","application_name":"rokku-proxy"}

I will try to stop all this once and maybe give a try.

Steps: 1.run docker-compose up (after removing all previous instances)

  1. Will place the ranger-s3-security.xml file 3.Create a virtualenv -and then will create aws folder, and then run sbt run in virtualenv.
  2. in virtualenv- Run setupS3Env.sh and dev_sts_get_credentials.sh
  3. Run this command - docker-compose exec ceph radosgw-admin user modify --uid ceph-admin --system
  4. Then check for aws s3api list-objects --bucket demobucket aws s3api put-object --bucket demobucket --key SOME_FILE commands.

Kindly Let me know if I am missing anything ?

arempter commented 4 years ago

ok, please do, point 5, is not really needed.

Please capture proxy log and add it as attachment, so we will see what proxy is saying during list operation.

DurgaChill commented 4 years ago

Hey, I am getting the below error during sbt run,

[info] {"timestamp":"2020-04-28T06:18:13.606Z","level":"ERROR","thread":"main","logger":"org.apache.ranger.authorization.hadoop.config.RangerConfiguration","message":"addResourceIfReadable(ranger-s3-audit.xml): couldn't find resource file location","context":"default","application_name":"rokku-proxy"}

What could be the reason for this ? I wasn't sure before if it had popped up. Apologies.

Thanks in advance.

arempter commented 4 years ago

Hey, no this one can be ignored. thats for audit....

DurgaChill commented 4 years ago

Hi, No luck still the same .

Attaching the complete logs for your reference. Kindly let me know what else I can provide with.

rokku_script logs.txt rokku_proxyLogs.txt rokku_proxyCurrentLogsLatest.txt

docker_compose_rangerLogs.txt docker_compose_atlasLogs.txt ceph_docker_logs.txt

Images in Ranger: image

View policy for testpolicy: image image

Any help is much appreciated.

DurgaChill commented 4 years ago

I have noticed one thing, when I ran sbt run for the very first time. I can see it took a lot of time to start, now it's just hardly a minute. Just a hunch, might strike something. Should I do anything apart from sbt run everytime ?

Thanks in advance.

arempter commented 4 years ago

All the setup seems fine from what you have sent.

Could you please try this one also (not s3api): aws s3 --endpoint=http://s3.localhost:8987 ls s3://home/testuser/ aws s3 --endpoint=http://localhost:8987 ls s3://home/testuser/

arempter commented 4 years ago

yep. confirmed. I was able to reproduce. It seems to be a problem with s3api. s3 works (above commands).

I also get aws s3api list-objects --bucket home --endpoint-url=http://localhost:8987

An error occurred (NoSuchBucket) when calling the ListObjects operation: Unknown

Could you please now use s3 to list objects?

arempter commented 4 years ago

you could also please try: aws s3api --endpoint-url=http://s3.localhost:8987 list-objects --bucket home

that one works for me

DurgaChill commented 4 years ago

Hey thanks a lot, it worked like a charm.

arempter commented 4 years ago

Good to hear! Thanks ;)

DurgaChill commented 4 years ago

just a quick question, will this setup work for some other ceph instance running on docker ?

arempter commented 4 years ago

yes, it should...

DurgaChill commented 4 years ago

That's good to hear. Can you just please let me know to specify the endpoint for the same? And also, what would be the config changes regarding the same. Thanks in advance

DurgaChill commented 4 years ago

I am trying NOT to deploy a ceph instance. I just need to hit a ceph instance already running on different server. Can you please let me know if that's something that can be done ?

As I can see a method in getAmazonS3 in S3SdkHelpers.scala

def getAmazonS3(authority: Authority, credentials: AWSCredentials = new BasicSessionCredentials("accesskey", "secretkey", "token") ): AmazonS3 = { val cliConf = new ClientConfiguration() cliConf.setMaxErrorRetry(1)

AmazonS3ClientBuilder
  .standard()
  .withClientConfiguration(cliConf)
  .withCredentials(new AWSStaticCredentialsProvider(credentials))
  .withPathStyleAccessEnabled(true)
  **.withEndpointConfiguration(new EndpointConfiguration(s"http://s3.localhost:${authority.port}", awsRegion))**
  .build()

}

where Endpoint configuration is mentioned as localhost. I might be referring to files that are irrelevant. Can you just throw some insights from where can I look at those config setup, so that I can connect to the ceph instance running on different server (NOT LOCALHOST) ? Thanks in advance.

arempter commented 4 years ago

please have a look at appliction.conf or reference.conf in the resources directory. All settings are there.

To change S3 settings there is the whole section:

storage.s3 {

Settings for reaching backing storage.

    host = ${?ROKKU_STORAGE_S3_HOST}
    port = ${?ROKKU_STORAGE_S3_PORT}
DurgaChill commented 4 years ago

Hey, thanks for the above info. Just a quick question on ceph access.

With the current rokku setup !

  1. Do we need to have admin access to the ceph instance ?
  2. Are there any default health checks to see if the ceph instance is up and running - by creation/deletion of buckets.

Thanks in advance.

arempter commented 4 years ago

p1. yes you need admin user, which is setup by admin { accesskey = ${?ROKKU_STORAGE_S3_ADMIN_ACCESSKEY} secretkey = ${?ROKKU_STORAGE_S3_ADMIN_SECRETKEY} }

p2. rokku has /ping endpoint. which you can query to see if rokku/rgw is alive

DurgaChill commented 4 years ago

Thanks for confirming.

Now, I am trying to interact entire rokku setup with ceph nano(https://github.com/ceph/cn) instead of ceph(rokku's).

Steps:

  1. Started Ceph cluster - one of the step in ceph nano setup --> which gives me the access key and secret key - enables me to create and delete buckets.
  2. In rokku project setup -->

Have removed the ceph component in docker compose.yaml file. Have set the aws credentials with the access key and secret key generated above in step1. Have changed the ports also in resources.conf

storage.s3 {
    # Settings for reaching backing storage.
    host = "127.0.0.1"
    port = 8001

    admin {
        accesskey = "xxxx"
        secretkey = "xxxx" /// generated in ceph nano
    }
    region = "us-east-1"
    v2SignatureEnabled = false
    enabledCache = false
    eligibleCachePaths = "/home/,/test/"
    maxEligibleCacheObjectSizeInBytes = 5242880
    cacheDStructName = "S3Cache"

    healthCheck {
       # can be one of:
       #   rgwListBuckets - uses RGW admin to list all buckets in Ceph
       #   s3ListBucket  - uses AWS S3 client to list single bucket
       method = "s3ListBucket"
       interval = 5000
       bucketName = "bucket1"
    }

Note: I did not receive any such session id in ceph nano - while creating a cluster. All i got is Endpoint: http://172.20.20.20:8001 Dashboard: http://172.20.20.20:5001 Access key: xxxx Secret key: xxxx Working directory: /~/cephnano/cluster

Anyways, I am not able to interact with this ceph nano cluster. Any idea or pointers that could help me at this point ? I haven't checked any ranger settings to be configured. Any help is appreciated.

Also, I know this is irrelevant in this thread. Do let me know, if I can open a new one for this scenario, since the original scenario has been solved.

Thanks in advance.

arempter commented 4 years ago

so I do not know how ceph nano is build / works, to be honest. We only test and run rokku for now with regular ceph. Feel free to play with it but, I am not sure if this will work...