search

apache / pinot

Apache Pinot - A realtime distributed OLAP datastore
https://pinot.apache.org/
Apache License 2.0
5.42k stars 1.27k forks source link

Caused by: software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: DFM25B81QM9RZ2WX, Extended Request ID: 7vSTw22PgYLApWtVYc4blcMDFPRq1NOUG/5tKNqvH/YWCQwyhEO87rv6OOL0AFRf8Ia/+M4voOtLHW656U10Ng==) #11446

Open talhakhan119 opened 1 year ago

talhakhan119 commented 1 year ago

I am unable to start my pinot controller pod in AWS EKS environment.

For that I have added configurations in values.template.yaml for pinot to connect it from aws s3

As I have confirmed my serviceaccount have correct permissions and policies to access AWS S3

As I have tested it while exec into another pod using the same service account:

in the pinot pod
apt-get update
apt-get install awscli
# aws s3 ls s3://perceptdata-pinot
              PRE controller-data/
              PRE pinot-data/

Configurations:

  # Extra configs will be appended to pinot-controller.conf file
  extra:
    configs: |-
      pinot.set.instance.id.to.hostname=true
      controller.task.scheduler.enabled=true
      controller.disable.ingestion.groovy=false
      pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
      pinot.controller.storage.factory.s3.region=eu-west-1
      pinot.controller.segment.fetcher.protocols=file,http,s3
      pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher

The data directory for aws s3 bucket is: 

data:
    dir: s3://perceptdata-pinot/controller-data

The following Error logs I am facing:

Initializing ControllerFilePathProvider
Data directory: s3://perceptdata-pinot/controller-data/
Failed to start a Pinot [CONTROLLER] at 17.271 since launch
java.lang.RuntimeException: Caught exception while initializing ControllerFilePathProvider
        at org.apache.pinot.controller.BaseControllerStarter.initControllerFilePathProvider(BaseControllerStarter.java:569) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.controller.BaseControllerStarter.setUpPinotController(BaseControllerStarter.java:380) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.controller.BaseControllerStarter.start(BaseControllerStarter.java:328) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.service.PinotServiceManager.startController(PinotServiceManager.java:118) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:87) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.lambda$startBootstrapServices$0(StartServiceManagerCommand.java:251) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:304) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startBootstrapServices(StartServiceManagerCommand.java:250) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.execute(StartServiceManagerCommand.java:196) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.admin.command.StartControllerCommand.execute(StartControllerCommand.java:187) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.Command.call(Command.java:33) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.Command.call(Command.java:29) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at picocli.CommandLine.executeUserObject(CommandLine.java:1953) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at picocli.CommandLine.access$1300(CommandLine.java:145) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2346) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2311) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at picocli.CommandLine.execute(CommandLine.java:2078) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:171) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:202) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
Caused by: org.apache.pinot.controller.api.resources.InvalidControllerConfigException: Caught exception while initializing file upload path provider
        at org.apache.pinot.controller.api.resources.ControllerFilePathProvider.<init>(ControllerFilePathProvider.java:107) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.controller.api.resources.ControllerFilePathProvider.init(ControllerFilePathProvider.java:49) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.controller.BaseControllerStarter.initControllerFilePathProvider(BaseControllerStarter.java:567) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        ... 20 more
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: TMMVB33627KABQ7N, Extended Request ID: VOWG6S7PM21PAQ6FNeGPIMFeMFiz/CnWf+MoJ7pyGTktnYuOEV7mzjNGnXe8DXx4ZP1jCDqXnco=)
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:106) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:84) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:42) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:94) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$5(BaseClientHandler.java:229) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:128) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:154) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:107) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:162) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:91) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.services.s3.DefaultS3Client.listObjectsV2(DefaultS3Client.java:5614) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.plugin.filesystem.S3PinotFS.isDirectory(S3PinotFS.java:572) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.plugin.filesystem.S3PinotFS.exists(S3PinotFS.java:437) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.spi.filesystem.NoClosePinotFS.exists(NoClosePinotFS.java:74) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.controller.api.resources.ControllerFilePathProvider.<init>(ControllerFilePathProvider.java:71) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.controller.api.resources.ControllerFilePathProvider.init(ControllerFilePathProvider.java:49) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.controller.BaseControllerStarter.initControllerFilePathProvider(BaseControllerStarter.java:567) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        ... 20 more
Shutting down Pinot Service Manager with all running Pinot instances...
Shutting down Pinot Service Manager admin application...
Deregistering service status handler

As I am using a service account to access aws in my pinot-controller pod as the specs of service account are as:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::333870179331:role/pinot
    meta.helm.sh/release-name: pinot
    meta.helm.sh/release-namespace: pinot-new
  creationTimestamp: "2023-08-25T06:54:37Z"
  labels:
    app: pinot
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/version: 0.2.7
    helm.sh/chart: pinot-0.2.7
    heritage: Helm
    release: pinot
  name: pinot
  namespace: pinot-new
  resourceVersion: "389804078"
  uid: 488a7bf2-15fe-4aac-a3ad-e82c5515b3c9

This is the policy for pinot :

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PinotAccessPolicy",
      "Effect": "Allow",
      "Action": [
        "*"
      ],
      "Resource": [
        "arn:aws:s3:::perceptdata-pinot/*",
        "arn:aws:s3:::perceptdata-pinot"
      ]
    }
  ]
}
Jackie-Jiang commented 1 year ago

Can you post this to the troubleshooting slack channel so that people running into similar problems might be able to help?

talhakhan119 commented 1 year ago

Can you post this to the troubleshooting slack channel so that people running into similar problems might be able to help?

Sure, Can you please provide the relevant channel link?

Jackie-Jiang commented 1 year ago

https://communityinviter.com/apps/apache-pinot/apache-pinot

wahab-io commented 5 months ago

Running into the similar issue. Is controller supposed to use serviceaccount when making calls to S3?

swaminathanmanish commented 5 months ago

@wahab-io - Could you try the workaround suggested here ? https://apache-pinot.slack.com/archives/C011C9JHN7R/p1694409197833749

wahab-io commented 5 months ago

@swaminathanmanish the above link is not working, can you please share the workaround here?

swaminathanmanish commented 5 months ago

@swaminathanmanish the above link is not working, can you please share the workaround here?

@xiangfu0 had suggested the following in that slack thread to the problem described in this issue -

<<Can you check IAM Role Trust Relationship: Make sure that your EKS nodes (or the EKS service itself) can assume the IAM role arn:aws:iam::333870179331:role/pinot. If the trust relationship is not configured correctly, then your nodes won’t be able to assume this role.

Also check the Annotations: You’ve annotated your service account with eks.amazonaws.com/role-arn. Make sure that this service account is actually being used by your pod. You can check this by describing the pod or statefulset (kubectl describe pod -n pinot-new) and looking at the Service Account field >>