search

apache / pinot

Apache Pinot - A realtime distributed OLAP datastore
https://pinot.apache.org/
Apache License 2.0
5.26k stars 1.23k forks source link

Servers can't consume messages due to permission denied error in the endpoint '/segmentConsumed' when ACL is enabled #11407

Open rodolphogarrido opened 10 months ago

rodolphogarrido commented 10 months ago

While using the ACL feature, Pinot Servers aren't able to consume messages from Kafka due to a permission denied error in the endpoint /segmentConsumed (more details in the log).

Ps: The Servers are able to consume a few events before the error starts, but after that no more events are consumed.

Cluster version

Apache Pinot version: 0.12.1

Cluster ACL configs:

Controller ACL conf: 


# ACLs
# Enable ACL enforcement in the controller
controller.admin.access.control.factory.class=org.apache.pinot.controller.api.access.BasicAuthAccessControlFactory

# Create users "admin" and "user". Keep in mind we're not enforcing any ACLs yet.
controller.admin.access.control.principals=admin,user
# Set the admin's password to "verysecret"
controller.admin.access.control.principals.admin.password=verysecret
# Set the user's password to "secret" and allow "READ" only
controller.admin.access.control.principals.user.password=secret
controller.admin.access.control.principals.user.permissions=READ
controller.admin.access.control.principals.user.tables=events,events_upsert_full,events_upsert_partial
# Enable the controller to fetch segments by providing the credentials as a token "Basic " + base64encode("admin:verysecret")
controller.segment.fetcher.auth.token="Basic YWRtaW46dmVyeXNlY3JldA=="

Server ACL conf:

# ACLs
# Enable the Server to fetch/upload segments by providing the credentials as a token "Basic " + base64encode("admin:verysecret")
pinot.server.segment.fetcher.auth.token="Basic YWRtaW46dmVyeXNlY3JldA=="
pinot.server.segment.uploader.auth.token="Basic YWRtaW46dmVyeXNlY3JldA=="
pinot.server.instance.auth.token="Basic YWRtaW46dmVyeXNlY3JldA=="

Broker ACL conf:

# Enable ACL enforcement in the Broker
# The factory class property is different for the broker
pinot.broker.access.control.class=org.apache.pinot.broker.broker.BasicAuthAccessControlFactory

# Create the users and password (must be exactly the same as the ones created for the controller)
pinot.broker.access.control.principals=admin,user
pinot.broker.access.control.principals.admin.password=verysecret
pinot.broker.access.control.principals.user.password=secret

# No need to set READ permissions here since broker requests are read-only

# This configuration option allow specification of usernames and passwords as well as optional ACL restrictions on a per-table table basis (access type is always READ)
pinot.broker.access.control.principals.user.tables=events,events_upsert_full,events_upsert_partial

Minion ACL conf:

# ACLs
segment.fetcher.auth.token="Basic YWRtaW46dmVyeXNlY3JldA=="
task.auth.token="Basic YWRtaW46dmVyeXNlY3JldA=="

Table config

{
    "tableName": "events_upsert_full",
    "tableType": "REALTIME",
    "segmentsConfig": {
      "timeColumnName": "ts",
      "schemaName": "events",
      "replication": "2",
      "replicasPerPartition": "3",
      "retentionTimeUnit": "HOURS",
      "retentionTimeValue": "2",
      "peerSegmentDownloadScheme": "http"
    },
    "tableIndexConfig": {
      "loadMode": "MMAP",
      "nullHandlingEnabled": true,
      "invertedIndexColumns": ["uuid", "name"],
      "streamConfigs": {
        "streamType": "kafka",
        "stream.kafka.topic.name": "events",
        "stream.kafka.broker.list": "broker0:9092,broker1:9092,broker2:9092",
        "stream.kafka.consumer.type": "lowlevel",
        "stream.kafka.consumer.prop.auto.offset.reset": "smallest",
        "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
        "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder",        
        "stream.kafka.decoder.prop.format": "AVRO",
        "stream.kafka.decoder.prop.schema.registry.rest.url": "http://schema-registry:8081",
        "realtime.segment.flush.threshold.rows": "5",
        "realtime.segment.flush.threshold.time": "2m",
        "realtime.segment.flush.threshold.segment.size": "5M"
      }
    },
    "ingestionConfig": {
        "batchIngestionConfig": {
          "segmentIngestionType": "APPEND",
          "segmentIngestionFrequency": "DAILY"
        },
        "transformConfigs": []
      },
    "tenants": {
        "broker": "DefaultTenant",
        "server": "one"
    },
    "metadata": {},
    "instanceAssignmentConfigMap": {
        "CONSUMING": {
          "tagPoolConfig": {
            "tag": "one_REALTIME",
            "poolBased": true,
            "numPools": 2
          },
          "replicaGroupPartitionConfig": {
            "replicaGroupBased": true,
            "numReplicaGroups": 2,
            "numInstancesPerReplicaGroup": 2
          }
        },
        "COMPLETED": {
            "tagPoolConfig": {
              "tag": "one_OFFLINE",
              "poolBased": true,
              "numPools": 2
            },
            "replicaGroupPartitionConfig": {
                "replicaGroupBased": true,
                "numReplicaGroups": 2,
                "numInstancesPerReplicaGroup": 2
              }
          }
      },
    "routing": {
        "instanceSelectorType": "strictReplicaGroup"
      },
      "upsertConfig": {
        "mode": "FULL",
        "hashFunction": "NONE",
        "deleteRecordColumn": "delete",
        "enableSnapshot": true

      },
      "task": {
        "taskTypeConfigsMap": {
          "MergeRollupTask": {
            "5m.mergeType": "concat",
            "5m.bucketTimePeriod": "5m",
            "5m.bufferTimePeriod": "1m"
          }
        }
      }
  }

Schema config

{
    "schemaName": "events",
    "primaryKeyColumns": [
        "name"
    ],
    "dimensionFieldSpecs": [
      {
        "name": "uuid",
        "dataType": "STRING"
      },
      {
        "name": "name",
        "dataType": "STRING",
        "defaultNullValue": ""
      },
      {
        "name": "delete",
        "dataType": "BOOLEAN",
        "defaultNullValue": "false"
      }
    ],
    "metricFieldSpecs": [
      {
        "name": "count",
        "dataType": "INT"
      }
    ],
    "dateTimeFieldSpecs": [{
      "name": "ts",
      "dataType": "TIMESTAMP",
      "format" : "1:MILLISECONDS:EPOCH",
      "granularity": "1:MILLISECONDS"
    }]
  }

Log error

2023-08-22 08:57:21 2023/08/22 11:57:21.716 ERROR [ServerSegmentCompletionProtocolHandler] [events_upsert_partial__1__0__20230822T1152Z] Could not send request http://pinot-controller-0:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=5&instance=Server_pinot-server-0_8098&offset=-1&name=events_upsert_partial__1__0__20230822T1152Z&rowCount=5&memoryUsedBytes=1368
2023-08-22 08:57:21 org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 403 (Forbidden) with reason: "Permission is denied for READ '/segmentConsumed'" while sending request: http://pinot-controller-0:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=5&instance=Server_pinot-server-0_8098&offset=-1&name=events_upsert_partial__1__0__20230822T1152Z&rowCount=5&memoryUsedBytes=1368 to controller: pinot-controller-0, version: Unknown
2023-08-22 08:57:21     at org.apache.pinot.common.utils.http.HttpClient.wrapAndThrowHttpException(HttpClient.java:442) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-08-22 08:57:21     at org.apache.pinot.common.utils.FileUploadDownloadClient.sendSegmentCompletionProtocolRequest(FileUploadDownloadClient.java:1039) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-08-22 08:57:21     at org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler.sendRequest(ServerSegmentCompletionProtocolHandler.java:217) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-08-22 08:57:21     at org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler.segmentConsumed(ServerSegmentCompletionProtocolHandler.java:184) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-08-22 08:57:21     at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.postSegmentConsumedMsg(LLRealtimeSegmentDataManager.java:1110) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-08-22 08:57:21     at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:650) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-08-22 08:57:21     at java.lang.Thread.run(Thread.java:829) [?:?]
2023-08-22 08:57:21 2023/08/22 11:57:21.717 ERROR [LLRealtimeSegmentDataManager_events_upsert_partial__1__0__20230822T1152Z] [events_upsert_partial__1__0__20230822T1152Z] Holding after response from Controller: {"offset":-1,"status":"NOT_SENT","streamPartitionMsgOffset":null,"isSplitCommitType":false,"buildTimeSec":-1}
2023-08-22 08:57:22 2023/08/22 11:57:22.482 ERROR [ServerSegmentCompletionProtocolHandler] [events_upsert_full__1__0__20230822T1152Z] Could not send request http://pinot-controller-0:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=5&instance=Server_pinot-server-0_8098&offset=-1&name=events_upsert_full__1__0__20230822T1152Z&rowCount=5&memoryUsedBytes=1368
2023-08-22 08:57:22 org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 403 (Forbidden) with reason: "Permission is denied for READ '/segmentConsumed'" while sending request: http://pinot-controller-0:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=5&instance=Server_pinot-server-0_8098&offset=-1&name=events_upsert_full__1__0__20230822T1152Z&rowCount=5&memoryUsedBytes=1368 to controller: pinot-controller-0, version: Unknown
2023-08-22 08:57:22     at org.apache.pinot.common.utils.http.HttpClient.wrapAndThrowHttpException(HttpClient.java:442) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-08-22 08:57:22     at org.apache.pinot.common.utils.FileUploadDownloadClient.sendSegmentCompletionProtocolRequest(FileUploadDownloadClient.java:1039) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-08-22 08:57:22     at org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler.sendRequest(ServerSegmentCompletionProtocolHandler.java:217) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-08-22 08:57:22     at org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler.segmentConsumed(ServerSegmentCompletionProtocolHandler.java:184) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-08-22 08:57:22     at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.postSegmentConsumedMsg(LLRealtimeSegmentDataManager.java:1110) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-08-22 08:57:22     at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:650) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-08-22 08:57:22     at java.lang.Thread.run(Thread.java:829) [?:?]
2023-08-22 08:57:22 2023/08/22 11:57:22.484 ERROR [LLRealtimeSegmentDataManager_events_upsert_full__1__0__20230822T1152Z] [events_upsert_full__1__0__20230822T1152Z] Holding after response from Controller: {"offset":-1,"status":"NOT_SENT","streamPartitionMsgOffset":null,"isSplitCommitType":false,"buildTimeSec":-1}
2023-08-22 08:57:24 2023/08/22 11:57:24.732 ERROR [ServerSegmentCompletionProtocolHandler] [events_upsert_partial__1__0__20230822T1152Z] Could not send request http://pinot-controller-0:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=5&instance=Server_pinot-server-0_8098&offset=-1&name=events_upsert_partial__1__0__20230822T1152Z&rowCount=5&memoryUsedBytes=1368
2023-08-22 08:57:24 org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 403 (Forbidden) with reason: "Permission is denied for READ '/segmentConsumed'" while sending request: http://pinot-controller-0:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=5&instance=Server_pinot-server-0_8098&offset=-1&name=events_upsert_partial__1__0__20230822T1152Z&rowCount=5&memoryUsedBytes=1368 to controller: pinot-controller-0, version: Unknown
2023-08-22 08:57:24     at org.apache.pinot.common.utils.http.HttpClient.wrapAndThrowHttpException(HttpClient.java:442) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-08-22 08:57:24     at org.apache.pinot.common.utils.FileUploadDownloadClient.sendSegmentCompletionProtocolRequest(FileUploadDownloadClient.java:1039) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-08-22 08:57:24     at org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler.sendRequest(ServerSegmentCompletionProtocolHandler.java:217) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-08-22 08:57:24     at org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler.segmentConsumed(ServerSegmentCompletionProtocolHandler.java:184) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-08-22 08:57:24     at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.postSegmentConsumedMsg(LLRealtimeSegmentDataManager.java:1110) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-08-22 08:57:24     at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:650) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-08-22 08:57:24     at java.lang.Thread.run(Thread.java:829) [?:?]

Server Appconfig

curl -X GET "http://localhost:38097/appconfigs" -H "accept: application/json" -H "Authorization: Basic YWRtaW46dmVyeXNlY3JldA=="
{
  "systemConfig" : {
    "arch" : "amd64",
    "name" : "Linux",
    "version" : "5.15.49-linuxkit-pr",
    "availableProcessors" : 4,
    "totalPhysicalMemory" : "7 GB",
    "freePhysicalMemory" : "164 MB",
    "totalSwapSpace" : "3 GB",
    "freeSwapSpace" : "2 GB"
  },
  "runtimeConfig" : {
    "numTotalThreads" : 69,
    "numCurrentThreads" : 52,
    "maxHeapSize" : "1 GB",
    "currentHeapSize" : "198 MB"
  },
  "pinotConfig" : {
    "pinot.server.instance.enable.split.commit" : "true",
    "pinot.server.storage.factory.class.s3" : "org.apache.pinot.plugin.filesystem.S3PinotFS",
    "pinot.server.segment.fetcher.s3.class" : "org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher",
    "pinot.server.storage.factory.s3.region" : "us-east-1",
    "pinot.server.storage.factory.s3.disableacl" : "false",
    "pinot.server.netty.port" : "8098",
    "pinot.cluster.name" : "pinot-s3-example",
    "pinot.server.instance.segment.store.uri" : "s3://pinot/deep-storage",
    "pinot.set.instance.id.to.hostname" : "true",
    "pinot.server.storage.factory.s3.accesskey" : "minio",
    "pinot.server.storage.factory.s3.endpoint" : "http://minio:9000",
    "pinot.server.segment.fetcher.protocols" : "file,http,s3",
    "pinot.server.instance.datadir" : "/var/pinot/server/data/index",
    "pinot.zk.server" : "zookeeper:2181",
    "pinot.server.adminapi.port" : "8097",
    "pinot.server.segment.uploader.auth.token" : "*****",
    "pinot.server.instance.id" : "Server_pinot-server-1_8098",
    "pinot.server.instance.auth.token" : "*****",
    "pinot.server.segment.fetcher.auth.token" : "*****",
    "pinot.server.instance.segmenttardir" : "/var/pinot/server/data/segment",
    "pinot.server.storage.factory.s3.secretkey" : "*****",
    "pinot.service.role" : "SERVER"
  },
  "jvmConfig" : {
    "args" : [ "-Dplugins.dir=/opt/pinot/plugins", "-Xms256M", "-Xmx1G", "-XX:+UseG1GC", "-XX:MaxGCPauseMillis=200", "-Xlog:gc:gc-pinot-server-1.log", "-Dplugins.dir=/opt/pinot/plugins", "-Dapp.name=pinot-admin", "-Dapp.pid=1", "-Dapp.repo=/opt/pinot/lib", "-Dapp.home=/opt/pinot", "-Dbasedir=/opt/pinot" ],
    "garbageCollectors" : [ "G1 Young Generation", "G1 Old Generation" ],
    "envVariables" : {
      "PATH" : "/usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "HOSTNAME" : "pinot-server-1",
      "PINOT_HOME" : "/opt/pinot",
      "JAVA_HOME" : "/usr/local/openjdk-11",
      "JAVA_OPTS" : "-Dplugins.dir=/opt/pinot/plugins -Xms256M -Xmx1G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc:gc-pinot-server-1.log",
      "PWD" : "/opt/pinot",
      "JAVA_VERSION" : "11.0.16",
      "LANG" : "C.UTF-8",
      "HOME" : "/root"
    },
    "libraryPath" : "/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib",
    "bootClassPath" : null,
    "systemProperties" : {
      "awt.toolkit" : "sun.awt.X11.XToolkit",
      "jdk.tls.rejectClientInitiatedRenegotiation" : "true",
      "helixmanager.flappingTimeWindow" : "1",
      "java.specification.version" : "11",
      "sun.cpu.isalist" : "",
      "sun.jnu.encoding" : "UTF-8",
      "java.class.path" : "/opt/pinot/lib/pinot-all-0.12.1-jar-with-dependencies.jar:/opt/pinot/plugins/pinot-input-format/pinot-thrift/pinot-thrift-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-input-format/pinot-confluent-avro/pinot-confluent-avro-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-input-format/pinot-parquet/pinot-parquet-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-input-format/pinot-json/pinot-json-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-input-format/pinot-protobuf/pinot-protobuf-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-input-format/pinot-orc/pinot-orc-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-input-format/pinot-csv/pinot-csv-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-input-format/pinot-avro/pinot-avro-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-segment-uploader/pinot-segment-uploader-default/pinot-segment-uploader-default-0.12.1.jar:/opt/pinot/plugins/pinot-file-system/pinot-hdfs/pinot-hdfs-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-file-system/pinot-gcs/pinot-gcs-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-file-system/pinot-s3/pinot-s3-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-file-system/pinot-adls/pinot-adls-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-segment-writer/pinot-segment-writer-file-based/pinot-segment-writer-file-based-0.12.1.jar:/opt/pinot/plugins/pinot-environment/pinot-azure/pinot-azure-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-metrics/pinot-yammer/pinot-yammer-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-metrics/pinot-dropwizard/pinot-dropwizard-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/pinot-minion-builtin-tasks-0.12.1.jar:/opt/pinot/plugins/pinot-stream-ingestion/pinot-kinesis/pinot-kinesis-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-stream-ingestion/pinot-kafka-2.0/pinot-kafka-2.0-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-batch-ingestion/pinot-batch-ingestion-standalone/pinot-batch-ingestion-standalone-0.12.1-shaded.jar",
      "java.vm.vendor" : "Oracle Corporation",
      "sun.arch.data.model" : "64",
      "java.vendor.url" : "https://openjdk.java.net/",
      "user.timezone" : "Etc/UTC",
      "os.name" : "Linux",
      "java.vm.specification.version" : "11",
      "sun.java.launcher" : "SUN_STANDARD",
      "sun.boot.library.path" : "/usr/local/openjdk-11/lib",
      "sun.java.command" : "org.apache.pinot.tools.admin.PinotAdministrator StartServer -zkAddress zookeeper:2181 -clusterName pinot-s3-example -configFileName /opt/pinot/conf/server.conf",
      "app.pid" : "1",
      "jdk.debug" : "release",
      "sun.cpu.endian" : "little",
      "user.home" : "/root",
      "user.language" : "en",
      "java.specification.vendor" : "Oracle Corporation",
      "java.version.date" : "2022-07-19",
      "java.home" : "/usr/local/openjdk-11",
      "file.separator" : "/",
      "basedir" : "/opt/pinot",
      "java.vm.compressedOopsMode" : "32-bit",
      "line.separator" : "\n",
      "java.specification.name" : "Java Platform API Specification",
      "java.vm.specification.vendor" : "Oracle Corporation",
      "java.awt.graphicsenv" : "sun.awt.X11GraphicsEnvironment",
      "sun.management.compiler" : "HotSpot 64-Bit Tiered Compilers",
      "plugins.dir" : "/opt/pinot/plugins",
      "java.runtime.version" : "11.0.16+8",
      "user.name" : "root",
      "path.separator" : ":",
      "os.version" : "5.15.49-linuxkit-pr",
      "java.runtime.name" : "OpenJDK Runtime Environment",
      "file.encoding" : "UTF-8",
      "java.vm.name" : "OpenJDK 64-Bit Server VM",
      "java.vendor.version" : "18.9",
      "app.name" : "pinot-admin",
      "java.vendor.url.bug" : "https://bugreport.java.com/bugreport/",
      "java.io.tmpdir" : "/tmp",
      "java.version" : "11.0.16",
      "user.dir" : "/opt/pinot",
      "os.arch" : "amd64",
      "java.vm.specification.name" : "Java Virtual Machine Specification",
      "java.awt.printerjob" : "sun.print.PSPrinterJob",
      "sun.os.patch.level" : "unknown",
      "app.home" : "/opt/pinot",
      "java.library.path" : "/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib",
      "java.vm.info" : "mixed mode, sharing",
      "java.vendor" : "Oracle Corporation",
      "java.vm.version" : "11.0.16+8",
      "app.repo" : "/opt/pinot/lib",
      "sun.io.unicode.encoding" : "UnicodeLittle",
      "java.class.version" : "55.0"
    }
  }

Controller appconf

 curl -X GET "http://localhost:9000/appconfigs" -H "accept: application/json" -H "Authorization: Basic YWRtaW46dmVyeXNlY3JldA=="
{
  "systemConfig" : {
    "arch" : "amd64",
    "name" : "Linux",
    "version" : "5.15.49-linuxkit-pr",
    "availableProcessors" : 4,
    "totalPhysicalMemory" : "7 GB",
    "freePhysicalMemory" : "163 MB",
    "totalSwapSpace" : "3 GB",
    "freeSwapSpace" : "1 GB"
  },
  "runtimeConfig" : {
    "numTotalThreads" : 203,
    "numCurrentThreads" : 125,
    "maxHeapSize" : "1 GB",
    "currentHeapSize" : "247 MB"
  },
  "pinotConfig" : {
    "controller.task.frequencyperiod" : "5m",
    "controller.data.dir" : "s3://pinot/deep-storage",
    "controller.enable.split.commit" : "true",
    "pinot.controller.segment.fetcher.s3.class" : "org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher",
    "pinot.controller.storage.factory.s3.disableacl" : "false",
    "controller.admin.access.control.principals.user.tables" : "events,events_upsert_full,events_upsert_partial",
    "controller.allow.hlc.tables" : "false",
    "pinot.cluster.name" : "pinot-s3-example",
    "controller.task.scheduler.enabled" : "true",
    "pinot.set.instance.id.to.hostname" : "true",
    "controller.host" : "pinot-controller-0",
    "pinot.controller.storage.factory.s3.endpoint" : "http://minio:9000",
    "controller.retention.frequencyperiod" : "30m",
    "pinot.zk.server" : "zookeeper:2181",
    "controller.helix.cluster.name" : "pinot-s3-example",
    "controller.admin.access.control.principals" : "admin,user",
    "pinot.controller.storage.factory.s3.accesskey" : "minio",
    "controller.zk.str" : "zookeeper:2181",
    "pinot.service.role" : "CONTROLLER",
    "controller.admin.access.control.principals.admin.password" : "*****",
    "controller.admin.access.control.factory.class" : "org.apache.pinot.controller.api.access.BasicAuthAccessControlFactory",
    "controller.access.protocols.http.port" : "9000",
    "controller.local.temp.dir" : "/tmp/pinot-tmp-data/",
    "controller.admin.access.control.principals.user.password" : "*****",
    "pinot.controller.segment.fetcher.auth.token" : "*****",
    "controller.port" : "9000",
    "pinot.controller.storage.factory.s3.secretkey" : "*****",
    "pinot.controller.storage.factory.class.s3" : "org.apache.pinot.plugin.filesystem.S3PinotFS",
    "pinot.controller.segment.fetcher.protocols" : "file,http,s3",
    "pinot.controller.storage.factory.s3.region" : "us-east-1",
    "controller.admin.access.control.principals.user.permissions" : "READ",
    "controller.realtime.segment.deepstoreuploadretryenabled" : "true"
  },
  "jvmConfig" : {
    "args" : [ "-Dplugins.dir=/opt/pinot/plugins", "-Xms256M", "-Xmx1G", "-XX:+UseG1GC", "-XX:MaxGCPauseMillis=200", "-Xlog:gc:gc-pinot-controller-0.log", "-Dplugins.dir=/opt/pinot/plugins", "-Dapp.name=pinot-admin", "-Dapp.pid=1", "-Dapp.repo=/opt/pinot/lib", "-Dapp.home=/opt/pinot", "-Dbasedir=/opt/pinot" ],
    "garbageCollectors" : [ "G1 Young Generation", "G1 Old Generation" ],
    "envVariables" : {
      "PATH" : "/usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "HOSTNAME" : "pinot-controller-0",
      "PINOT_HOME" : "/opt/pinot",
      "JAVA_HOME" : "/usr/local/openjdk-11",
      "JAVA_OPTS" : "-Dplugins.dir=/opt/pinot/plugins -Xms256M -Xmx1G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc:gc-pinot-controller-0.log",
      "PWD" : "/opt/pinot",
      "JAVA_VERSION" : "11.0.16",
      "LANG" : "C.UTF-8",
      "HOME" : "/root"
    },
    "libraryPath" : "/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib",
    "bootClassPath" : null,
    "systemProperties" : {
      "awt.toolkit" : "sun.awt.X11.XToolkit",
      "jdk.tls.rejectClientInitiatedRenegotiation" : "true",
      "helixmanager.flappingTimeWindow" : "1",
      "java.specification.version" : "11",
      "sun.cpu.isalist" : "",
      "sun.jnu.encoding" : "UTF-8",
      "java.class.path" : "/opt/pinot/lib/pinot-all-0.12.1-jar-with-dependencies.jar:/opt/pinot/plugins/pinot-input-format/pinot-thrift/pinot-thrift-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-input-format/pinot-confluent-avro/pinot-confluent-avro-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-input-format/pinot-parquet/pinot-parquet-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-input-format/pinot-json/pinot-json-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-input-format/pinot-protobuf/pinot-protobuf-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-input-format/pinot-orc/pinot-orc-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-input-format/pinot-csv/pinot-csv-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-input-format/pinot-avro/pinot-avro-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-segment-uploader/pinot-segment-uploader-default/pinot-segment-uploader-default-0.12.1.jar:/opt/pinot/plugins/pinot-file-system/pinot-hdfs/pinot-hdfs-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-file-system/pinot-gcs/pinot-gcs-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-file-system/pinot-s3/pinot-s3-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-file-system/pinot-adls/pinot-adls-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-segment-writer/pinot-segment-writer-file-based/pinot-segment-writer-file-based-0.12.1.jar:/opt/pinot/plugins/pinot-environment/pinot-azure/pinot-azure-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-metrics/pinot-yammer/pinot-yammer-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-metrics/pinot-dropwizard/pinot-dropwizard-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/pinot-minion-builtin-tasks-0.12.1.jar:/opt/pinot/plugins/pinot-stream-ingestion/pinot-kinesis/pinot-kinesis-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-stream-ingestion/pinot-kafka-2.0/pinot-kafka-2.0-0.12.1-shaded.jar:/opt/pinot/plugins/pinot-batch-ingestion/pinot-batch-ingestion-standalone/pinot-batch-ingestion-standalone-0.12.1-shaded.jar",
      "java.vm.vendor" : "Oracle Corporation",
      "sun.arch.data.model" : "64",
      "java.vendor.url" : "https://openjdk.java.net/",
      "user.timezone" : "Etc/UTC",
      "os.name" : "Linux",
      "java.vm.specification.version" : "11",
      "sun.java.launcher" : "SUN_STANDARD",
      "sun.boot.library.path" : "/usr/local/openjdk-11/lib",
      "sun.java.command" : "org.apache.pinot.tools.admin.PinotAdministrator StartController -zkAddress zookeeper:2181 -configFileName /opt/pinot/conf/controller.conf",
      "app.pid" : "1",
      "jdk.debug" : "release",
      "sun.cpu.endian" : "little",
      "user.home" : "/root",
      "user.language" : "en",
      "java.specification.vendor" : "Oracle Corporation",
      "java.version.date" : "2022-07-19",
      "java.home" : "/usr/local/openjdk-11",
      "file.separator" : "/",
      "basedir" : "/opt/pinot",
      "java.vm.compressedOopsMode" : "32-bit",
      "line.separator" : "\n",
      "java.specification.name" : "Java Platform API Specification",
      "java.vm.specification.vendor" : "Oracle Corporation",
      "java.awt.graphicsenv" : "sun.awt.X11GraphicsEnvironment",
      "sun.management.compiler" : "HotSpot 64-Bit Tiered Compilers",
      "plugins.dir" : "/opt/pinot/plugins",
      "java.runtime.version" : "11.0.16+8",
      "user.name" : "root",
      "path.separator" : ":",
      "os.version" : "5.15.49-linuxkit-pr",
      "java.runtime.name" : "OpenJDK Runtime Environment",
      "file.encoding" : "UTF-8",
      "java.vm.name" : "OpenJDK 64-Bit Server VM",
      "java.vendor.version" : "18.9",
      "app.name" : "pinot-admin",
      "java.vendor.url.bug" : "https://bugreport.java.com/bugreport/",
      "java.io.tmpdir" : "/tmp",
      "java.version" : "11.0.16",
      "user.dir" : "/opt/pinot",
      "os.arch" : "amd64",
      "java.vm.specification.name" : "Java Virtual Machine Specification",
      "java.awt.printerjob" : "sun.print.PSPrinterJob",
      "sun.os.patch.level" : "unknown",
      "app.home" : "/opt/pinot",
      "java.library.path" : "/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib",
      "java.vm.info" : "mixed mode, sharing",
      "java.vendor" : "Oracle Corporation",
      "java.vm.version" : "11.0.16+8",
      "app.repo" : "/opt/pinot/lib",
      "sun.io.unicode.encoding" : "UnicodeLittle",
      "java.class.version" : "55.0"
    }
  }
}

Thank you very much!

Jackie-Jiang commented 10 months ago

cc @apucher @xiangfu0 @snleee

This is also a good question to be posted in the slack troubleshooting channel

rodolphogarrido commented 9 months ago

Hi @Jackie-Jiang, as suggested I have posted in the slack troubleshooting channel and the issue was reproduced by @HubertWo, as well as been reported to be affecting another user.

Would you consider to tag this as a BUG?

Thank you very much!

CA1087 commented 9 months ago

Hi Team,

We are facing the same issue as well.

jddsc commented 9 months ago

Hi team, same issue as well. Could you share your workaround (if any)? @rodolphogarrido

xiangfu0 commented 8 months ago

Looking into this, hypnosis is that server calls to controller may not carry the credential.

mayankshriv commented 8 months ago

@KKcorps assigning to you, please take a look.

KKcorps commented 8 months ago

@rodolphogarrido does this happen on the first segment commit or do you see some segments getting committed correctly and then all of a sudden it starts failing?

You can check the logs for if there's any successfull call for /segmentConsumed

KKcorps commented 8 months ago

Also, has the ACL configs been enabled since the first deployment or were they enabled later on? If later on, were all the pinot components restarted or only controller or server?

CA1087 commented 8 months ago

@KKcorps - in my case ACL config was enabled since the first deployment and it was working for a while until it stops working and if i restart the components it works for sometime till this happens again.

We dont see any pattern on why this is happening but it stops all of a sudden.

KKcorps commented 8 months ago

@KKcorps - in my case ACL config was enabled since the first deployment and it was working for a while until it stops working and if i restart the components it works for sometime till this happens again.

We dont see any pattern on why this is happening but it stops all of a sudden.

Thanks for the reply. Can you also check the server logs for /segmentConsumed calls and see if all have failed or some are successfull

CA1087 commented 8 months ago

@KKcorps - All the calls are getting failed.

KKcorps commented 8 months ago

@KKcorps - All the calls are getting failed.

Thanks, that means that there is auth token mismatch. Between server and controller. I tried reproducing it in local with 0.12.1 but it's working fine

can you validate the pinot.server.segment.uploader.auth.token and see if it matches the base64 encoded value in controller

CA1087 commented 8 months ago

yes @KKcorps I verified the same and it was right correctly in the server config. The server is able to consume message for sometime and it stops after a point.

KKcorps commented 8 months ago

yes @KKcorps I verified the same and it was right correctly in the server config. The server is able to consume message for sometime and it stops after a point.

Yes that I understand. So what's happening is that the consumption is not affected. Once you consume enough data pinot needs to commit that segment. It's during that call that you are getting the error.

Will see if something is missing. also will it be possible for you to reproduce it with pinot 1.0

CA1087 commented 8 months ago

@KKcorps - One more thing that i notice that in our config we are using something like this pinot.server.segment.uploader.auth.token=${env:PINOT_AUTH} and PINOT_AUTH was not getting updated inside /var/pinot/server/config/pinot-server.conf. Will this be related?

KKcorps commented 8 months ago

@KKcorps - One more thing that i notice that in our config we are using something like this pinot.server.segment.uploader.auth.token=${env:PINOT_AUTH} and PINOT_AUTH was not getting updated inside /var/pinot/server/config/pinot-server.conf. Will this be related?

Yes, that is the exact property that will cause the issue. It needs to have correct value

CA1087 commented 8 months ago

What should be the correct value? I have the env variable named PINOT_AUTH set from the secret in kubernetes.

rodolphogarrido commented 8 months ago

Hi, @KKcorps how are you? Thanks for looking into this issue!

Answering your questions:

  1. The first segment commit is getting commited, but subsequent ones are not commited and this is when the error happens.

  2. Since this is an ephemeral test cluster I have tried both cases with a fresh deployment for each case (including Zookeeper), with the ACL enabled since the first deployment, as well as enabling it latter and restarting every component and faced the issue in both cases.

  3. In my case I can confirm that every token has the correct value in base64 (hardcoded them to the config for this test).

KKcorps commented 8 months ago

Hi, @KKcorps how are you? Thanks for looking into this issue!

Answering your questions:

  1. The first segment commit is getting commited, but subsequent ones are not commited and this is when the error happens.

  2. Since this is an ephemeral test cluster I have tried both cases with a fresh deployment for each case (including Zookeeper), with the ACL enabled since the first deployment, as well as enabling it latter and restarting every component and faced the issue in both cases.

  3. In my case I can confirm that every token has the correct value in base64 (hardcoded them to the config for this test).

Thanks a lot for clarifying! Yeah if the first commit is happening and subsequent ones are not then it is quite weird.

One more question were you facing this issue in older pinot version as well? also can you try the latest pinot 1.0 release and see if it occurs

KKcorps commented 8 months ago

What should be the correct value? I have the env variable named PINOT_AUTH set from the secret in kubernetes.

I meant the ${env:PINOT_AUTH} should resolve to the base64 auth token e.g. Basic YWRtaW46dmVyeXNlY3JldA==

rodolphogarrido commented 8 months ago

@KKcorps after double checking, it seems that the first segment isn't actually beeing commited (sorry for the mistake). Pinot is able to consume some record (since I can query a few ingested records), but when it tries to commit the first segment it fails and than consumption also stops.

I started testing pinot with the version 0.12.1 (didn't try older version). I've tried the new version (1.0.0) and the same issue happens.

Trying with version 1.0.0, the server logs show:

2023-10-13 07:17:26 2023/10/13 10:17:26.493 ERROR [LLRealtimeSegmentDataManager_events_upsert_full__1__0__20231013T1014Z] [events_upsert_full__1__0__20231013T1014Z] Holding after response from Controller: {"status":"NOT_SENT","streamPartitionMsgOffset":null,"isSplitCommitType":false,"buildTimeSec":-1,"offset":-1}
2023-10-13 07:17:29 2023/10/13 10:17:29.438 ERROR [ServerSegmentCompletionProtocolHandler] [events_upsert_partial__1__0__20231013T1014Z] Could not send request http://pinot-controller-0:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=2&instance=Server_pinot-server-0_8098&offset=-1&name=events_upsert_partial__1__0__20231013T1014Z&rowCount=2&memoryUsedBytes=1140
2023-10-13 07:17:29 org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 403 (Forbidden) with reason: "Permission is denied for READ '/segmentConsumed'" while sending request: http://pinot-controller-0:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=2&instance=Server_pinot-server-0_8098&offset=-1&name=events_upsert_partial__1__0__20231013T1014Z&rowCount=2&memoryUsedBytes=1140 to controller: pinot-controller-0, version: Unknown
2023-10-13 07:17:29     at org.apache.pinot.common.utils.http.HttpClient.wrapAndThrowHttpException(HttpClient.java:448) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
2023-10-13 07:17:29     at org.apache.pinot.common.utils.FileUploadDownloadClient.sendSegmentCompletionProtocolRequest(FileUploadDownloadClient.java:1129) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
2023-10-13 07:17:29     at org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler.sendRequest(ServerSegmentCompletionProtocolHandler.java:221) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
2023-10-13 07:17:29     at org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler.segmentConsumed(ServerSegmentCompletionProtocolHandler.java:188) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
2023-10-13 07:17:29     at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.postSegmentConsumedMsg(LLRealtimeSegmentDataManager.java:1152) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
2023-10-13 07:17:29     at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:700) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
2023-10-13 07:17:29     at java.lang.Thread.run(Thread.java:829) [?:?]
2023-10-13 07:17:29 2023/10/13 10:17:29.440 ERROR [LLRealtimeSegmentDataManager_events_upsert_partial__1__0__20231013T1014Z] [events_upsert_partial__1__0__20231013T1014Z] Holding after response from Controller: {"status":"NOT_SENT","streamPartitionMsgOffset":null,"isSplitCommitType":false,"buildTimeSec":-1,"offset":-1}

If I try the same request from the error, using the admin secret:

curl -i -X GET -H 'Content-Type: application/json' -u "admin:verysecret" localhost:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=2&instance=Server_pinot-server-0_8098&offset=-1&name=events_upsert_partial__1__0__20231013T1014Z&rowCount=2&memoryUsedBytes=1140

The output is:

{"status":"COMMIT","isSplitCommitType":true,"buildTimeSec":126,"streamPartitionMsgOffset":"2","controllerVipUrl":"http://pinot-controller-0:9000","offset":2}
ykulah commented 5 months ago

This issue still persists on 1.0.0, Pinot servers cannot authenticate themselves to Pinot controller. This is odd because documentation says Pinot supports basic auth. This issue causes Pinot not to be able to function with a level of security.

Is there any known workaround so far?

vineethvp commented 4 months ago

Hi, Facing the same issue in 1.0.0 version. Is there any ETA on when this be fixed or any workaround? This is a blocker as new segments are not getting created after threshold is met.

Jackie-Jiang commented 4 months ago

@zhtaoxiang Can you please take a look?

EnzoDechaene commented 1 month ago

any news on this ?

vineethvp commented 1 month ago

@EnzoDechaene

Check your auth configurations. Add below configs to server if not added yet.

pinot.server.segment.uploader.auth.token="Basic XXXXXXX" pinot.server.instance.auth.token="Basic XXXXXXX"