estebanz01
commented
8 months ago additional information from the pinot controller:
Closed estebanz01 closed 7 months ago
estebanz01
commented
8 months ago additional information from the pinot controller:
Jackie-Jiang
commented
8 months ago Can you also check the WARN log from minion?
@snleee @swaminathanmanish Please help take a look
estebanz01
commented
8 months ago Sure thing. All warn information goes to the same output, right ? or it's there another specific location I can look into.
estebanz01
commented
8 months ago here's the full log of a fresh minion pod:
and the pod description:
estebanz01
commented
7 months ago OK, so while reading #12458 I noticed that my S3 bucket were empty and I found surprising that minions need S3 to work, so I went to look at the configuration for the controller and I found out that if I specify the property controller.data.dir twice, it merges both values instead of overriding them 🙃 so now I have data in my S3 bucket, but now the controller is giving the following error:
pinot-controller-0 controller INFO: [HttpServer] Started.
pinot-controller-0 controller 17:43:02.118 [grizzly-http-server-3] ERROR org.apache.pinot.controller.util.CompletionServiceHelper - Server: null returned error: 404
pinot-controller-0 controller 17:43:02.123 [grizzly-http-server-3] ERROR org.apache.pinot.controller.util.CompletionServiceHelper - Server: null returned error: 404
pinot-controller-0 controller 17:43:02.125 [grizzly-http-server-3] ERROR org.apache.pinot.controller.util.CompletionServiceHelper - Connection error. Details: java.net.UnknownHostException: Controller_null: Name or service not known
pinot-controller-0 controller 17:56:00.608 [grizzly-http-server-3] ERROR org.apache.pinot.controller.util.CompletionServiceHelper - Server: null returned error: 404
pinot-controller-0 controller 17:56:00.610 [grizzly-http-server-3] ERROR org.apache.pinot.controller.util.CompletionServiceHelper - Server: null returned error: 404
pinot-controller-0 controller 17:56:00.611 [grizzly-http-server-3] ERROR org.apache.pinot.controller.util.CompletionServiceHelper - Connection error. Details: java.net.UnknownHostException: Controller_null: Name or service not knownhere's the task information, according to the UI:
and here's the configmap that the pinot controller pods are using:
Name: pinot-controller-config
Namespace: pinot
Labels: app.kubernetes.io/managed-by=Helm
Annotations: meta.helm.sh/release-name: pinot
meta.helm.sh/release-namespace: pinot
Data
====
pinot-controller.conf:
----
controller.helix.cluster.name=pinot
controller.port=9000
controller.vip.host=pinot-controller
controller.vip.port=9000
controller.data.dir=s3://<bucket-name>/pinot-data/pinot/controller-data
controller.zk.str=pinot-zookeeper:2181
pinot.set.instance.id.to.hostname=true
controller.task.scheduler.enabled=true
controller.local.temp.dir=/var/pinot/controller/data
pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.controller.storage.factory.s3.region=eu-west-1
pinot.controller.segment.fetcher.protocols=file,http,s3
pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.controller.storage.factory.s3.disableAcl=false
BinaryData
====
Events: <none>from what I understand, the controller is trying to fetch segments from either a Null hostname or an invalid one. But the hosts are correct or appears to be correct.
Any ideas on how to make it work after this progress?
estebanz01
commented
7 months ago btw, what does pinot.set.instance.id.to.hostname=true does and if I put it in false, how can I specify an alternative hostname?
Jackie-Jiang
commented
7 months ago You may look up the usage of CommonConstants.SET_INSTANCE_ID_TO_HOSTNAME_KEY from the code.
You can use controller.host key to specify the host name
estebanz01
commented
7 months ago OK, after lots of trial and error, this is what I did to have a working hybrid table with minion tasks and S3 deep storage:
Basically, we had to configure the S3 filesystem on controller, server and minion so the workers can fetch and upload/download data when needed. I'm not sure how with other deep storage options it might look like, but it seems that all three components must be in config-sync, if that makes sense.
Thanks for the help on this!
Hola! 👋
I'm having some trouble with a pinot cluster deployed into kubernetes with minion enabled. I want to move data from real time table to offline table but it's failing with the following information:
That's the only error I see in the minion pods and there's nothing else on the other pods for pinot. Any ideas on how to debug this further? here's the schema and table config for my hybrid table:
Schema definition
```json { "schemaName": "data_counting", "dimensionFieldSpecs": [ { "name": "device_name", "dataType": "STRING" } ], "metricFieldSpecs": [ { "name": "bytes_sent", "dataType": "LONG" } ], "dateTimeFieldSpecs": [ { "name": "__key", "dataType": "TIMESTAMP", "format": "1:MICROSECONDS:EPOCH", "granularity": "1:MICROSECONDS" }, { "name": "__metadata$eventTime", "dataType": "TIMESTAMP", "format": "1:MICROSECONDS:EPOCH", "granularity": "1:MICROSECONDS" } ] } ```Table configuration (REALTIME)
```json { "REALTIME": { "tableName": "data_counting_REALTIME", "tableType": "REALTIME", "segmentsConfig": { "schemaName": "data_counting", "replication": "1", "retentionTimeUnit": "DAYS", "retentionTimeValue": "15", "replicasPerPartition": "1", "minimizeDataMovement": false, "timeColumnName": "__key" }, "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant", "tagOverrideConfig": {} }, "tableIndexConfig": { "invertedIndexColumns": [], "noDictionaryColumns": [], "streamConfigs": { "streamType": "pulsar", "stream.pulsar.topic.name": "persistent://client/devices/all", "stream.pulsar.bootstrap.servers": "pulsar://pulsar-proxy.pulsar.svc.cluster.local:6650", "stream.pulsar.prop.auto.offset.reset": "smallest", "stream.pulsar.consumer.type": "lowlevel", "stream.pulsar.fetch.timeout.millis": "20000", "stream.pulsar.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder", "stream.pulsar.consumer.factory.class.name": "org.apache.pinot.plugin.stream.pulsar.PulsarConsumerFactory", "realtime.segment.flush.threshold.rows": "10000", "realtime.segment.flush.threshold.time": "1h", "stream.pulsar.metada.populate": "true", "stream.pulsar.metadata.fields": "eventTime" }, "loadMode": "MMAP", "onHeapDictionaryColumns": [], "varLengthDictionaryColumns": [], "enableDefaultStarTree": false, "enableDynamicStarTreeCreation": false, "aggregateMetrics": false, "nullHandlingEnabled": false, "rangeIndexColumns": [], "rangeIndexVersion": 2, "optimizeDictionary": false, "optimizeDictionaryForMetrics": false, "noDictionarySizeRatioThreshold": 0.85, "autoGeneratedInvertedIndex": false, "createInvertedIndexDuringSegmentGeneration": false, "sortedColumn": [], "bloomFilterColumns": [] }, "metadata": {}, "quota": {}, "task": { "taskTypeConfigsMap": { "RealtimeToOfflineSegmentsTask": { "bucketTimePeriod": "1h", "bufferTimePeriod": "2h", "mergeType": "concat", "maxNumRecordsPerSegment": "100000", "schedule": "0 * * * * ?" } } }, "routing": {}, "query": { "timeoutMs": 60000 }, "ingestionConfig": { "continueOnError": false, "rowTimeValueCheck": false, "segmentTimeValueCheck": true }, "isDimTable": false } } ```Table configuration (OFFLINE)
```json { "OFFLINE": { "tableName": "data_counting_OFFLINE", "tableType": "OFFLINE", "segmentsConfig": { "schemaName": "data_counting", "replication": "1", "replicasPerPartition": "1", "timeColumnName": "__key", "minimizeDataMovement": false, "segmentPushType": "APPEND", "segmentPushFrequency": "HOURLY" }, "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant" }, "tableIndexConfig": { "invertedIndexColumns": [], "noDictionaryColumns": [], "rangeIndexColumns": [], "rangeIndexVersion": 2, "createInvertedIndexDuringSegmentGeneration": false, "autoGeneratedInvertedIndex": false, "sortedColumn": [], "bloomFilterColumns": [], "loadMode": "MMAP", "onHeapDictionaryColumns": [], "varLengthDictionaryColumns": [], "enableDefaultStarTree": false, "enableDynamicStarTreeCreation": false, "aggregateMetrics": false, "nullHandlingEnabled": false, "optimizeDictionary": false, "optimizeDictionaryForMetrics": false, "noDictionarySizeRatioThreshold": 0.85 }, "metadata": {}, "quota": {}, "routing": {}, "query": {}, "ingestionConfig": { "continueOnError": false, "rowTimeValueCheck": false, "segmentTimeValueCheck": true }, "isDimTable": false } } ```I'm using apache pulsar 3.2.0 and apache pinot version 1.0.0.