druid-io / docker-druid

Druid Docker
197 stars 159 forks source link

403 Forbidden during docker run #67

Open moezubair opened 5 years ago

moezubair commented 5 years ago

Hey, I am getting the following error when i run the docker image:

2019-01-22 16:56:43,455 DEBG 'druid-historical' stdout output:
2019-01-22T16:56:43,446 ERROR [ZkCoordinator] io.druid.server.coordination.SegmentLoadDropHandler - Failed to load segment for dataSource: {class=io.druid.server.coordination.SegmentLoadDropHandler, exceptionType=class io.druid.segment.loading.SegmentLoadingException, exceptionMessage=Exception loading segment[wikipedia_2013-08-01T00:00:00.000Z_2013-08-02T00:00:00.000Z_2013-08-08T21:22:48.989Z], segment=DataSegment{size=24664730, shardSpec=NoneShardSpec, metrics=[count, delta, variation, added, deleted], dimensions=[dma_code, continent_code, geo, area_code, robot, country_name, network, city, namespace, anonymous, unpatrolled, page, postal_code, language, newpage, user, region_lookup], version='2013-08-08T21:22:48.989Z', loadSpec={type=>s3_zip, bucket=>static.druid.io, key=>data/segments/wikipedia/20130801T000000.000Z_20130802T000000.000Z/2013-08-08T21_22_48.989Z/0/index.zip}, interval=2013-08-01T00:00:00.000Z/2013-08-02T00:00:00.000Z, dataSource='wikipedia', binaryVersion='9'}}
io.druid.segment.loading.SegmentLoadingException: Exception loading segment[wikipedia_2013-08-01T00:00:00.000Z_2013-08-02T00:00:00.000Z_2013-08-08T21:22:48.989Z]
    at io.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:265) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.server.coordination.SegmentLoadDropHandler.addSegment(SegmentLoadDropHandler.java:307) [druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:47) [druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.server.coordination.ZkCoordinator$1.childEvent(ZkCoordinator.java:118) [druid-services-0.12.1-selfcontained.jar:0.12.1]
    at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:520) [druid-services-0.12.1-selfcontained.jar:0.12.1]
    at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:514) [druid-services-0.12.1-selfcontained.jar:0.12.1]
    at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93) [druid-services-0.12.1-selfcontained.jar:0.12.1]
    at org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:296) [druid-services-0.12.1-selfcontained.jar:0.12.1]
    at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85) [druid-services-0.12.1-selfcontained.jar:0.12.1]
    at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:512) [druid-services-0.12.1-selfcontained.jar:0.12.1]
    at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [druid-services-0.12.1-selfcontained.jar:0.12.1]
    at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:771) [druid-services-0.12.1-selfcontained.jar:0.12.1]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_171]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_171]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_171]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_171]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: java.lang.RuntimeException: org.jets3t.service.ServiceException: Request Error. -- ResponseCode: 403, ResponseStatus: Forbidden, RequestId: B11F1C837771B9B6, HostId: vVwwPXur/GyoYZ3wmK0pSa9gtYL06t5dtFJIqerGDqQwM7Rek/ABBrnbGJQtLCgiwDzzItlCIh0=
    at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:327) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:176) ~[?:?]
    at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:60) ~[?:?]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:205) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:193) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:151) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:133) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:108) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.server.SegmentManager.getAdapter(SegmentManager.java:196) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.server.SegmentManager.loadSegment(SegmentManager.java:157) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:261) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    ... 18 more
Caused by: org.jets3t.service.ServiceException: Request Error.
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:426) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:279) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:1052) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2264) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2193) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:1120) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:575) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.storage.s3.S3Utils.isObjectInBucket(S3Utils.java:96) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:318) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:314) ~[?:?]
    at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:63) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.storage.s3.S3Utils.retryS3Operation(S3Utils.java:89) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:312) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:176) ~[?:?]
    at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:60) ~[?:?]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:205) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:193) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:151) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:133) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:108) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.server.SegmentManager.getAdapter(SegmentManager.java:196) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.server.SegmentManager.loadSegment(SegmentManager.java:157) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:261) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    ... 18 more
Caused by: org.jets3t.service.impl.rest.HttpException: 403 Forbidden
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:425) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:279) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:1052) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2264) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2193) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:1120) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:575) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.storage.s3.S3Utils.isObjectInBucket(S3Utils.java:96) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:318) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:314) ~[?:?]
    at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:63) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.storage.s3.S3Utils.retryS3Operation(S3Utils.java:89) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:312) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:176) ~[?:?]
    at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:60) ~[?:?]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:205) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:193) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:151) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:133) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:108) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.server.SegmentManager.getAdapter(SegmentManager.java:196) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.server.SegmentManager.loadSegment(SegmentManager.java:157) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    at io.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:261) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
    ... 18 more

2019-01-22 16:56:43,459 DEBG 'druid-coordinator' stdout output:
2019-01-22T16:56:43,458 INFO [main-EventThread] io.druid.server.coordinator.CuratorLoadQueuePeon - Server[/druid/loadQueue/172.17.0.2:8083] done processing [/druid/loadQueue/172.17.0.2:8083/wikipedia_2013-08-01T00:00:00.000Z_2013-08-02T00:00:00.000Z_2013-08-08T21:22:48.989Z]

2019-01-22 16:56:43,459 DEBG 'druid-historical' stdout output:
2019-01-22T16:56:43,458 INFO [ZkCoordinator] io.druid.server.coordination.ZkCoordinator - Completed request [LOAD: wikipedia_2013-08-01T00:00:00.000Z_2013-08-02T00:00:00.000Z_2013-08-08T21:22:48.989Z]
2019-01-22T16:56:43,458 INFO [ZkCoordinator] io.druid.server.coordination.ZkCoordinator - zNode[/druid/loadQueue/172.17.0.2:8083/wikipedia_2013-08-01T00:00:00.000Z_2013-08-02T00:00:00.000Z_2013-08-08T21:22:48.989Z] was removed
nickwallen commented 5 years ago

I am getting this also, but am not sure why. The stack trace seems to refer to retrieving data from an S3 bucket. The only use of S3 that I see here is in pulling down the example Wikipedia data.

nickwallen commented 5 years ago

The sample-data.sql has this ingest spec embedded in it. I assume the data set is not publicly available in S3?

{
  "dataSource": "wikipedia",
  "interval": "2013-08-01T00:00:00.000Z/2013-08-02T00:00:00.000Z",
  "version": "2013-08-08T21:22:48.989Z",
  "loadSpec": {
    "type": "s3_zip",
    "bucket": "static.druid.io",
    "key": "data/segments/wikipedia/20130801T000000.000Z_20130802T000000.000Z/2013-08-08T21_22_48.989Z/0/index.zip"
  },
  "dimensions": "dma_code,continent_code,geo,area_code,robot,country_name,network,city,namespace,anonymous,unpatrolled,page,postal_code,language,newpage,user,region_lookup",
  "metrics": "count,delta,variation,added,deleted",
  "shardSpec": {
    "type": "none"
  },
  "binaryVersion": 9,
  "size": 24664730,
  "identifier": "wikipedia_2013-08-01T00:00:00.000Z_2013-08-02T00:00:00.000Z_2013-08-08T21:22:48.989Z"
}
moezubair commented 5 years ago

That seems to be the issue....the supervisor config does have S3 config in it.

what needs to be changed to load this from a different file maybe?

moezubair commented 5 years ago

@nickwallen if you're still facing the problem, i found the fix.

The S3 cred info in supervisord.conf is no longer valid. I got the updated one from the druid main repo and it worked:

-Ddruid.s3.accessKey=AKIAJI7DG7CDECGBQ6NA -Ddruid.s3.secretKey=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