search

apache / pinot

Apache Pinot - A realtime distributed OLAP datastore
https://pinot.apache.org/
Apache License 2.0
5.31k stars 1.24k forks source link

/ingestFromFile rest call fails to upload segment if user name password is enabled Pinot 0.9.2 #9027

Open pjpringle opened 2 years ago

pjpringle commented 2 years ago

Looks like auth token needs to be passed through when calling the controller API

/ingestFromFile

which calls

/segments api

Caused by: org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 403 (Forbidden) with reason: "Permission is denied for access type 'CREATE' to the endpoint 'http://somehost:9000/v2/segments' for table 'dimSomeTable_OFFLINE'" while sending request: http://somehost:9000/v2/segments?tableName=dimSomeTable to controller: somehost, version: Unknown
        at org.apache.pinot.common.utils.FileUploadDownloadClient.sendRequest(FileUploadDownloadClient.java:510) ~[pinot-all-0.9.2-jar-with-dependencies.jar:0.9.2-3e5e5aca07ad796d453ad
d80a2b2b1bf1d654cf1]
        at org.apache.pinot.common.utils.FileUploadDownloadClient.uploadSegment(FileUploadDownloadClient.java:791) ~[pinot-all-0.9.2-jar-with-dependencies.jar:0.9.2-3e5e5aca07ad796d453
add80a2b2b1bf1d654cf1]
        at org.apache.pinot.segment.local.utils.SegmentPushUtils.lambda$pushSegments$0(SegmentPushUtils.java:120) ~[pinot-all-0.9.2-jar-with-dependencies.jar:0.9.2-3e5e5aca07ad796d453a
dd80a2b2b1bf1d654cf1]
        at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:50) ~[pinot-all-0.9.2-jar-with-dependencies.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]
        at org.apache.pinot.segment.local.utils.SegmentPushUtils.pushSegments(SegmentPushUtils.java:117) ~[pinot-all-0.9.2-jar-with-dependencies.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b
1bf1d654cf1]
        at org.apache.pinot.segment.local.utils.IngestionUtils.uploadSegment(IngestionUtils.java:204) ~[pinot-all-0.9.2-jar-with-dependencies.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf
1d654cf1]
        ... 28 more
2022/07/06 23:31:00.893 ERROR [PinotIngestionRestletResource] [jersey-server-managed-async-executor-5] Caught exception when ingesting file into table: dimSomeTable_OFFLINE. Caught
exception while uploading segments. Push mode: TAR, segment tars: [[file:/local/data/logs/s3:/blah/upload_
dir/working_dir_dimSomeTable_OFFLINE_1657164660862/segment_tar_dir/dimSomeTable_1657164660863.tar.gz]]
java.lang.RuntimeException: Caught exception while uploading segments. Push mode: TAR, segment tars: [[file://local/data/logs/s3:/blah/upload_
dir/working_dir_dimSomeTable_OFFLINE_1657164660862/segment_tar_dir/dimSomeTable_1657164660863.tar.gz]]
        at org.apache.pinot.segment.local.utils.IngestionUtils.uploadSegment(IngestionUtils.java:207) ~[pinot-all-0.9.2-jar-with-dependencies.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf
1d654cf1]
Jackie-Jiang commented 2 years ago

Yes. This API requires the auth token because it will try to generate and add a segment to the table (access type CREATE)

pjpringle commented 2 years ago

Issue here is that, an api which is provided the token /ingestFromFile, calls another api /segments, and does not propagate the token.

Jackie-Jiang commented 2 years ago

I see. I saw a recent PR #8670 that might solve this issue. @apucher Can you please help verify if the PR can fix the issue?

apucher commented 2 years ago

Hi @pjpringle @Jackie-Jiang currently, pinot uses the controller's client credential to perform file upload. This should work once your've configured the segment fetcher credentials (you need those anyways)