getting service error without explanation when using aggregation service

TomerGoldstein commented 3 months ago

Hi i have enrolled and managed to deploy the aggregation service and it looks ok (i see metrics logs and everything) i do however have some questions:

i have some reports (with and without clear text option) i got a domain file, i took both of them and tried to use them in the local testing tool - everything looked ok- got a good output i have used the non-encrypted output then, i took the reports and the domain file and use them with the deployed aggregation service (of course this time encrypted since the local tool doesnt accept encrypted files) i got the following error (in this example i have sent only 1 report but also when i sent 200 i get the same error but with 200 as count): "result_info": { "return_code": "SUCCESS_WITH_ERRORS", "return_message": "Aggregation job successfully processed but some reports have errors.", "error_summary": { "error_counts": [ { "category": "SERVICE_ERROR", "count": 1, "description": "Internal error occurred during operation." }, { "category": "NUM_REPORTS_WITH_ERRORS", "count": 1, "description": "Total number of reports that had an error. These reports were not considered in aggregation. See additional error messages for details on specific reasons." } ], "error_messages": [] }, the output report contains all the keys from the domain file but all of the metrics are just pure noise.. also i dont understand this error message, and i think it will be great to get more elaborate error message (since it is empty and doesnt give anymore info) im using the latests 2.4.2 version im really stuck right now can you help?
looking at the auto scailing group for the Aggregation Service i saw that no policy for auto scaling has been created after deploy (im looking in AWS->auto scaling groups -> my service -> automatic scaling or instance manager, however when sending some jobs i did see that the instnace number went up and then down.. what am i missing? do i need to create those policies myself? do you have a recommendations on what metrics to use for auto scaling? including thresholds? thanks!!

TomerGoldstein commented 3 months ago

colsed

TomerGoldstein commented 3 months ago

closed by mistake we have a new error : when sending the request: "result_info": { "return_code": "PRIVACY_BUDGET_AUTHENTICATION_ERROR", "return_message": "com.google.aggregate.adtech.worker.exceptions.AggregationJobProcessException: Aggregation service is not authenticated to call privacy budget service. This could happen due to a misconfiguration during enrollment. Please contact support for resolution. \n com.google.aggregate.adtech.worker.aggregation.concurrent.ConcurrentAggregationProcessor.consumePrivacyBudgetUnits(ConcurrentAggregationProcessor.java:465) \n com.google.aggregate.adtech.worker.aggregation.concurrent.ConcurrentAggregationProcessor.process(ConcurrentAggregationProcessor.java:342) \n com.google.aggregate.adtech.worker.WorkerPullWorkService.run(WorkerPullWorkService.java:143)\nThe root cause is: com.google.scp.operator.cpio.distributedprivacybudgetclient.TransactionEngine$TransactionEngineException: PRIVACY_BUDGET_CLIENT_UNAUTHENTICATED \n com.google.scp.operator.cpio.distributedprivacybudgetclient.TransactionEngineImpl.createTransactionEngineException(TransactionEngineImpl.java:213) \n com.google.scp.operator.cpio.distributedprivacybudgetclient.TransactionEngineImpl.proceedToNextPhase(TransactionEngineImpl.java:68) \n com.google.scp.operator.cpio.distributedprivacybudgetclient.TransactionEngineImpl.executeDistributedPhase(TransactionEngineImpl.java:206)", "error_summary": { "error_counts": [], "error_messages": [] },

we are using 2.5.0 version (newest one)
we are using the new ARN you have sent us during our meeting (which seems to solve the SERVICE_ERROR issue)
we have sent you the relevant files (main.tf, release_params.auto.tfvars,{env}.auto.tfvars) via mail
we use the new ami 2.5.0 (pre built and copied to our region with our AWS ID as our ami_owner) i do see that you hae a record of this error here but we are not sure what to do with this

pisutpon commented 3 months ago

We got the same issue when we upgraded the aggregation service to the version 2.5. The return code was PRIVACY_BUDGET_AUTHENTICATION_ERROR

yuki-tsuchiya commented 3 months ago

We are experiencing the same issue when we upgraded the aggregation service to version 2.5. If anyone has a solution or workaround for this problem, we would greatly appreciate it.

TomerGoldstein commented 3 months ago

we have downgraded to version 2.4.3 and it works for us @yuki-tsuchiya @pisutpon

nlrussell commented 3 months ago

@TomerGoldstein @yuki-tsuchiya @pisutpon Thanks for making us aware of the issue. We're investigating it and will provide an update when we have one. In the meantime, please continue to use version 2.4.3.

maybellineboon commented 3 months ago

Hi @TomerGoldstein ,

Would you be able to provide the full getJob response or timing of the request.

TomerGoldstein commented 2 months ago

@maybellineboon hey! this is a response that i got when using version 2.5.0:


    "result_info": {
        "return_code": "PRIVACY_BUDGET_AUTHENTICATION_ERROR",
        "return_message": "com.google.aggregate.adtech.worker.exceptions.AggregationJobProcessException: Aggregation service is not authenticated to call privacy budget service. This could happen due to a misconfiguration during enrollment. Please contact support for resolution. \n com.google.aggregate.adtech.worker.aggregation.concurrent.ConcurrentAggregationProcessor.consumePrivacyBudgetUnits(ConcurrentAggregationProcessor.java:465) \n com.google.aggregate.adtech.worker.aggregation.concurrent.ConcurrentAggregationProcessor.process(ConcurrentAggregationProcessor.java:342) \n com.google.aggregate.adtech.worker.WorkerPullWorkService.run(WorkerPullWorkService.java:143)\nThe root cause is: com.google.scp.operator.cpio.distributedprivacybudgetclient.TransactionEngine$TransactionEngineException: PRIVACY_BUDGET_CLIENT_UNAUTHENTICATED \n com.google.scp.operator.cpio.distributedprivacybudgetclient.TransactionEngineImpl.createTransactionEngineException(TransactionEngineImpl.java:213) \n com.google.scp.operator.cpio.distributedprivacybudgetclient.TransactionEngineImpl.proceedToNextPhase(TransactionEngineImpl.java:68) \n com.google.scp.operator.cpio.distributedprivacybudgetclient.TransactionEngineImpl.executeDistributedPhase(TransactionEngineImpl.java:206)",
        "error_summary": {
            "error_counts": [],
            "error_messages": []
        },
        "finished_at": "2024-06-12T17:18:07.491437708Z"
    }

please let me know if i can do anything else to help :)

also this is a response for the same exact inputs (chang the name of directories but these are the same files):

    "result_info": {
        "return_code": "PRIVACY_BUDGET_AUTHENTICATION_ERROR",
        "return_message": "com.google.aggregate.adtech.worker.exceptions.AggregationJobProcessException: Aggregation service is not authenticated to call privacy budget service. This could happen due to a misconfiguration during enrollment. Please contact support for resolution. \n com.google.aggregate.adtech.worker.aggregation.concurrent.ConcurrentAggregationProcessor.consumePrivacyBudgetUnits(ConcurrentAggregationProcessor.java:465) \n com.google.aggregate.adtech.worker.aggregation.concurrent.ConcurrentAggregationProcessor.process(ConcurrentAggregationProcessor.java:342) \n com.google.aggregate.adtech.worker.WorkerPullWorkService.run(WorkerPullWorkService.java:143)\nThe root cause is: com.google.scp.operator.cpio.distributedprivacybudgetclient.TransactionEngine$TransactionEngineException: PRIVACY_BUDGET_CLIENT_UNAUTHENTICATED \n com.google.scp.operator.cpio.distributedprivacybudgetclient.TransactionEngineImpl.createTransactionEngineException(TransactionEngineImpl.java:213) \n com.google.scp.operator.cpio.distributedprivacybudgetclient.TransactionEngineImpl.proceedToNextPhase(TransactionEngineImpl.java:68) \n com.google.scp.operator.cpio.distributedprivacybudgetclient.TransactionEngineImpl.executeDistributedPhase(TransactionEngineImpl.java:206)",
        "error_summary": {
            "error_counts": [],
            "error_messages": []
}

maybellineboon commented 2 months ago

Hi @TomerGoldstein ,

Thank you for sharing the information. We are working on the fix and will update everyone once the fix has been rolled out.

nlrussell commented 2 months ago

Hi all, a fix for the issue with Aggregation Service 2.5.0 has been identified, and we expect it to be rolled out by Monday, June 24. Please refer to the Public Status Dashboard for real-time updates: https://status.privacysandbox.com/

nlrussell commented 2 months ago

Hello, we have resolved the issue with Aggregation Service version 2.5.0. All users can now upgrade to 2.5.0 following the instructions for AWS or GCP. Thank you for your patience and continued support.

privacysandbox / aggregation-service

getting service error without explanation when using aggregation service #59