FetchAndJoinCustomAudienceRequest fails when getting JSON object from the server

guybashan commented 10 months ago

Hi, It seems like the FetchAndJoinCustomAudienceRequest fails when getting the JSON object from the server. I tried 2 different responses: one it taken from the original docs of Google and one is taken from one of the posts in here. Both are failing for 2 different reasons.

I can clearly see that the request is actually sent to my mock server (PostMan).

Here is the code for the custom fetch and join:

                    FetchAndJoinCustomAudienceRequest request =
                            new FetchAndJoinCustomAudienceRequest.Builder(
                                    Uri.parse(urlFetchAndJoin)
                            )
                                    .setName(name)
                                    .setFetchUri(Uri.parse(urlFetchAndJoin))
                                    .setActivationTime(activationTime)
                                    .setExpirationTime(expirationTime)
                                    .build();

                    Log.i(TAG, "Executing Fetch and Join request");
                    customAudienceManager.fetchAndJoinCustomAudience(
                            request,
                            executor,
                            new OutcomeReceiver<Object, Exception>() {
                                @Override
                                public void onResult(Object ignoredResult) {
                                    Log.i(TAG, "Fetch and join executed successfully");
                                    completer.set(null);
                                }

                                @Override
                                public void onError(Exception error) {
                                    Log.e(TAG, "Failed while doing fetch and join", error);
                                    completer.setException(error);
                                }
                            });

My first response looks like this:

{
    "name": "running-shoes",
    "activationTimestamp": 1680603133000,
    "expirationTimestamp": 1683281533000,
    "user_bidding_signals": {
        "signal1": "value"
    },
    "trusted_bidding_data": {
        "trusted_bidding_uri": "https://example-dsp.com/..",
        "trusted_bidding_keys": [
            "k1",
            "k2"
        ]
    },
    "bidding_logic_uri": "https://example-dsp.com/...",
    "ads": [
        {
            "render_uri": "https://example-dsp.com/...",
            "metadata": {},
            "ad_filters": {
                "frequency_cap": {
                    "win": [
                        {
                            "ad_counter_key": "key1",
                            "max_count": 2,
                            "interval_in_seconds": 60
                        }
                    ],
                    "view": [
                        {
                            "ad_counter_key": "key2",
                            "max_count": 10,
                            "interval_in_seconds": 3600
                        }
                    ]
                },
                "app_install": {
                    "package_names": [
                        "package.name.one",
                        "package.name.two"
                    ]
                }
            }
        }
    ]
}

And this is the exception I am getting:

java.io.InvalidObjectException: The service received an invalid object from the server.
                                                                                                        at android.adservices.common.AdServicesStatusUtils.asException(AdServicesStatusUtils.java:201)
                                                                                                        at android.adservices.common.AdServicesStatusUtils.asException(AdServicesStatusUtils.java:210)
                                                                                                        at android.adservices.customaudience.CustomAudienceManager$2.lambda$onFailure$1(CustomAudienceManager.java:255)
                                                                                                        at android.adservices.customaudience.CustomAudienceManager$2$$ExternalSyntheticLambda1.run(Unknown Source:4)

My second response looks like this:

{
    "name": "ca_397",
    "activation_time": 1667499274000,
    "expiration_time": 1670077674000,
    "ads": [
        {
            "metadata": {
                "valid": 1
            },
            "render_uri": "https://example.com/fetch_and_join_ad1"
        },
        {
            "metadata": {
                "valid": 2
            },
            "render_uri": "https://example.com/fetch_and_join_ad2"
        }
    ],
    "bidding_logic_uri": "https://example.com/b/ca_397",
    "daily_update_uri": "https://example.com/du",
    "trusted_bidding_data": {
        "trusted_bidding_keys": [
            "key1",
            "key2"
        ],
        "trusted_bidding_uri": "https://example.com/t"
    },
    "user_bidding_signals": {
        "arbitrary": "yes",
        "valid": true
    }
}

And this is the exception I am getting:

java.lang.IllegalArgumentException
                                                                                                        at android.adservices.common.AdServicesStatusUtils.asException(AdServicesStatusUtils.java:179)
                                                                                                        at android.adservices.common.AdServicesStatusUtils.asException(AdServicesStatusUtils.java:210)
                                                                                                        at android.adservices.customaudience.CustomAudienceManager$2.lambda$onFailure$1(CustomAudienceManager.java:255)
                                                                                                        at android.adservices.customaudience.CustomAudienceManager$2$$ExternalSyntheticLambda1.run(Unknown Source:4)

Can anyone help me to figure out what I am missing?

cshmerling commented 10 months ago

My first hunch is that it is failing because both the initial call to the fetch API and the JSON response both include the activation and expiration time. The fetch flow only allows the JSON response to set the activation and expiration time if the initial API call omits them.

Can you try again by dropping those fields from the call or the response?

guybashan commented 10 months ago

Hi @cshmerling thanks for you response. It tried it and it still fails.

I think it might be related to the fact that the call actually makes a request to each URL in the JSON payload that I am creating. But this is only a mock response with no "real" urls.

The question is: where can I find an example of a qualified response for each of these URLS (trusted_bidding_uri, bidding_logic_uri, render_uri)?

cshmerling commented 10 months ago

In case this response doesn't clear the issue up, here is a command that will enable more verbose logging:

adb shell setprop log.tag.adservices.fledge VERBOSE

You can use that and filter logcat by adservices and there should be some logs preceding any exceptions that might help point you to the root cause.

One thing to clarify, the platform does not make a request to the various endpoints in the custom audience during join time. Those are only called during the auction (or when the daily update job happens), NOT when the audience is being joined/created.

All that being said- did some testing of my own. It seems the level of validation that happens when creating the custom audience is that the URLs in the custom audience need to match the same top level domain as where the request for the custom audience is going to. So if your test server endpoint is mytestserver.com, the URLs for the various fields need to be mytestserver.com/fetch_and_join_ad1, mytestserver.com/trusted, etc. Using the second example you posted, can you try updating the URLs to match that pattern using your test server address as the base?

guybashan commented 9 months ago

Hi @cshmerling, Thanks a lot for your response.

As for your feedback, I refined the server response to be:

{
  "name": "running-shoes",
  "activationTimestamp": 1667559200000,
  "expirationTimestamp": 1669987200000,
  "user_bidding_signals": {
    "signal1": "value"
  },
  "trusted_bidding_data": {
    "trusted_bidding_uri": "https://f98c7c50-85bc-4b31-bc5b-1af74587edfa.mock.pstmn.io/trusted_bidding",
    "trusted_bidding_keys": ["k1", "k2"]
  },
  "bidding_logic_uri": "https://f98c7c50-85bc-4b31-bc5b-1af74587edfa.mock.pstmn.io/bidding_logic",
  "ads": [
    {
      "render_uri": "https://f98c7c50-85bc-4b31-bc5b-1af74587edfa.mock.pstmn.io/render",
      "metadata": {},
      "ad_filters": {
        "frequency_cap": {
          "win": [
            {
              "ad_counter_key": 1,
              "max_count": 2,
              "interval_in_seconds": 60
            }
          ],
          "view": [
            {
              "ad_counter_key": 2,
              "max_count": 10,
              "interval_in_seconds": 3600
            }
          ]
        },
        "app_install": {
          "package_names": [
            "package.name.one",
            "package.name.two"
          ]
        }
      }
    }
  ]
}

Please note that Google docs are incorrect, since in the examples the value of this property: "ad_counter_key" is String and probably it should be a number.

After doing this update and adding the detailed log, we can see that the general flow looks OK, but it still fails for some reason. We do get a new additional exception that might shed some light. Here is the full log:

2023-11-14 16:17:09.266 11615-11628 adservices.fledge       com.google.android.adservices.api    V  Executing fetchAndJoinCustomAudience.
2023-11-14 16:17:09.267 11615-11639 adservices.fledge       com.google.android.adservices.api    V  fetchCustomAudience is enabled.
2023-11-14 16:17:09.267 11615-11639 adservices.fledge       com.google.android.adservices.api    V  In fetchCustomAudience filterAndValidateRequest
2023-11-14 16:17:09.268 11615-11639 adservices.fledge       com.google.android.adservices.api    V  Validating caller package name.
2023-11-14 16:17:09.268 11615-11639 adservices.fledge       com.google.android.adservices.api    V  Asserting package name "com.youappi.dsp.protectedaudience" is valid for uid 10184
2023-11-14 16:17:09.268 11615-11639 adservices.fledge       com.google.android.adservices.api    V  Candidate package name "com.youappi.dsp.protectedaudience"
2023-11-14 16:17:09.268 11615-11639 adservices.fledge       com.google.android.adservices.api    V  Validating API is not throttled.
2023-11-14 16:17:09.268 11615-11639 adservices.fledge       com.google.android.adservices.api    V  Checking if API is throttled for package: com.youappi.dsp.protectedaudience 
2023-11-14 16:17:09.268 11615-11639 adservices.fledge       com.google.android.adservices.api    V  Checking caller is in foreground.
2023-11-14 16:17:09.271 11615-11639 adservices.fledge       com.google.android.adservices.api    V  Using URI host as ad tech's identifier.
2023-11-14 16:17:09.271 11615-11639 adservices.fledge       com.google.android.adservices.api    V  Validating caller package is in allow list.
2023-11-14 16:17:09.271 11615-11639 adservices.fledge       com.google.android.adservices.api    V  Validating per-app user consent.
2023-11-14 16:17:09.272 11615-11639 adservices.fledge       com.google.android.adservices.api    D  Validating the CustomAudienceBlob's name.
2023-11-14 16:17:09.272 11615-11639 adservices.fledge       com.google.android.adservices.api    D  Validating the CustomAudienceBlob's activation time.
2023-11-14 16:17:09.272 11615-11639 adservices.fledge       com.google.android.adservices.api    D  Validating the CustomAudienceBlob's expiration time with calculated activation time as 2023-11-14T14:17:09.266Z.
2023-11-14 16:17:09.272 11615-11639 adservices.fledge       com.google.android.adservices.api    D  The CustomAudienceBlob has a valid buyer: f98c7c50-85bc-4b31-bc5b-1af74587edfa.mock.pstmn.io.
2023-11-14 16:17:09.273 11615-11639 adservices.fledge       com.google.android.adservices.api    V  Completed fetchCustomAudience filterAndValidateRequest
2023-11-14 16:17:09.273 11615-11639 adservices.fledge       com.google.android.adservices.api    V  In fetchCustomAudience performFetch
2023-11-14 16:17:09.273 11615-11639 adservices.fledge       com.google.android.adservices.api    V  Sending request from fetchCustomAudience performFetch
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  Adding name
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  24137424 Found name in JSON response
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  Adding bidding_logic_uri
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  24137424 Found bidding_logic_uri in JSON response
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  Adding user_bidding_signals
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  Adding trusted_bidding_data
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  24137424 Found trusted_bidding_data in JSON response
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  Adding ads
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  24137424 Found ads in JSON response
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  Adding name
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  162864841 Found name in JSON response
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  Adding bidding_logic_uri
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  162864841 Found bidding_logic_uri in JSON response
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  Adding user_bidding_signals
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  Adding trusted_bidding_data
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  162864841 Found trusted_bidding_data in JSON response
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  Adding ads
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  162864841 Found ads in JSON response
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  Adding owner
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  167086542 Found owner in JSON response
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  Adding buyer
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  167086542 Found buyer in JSON response
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  Adding name
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  167086542 Found name in JSON response
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  Adding activation_time
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  167086542 Found activation_time in JSON response
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  Adding expiration_time
2023-11-14 16:17:09.547 11615-11640 adservices.fledge       com.google.android.adservices.api    V  167086542 Found expiration_time in JSON response
2023-11-14 16:17:09.548 11615-11640 adservices.fledge       com.google.android.adservices.api    D  Error encountered in fetchCustomAudience execution
                                                                                                    java.io.InvalidObjectException: Fused custom audience is incomplete.
                                                                                                        at com.android.adservices.service.customaudience.FetchCustomAudienceImpl.lambda$validateResponse$1(FetchCustomAudienceImpl.java:378)
                                                                                                        at com.android.adservices.service.customaudience.FetchCustomAudienceImpl.$r8$lambda$FassehbCbH4RDs2Nc7fIx7_F_sw(FetchCustomAudienceImpl.java:0)
                                                                                                        at com.android.adservices.service.customaudience.FetchCustomAudienceImpl$$ExternalSyntheticLambda4.call(R8$$SyntheticClass:0)
                                                                                                        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
                                                                                                        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
                                                                                                        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
                                                                                                        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                                                                                                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
                                                                                                        at java.lang.Thread.run(Thread.java:1012)
2023-11-14 16:17:09.548 17599-17632 FetchAndJoin            com.youappi.dsp.protectedaudience    E  Failed while doing fetch and join
                                                                                                    java.io.InvalidObjectException: The service received an invalid object from the server.
                                                                                                        at android.adservices.common.AdServicesStatusUtils.asException(AdServicesStatusUtils.java:201)
                                                                                                        at android.adservices.common.AdServicesStatusUtils.asException(AdServicesStatusUtils.java:210)
                                                                                                        at android.adservices.customaudience.CustomAudienceManager$2.lambda$onFailure$1(CustomAudienceManager.java:255)
                                                                                                        at android.adservices.customaudience.CustomAudienceManager$2$$ExternalSyntheticLambda1.run(Unknown Source:4)
                                                                                                        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                                                                                                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
                                                                                                        at java.lang.Thread.run(Thread.java:1012)

Can you identify what are we missing? Thanks

guybashan commented 9 months ago

Update: as for the latest update of the IDE and devices I simply stopped getting any logs related with "adservices". Everything stopped working for some reason.

cshmerling commented 9 months ago

Appreciate the patience for my response.

I'll need to look a bit more at where the custom audience issue is, you've (un)luckily hit an error message I've not seen before.

In the meantime, I'll double check the documentation RE: ad counter key and see if it needs to be updated.

On the topic of IDE, which version of Android Studio are you using? Stable, Beta, Canary? The only immediate suggestion I would have is make sure logcat is pointing to the proper device, as I've found when I've updated my virtual device, logcat points to the old device that is offline, so I don't see ANY logs. Not sure if this is the case for you or if you are just not seeing any adservices logs specficially.

cshmerling commented 9 months ago

For the issue with the custom audience, it looks like it is missing the daily_update_uri used for the daily background fetch. We've updated the documentation to reflect this and the fact that ad counter keys should be numbers.

i.e. add "daily_update_uri": "https://example-dsp.com/..." to the response from the fetch URL

In the medium term we are going to look at improving logging/errors for cases like this as it should be easier for a user to debug.

guybashan commented 9 months ago

Thanks @cshmerling. now the request finally seems to work properly without an exception. Is there a way of checking things after adding the custom audience?

cshmerling commented 9 months ago

Unfortunately, there is no way to read which custom audiences are on device directly. Your best bet would be to trigger an auction specifying your buyer and seeing the resulting renderUrl from the winning ad. If there is no winning ad, would need to verify your server is being hit when trying to download bidding logic or some other URL

android / privacy-sandbox-samples

FetchAndJoinCustomAudienceRequest fails when getting JSON object from the server #88