Release build 4.0.0 fails - Provide build images and a stable build procedure

[fhoering]

Currently there is a requirement to build each image before being able to upload the image to AWS with terraform:

https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_aws_guide.md#step-12-building-the-amazon-machine-image-ami

if the --with-ami flag is specified, the script will try to build an AMI in AWS .. The script outputs the AMI ID of each service you build. You will need these IDs when deploying the cloud setup,

It seems however that it is not possible anymore to build the currently released image 4.0.0

⚠ Note: This version is not buildable anymore due to a downstream dependency change. A patch for this version will be released shortly.

On our side the build fails with:

ERROR: @com_google_googleurl//build_config:system_icu :: Error loading option @com_google_googleurl//build_config:system_icu: java.io.IOException: Error downloading [https://storage.googleapis.com/quiche-envoy-integration/googleurl_9cdb1f4d1a365ebdbcbf179dadf7f8aa5ee802e7.tar.gz] to /bazel_root/build_ubuntu_b77b7d5/3791edafd24dd71ebf1c8ddd5e6a5a91/external/com_google_googleurl/temp2963993031178500316/googleurl_9cdb1f4d1a365ebdbcbf179dadf7f8aa5ee802e7.tar.gz: GET returned 404 Not Found

Any advice on how to build this image ? (as it is the latest prod release and as this is necessary to be able to deploy with terraform)

Can we build and use an old version (3.11.0) or should we use the most recent version ( 4.1.0 ) ?

I think it would be sensible that each release includes prebuilt AWS images (It looks like Edge is already providing some pre-built images for ad selection API on Azure.

It also seems important to have a stable build process for some time to be able to recompile the code, audit it, verify the checksum and run it locally.

[chatterjee-priyanka]

Hi @fhoering

Apologies that B&A 4.0 is not buildable anymore. All adtechs who built and deployed B&A 4.0 to prod, should not be impacted. All adetchs who are trying to build B&A using non prod images, is recommended to use B&A 4.1.

We are aware of the issue and actively working on releasing [4.1]((https://github.com/privacysandbox/bidding-auction-servers/releases). Currently [4.1]((https://github.com/privacysandbox/bidding-auction-servers/releases) is marked pre-release and any adtech building B&A in test_mode using non-prod hashes should be able to depend on the version.

We are expected to mark [4.1]((https://github.com/privacysandbox/bidding-auction-servers/releases) as the latest B&A release by October 14, 2024 end of day US Pacific time, at that point all adtechs can use B&A in prod mode with prod hashes (with TEE attestation enabled). We will also mark B&A 4.0 has "limited support, non buildable" and give some time to the adtechs who deployed the version in prod to upgrade their deployments to 4.1.

We recommend Criteo to build [4.1]((https://github.com/privacysandbox/bidding-auction-servers/releases) using non-prod hashes in test_mode.

We are working with our teams to send a scaled announcement to all onboarded and engaged adtechs to upgrade to 4.1 as we mark it latest release on Github. We will confirm here as well.

[fhoering]

OK. Thanks. @chatterjee-priyanka We will move forward with version 4.1.

I think it would be sensible that each release includes prebuilt AWS images (It looks like Edge is already providing some pre-built images for ad selection API on Azure.

Do you think in the future you could provide pre-built images ?

It also seems important to have a stable build process for some time to be able to recompile the code, audit it, verify the checksum and run it locally.

Do you think in the future the build pipeline can stay stable for one release ? Not sure exactly what the issue was but I guess most things can be fixed by using stable versions of dependencies (otherwise it would change the build hash anyway)

[dankocoj-google]

Hello Fabian,

While we investigate providing pre-built images, in the meantime you can try out the AWS CodeBuild integration to generate AMIs.

All feedback is welcome, there is still a lot of automation we can add to make the CodeBuild setup smoother.

[chatterjee-priyanka]

Hi @fhoering,

Re B&A release: We have released B&A 4.1 and marked as latest. Adtechs can depend on it for test_mode using non-prod images and deploy to production using prod images.

We will provide limited support for B&A 4.0 till November 14, 2024 for adtechs who already deployed to production. B&A 4.0 will be disallowed in production from Nov 15, 2024. Note the dates mentioned in the release page for 4.0.

Privacy Sandbox support team have communicated to onboarded ad-techs.

Regarding AWS pre-built images: Please follow what @dankocoj-google suggested above and share feedback.

We understand that pre-built images can help adtechs especially in such scenarios. Our teams are working towards that automation. We have a plan to support this in the future and we will keep the ecosystem posted when available.

Regarding stable releases: We aim to support multiple stable releases for a period of time.

This was a one-off issue where the GCS bucket that B&A and KV services depended on was accidentally removed. That being said, in a near future release, B&A and KV are moving away from depending on that GCS bucket altogether to a stable url. In the interim, we provide an assurance that a similar issue won't recur related to the dependency. We are also reviewing all our third party dependencies to ensure they are stable.