[batch] track and bill for network egress

[danking]

What happened?

Bytes egressing from Batch workers to the public internet are not free. We should track every egressing byte and assign the cost of that byte to the job which produced it.

I believe there are two ways to trigger substantial egress:

1. The main container opens a network connection and bytes are sent to the other IP from the container.
2. An output file whose destination is a different cloud provider. I think this is currently impossible due to lack of permissions, but we should either explicitly prohibit this or ensure our solution encompasses it. In particular, I am concerned OpenID could be used to grant permission for a GCP identity to write to S3 or ABS.

Pulling an image shouldn’t trigger substantial egress.

In the first case, there are three kinds of possible egress:

1. Egress to the Public Internet.
2. Egress to a VM in a different Google region.
3. Egress to a Google Service in a different Google region (e.g. uploading to a bucket in a different region).

I believe (2) and (3) are charged equivalently. (1) is simply Internet egress pricing. In (3), I’m not sure who pays the egress from a VM to a bucket in a different region. I assume the VM owner.

In all three cases, the destination’s location matters. For public Internet egress, we can use GeoIP to determine the region of the planet. I’m not sure if we can determine the region of (2) and (3). If we can’t, we should either prevent such traffic or we should charge the maximum egress.

A final caveat is that we use Premium Networking. As a result, our traffic can use Google’s internal backbone. It’s not clear to me if this means that a packet from us-central to a public IP in Australia incurs just Internet egress or that and a region-to-region egress to pay for the use of GCP’s internal global backbone.

The priority of various considerations:

1. Top priority within this issue is to track and recover costs. Even if this means charging a flat fee across all possible kinds of egress. Even if that fee is substantially higher than the real cost to us.
2. Second priority is to surface this information to the user. Simply providing, in the job page, the usage and cost of each resource for this job.
3. Fine grained egress so that users can actually intentionally use it at cost or near cost to, for example, move data between clouds or regions.

Version

0.2.120

Relevant log output

No response

[jigold]

I can write an RFC for how to do this with regards to billing updates and the database. I don't think it's too difficult, but it will take a bit of work to add some new metadata that says whether a resources is by_time or by_unit and compute usage accordingly per billing update. If we are just using the bytes uploaded and downloaded that are tracked by the resource usage monitor, then I think we can do a first pass at adding this functionality. If we have to track by IP address, I don't know how to do that and would have to look into it.

[danking]

On the first sentence, RFC would be great, and yeah, having a by_time and by_unit would be generally useful. It might be nice to eventually charge a fixed per-job fee if pre-job costs begin to dominate for short lived jobs.

---

On the last sentence:

There are two major questions, the first of which is much higher priority. We probably need to do a bit of research, at least on the second question.

1. How can we allow public Internet egress without risking untracked cost? I suspect we must track bytes and charge some, possibly very high, rate.

2. How can we allow public Internet egress at or near the real cost to us?

The second question is complex because Google's egress pricing is complex. To directly respond to your comment: I don't think we need to disaggregate by destination IP address, but we do need to disaggregate by destination "type & location". GeoIP might allow us to do this in iptables, we should figure out what is and isn't possible and how hard it would be.

---

The following is distilled from Network Pricing.

There are six types of egress:

1. VM-to-Internet
2. VM-to-VM or VM-to-Google-Service (which are charged equally)
3. Spanner-to-VM
4. VM-to-Spanner
5. GCS-to-VM
6. VM-to-GCS

Egress types (3) and (5) do not apply to us because hail-vdc does not have Spanner and user jobs cannot read from hail-vdc buckets. Egress types (4) and (6) are slightly ambiguous. We should create a support ticket to verify, but I believe they're charged just like (2). This means we are concerned with just two types of egress:

1. VM-to-Internet
2. VM-to-VM / VM-to-Google-Service

Each type has a different cost table based on the destination location. In these tables, the cheapest price applies, so, for example, for traffic form us-central1-a to us-central1-a the within-zone price applies, not the within-region price.

1. VM-to-Internet. Prices decrease with more usage.

   1. Standard Tier Networking. For the first 10 TiB: 0.085 USD per GiB.

   2. Premium Tier Networking (we are using this currently). For the first 1 TiB:

      Destination	Cost (USD per GiB)
      Anywhere except China, except Australia, but including Hong Kong	0.12
      China except Hong Kong	0.23
      Australia	0.19

2. VM-to-VM or VM-to-Google-Service.

Location Type	Cost (USD per GiB)
Within-Zone	0.00
Within-Region (but different Zones)	0.01
Within-US/Canada	0.01
Within-Europe	0.02
Within-Asia	0.05
Within-South-America	0.08
Within-Oceania	0.08
Across continents except to/from Indonesia and Oceania	0.08
To/from Indonesia or Oceania	0.15

[danking]

And to directly respond to this comment:

If we are just using the bytes uploaded and downloaded that are tracked by the resource usage monitor, then I think we can do a first pass at adding this functionality.

This sounds great! This would resolve question 1 and eliminate the risk. We should charge the highest possible price: 0.23 USD/GiB. Answering question 2 can proceed slowly and carefully knowing that we don't have a cost risk.