Add attribute to indicate 'bandwidth ceiling' for current connection type

[igrigorik]

navigator.connection.type -> cellular does not provide sufficient information to optimize for different connection types. For example, significant portion of mobile users in India are still (and will continue to be) on 2G networks. Similarly, there is a world of a difference between 2.5G, 3.5G, and 4G.

The ability to distinguish between these connection types is important for analytics (https://github.com/w3c-webmob/netinfo-usecases/issues/65), and content adaptation - e.g. segmenting navigation and purchasing behavior, or serving an alternative experience for users on older generation networks.

One way to address this is to break apart type -> cellular into more fine-grained buckets. However, the downside to this approach is that we then have to agree on what that enum should be, and we'll have to update it once new connection types become available. Instead, I suggest we add a 'max theoretical bandwidth for this connection type' attribute, which exposes a single value in Kbps or Mbps... Which, incidentally, is exactly how FF originally implemented the bandwidth attribute defined in previous version of the spec: http://hg.mozilla.org/mozilla-central/rev/e652d673382f#l2.56

tl;dr:

• Introduce new attribute to expose max theoretical bandwidth for current (last hop) connection type.
• This attribute would expose a more fine-grained estimate than type - i.e. take into account generation of the cellular network, ethernet link-speed, etc.
• This attribute should have consistent units: kbps or mbps.
• This attribute should have a name that clearly indicates that it is not providing an estimate of end-to-end bandwidth, but is indicating the maximum theoretical bandwidth based on last hop connection type - i.e. it should not be called bandwidth. Instead, bandwidthCeiling or some such.

Also, yes, this attribute should be used carefully. It is not an indicator of end-to-end bandwidth, nor should it be used as such, but it is still a very useful signal as it places an upper bound (ceiling) on what the network can deliver, which is particularly useful and important for improving user experience on old(er) generation networks.

[marcoscaceres]

I wonder if we would even need to expose the bandwidthCeiling as this is very regional (hence, it might require look-up tables on a per country or per state-level or per city-level).

For instance, my carrier here in Canada claims it is 3G, but I rarely, if ever, get anything close to 3G speeds. It also doesn't work outside any of the main cities and gets bounced to different carriers - where I get charged "away rates" (see below).

If this is the situation in a "developed" country like Canada, I don't want to imagine how sad the state of affairs is in other countries. My worry is that UAs will have to always report the lowest possible speed to avoid abuse of this feature (if we introduce the fine-grained types).

WIND NETWORK BEST PHONE DISPLAYS: WIND Home, WIND Zone, WIND, or 302490 Good indoor and outdoor coverage where you can use your unlimited features.

VARIABLE NETWORK & COVERAGE PHONE DISPLAYS: Either WIND OR AWAY Good outdoor and satisfactory indoor coverage depending on location factors. In this area it is possible to get WIND or our Partner network so check display to ensure you know if you will be using WIND service or charged Away pay per use rates.

PARTNER NETWORK (AWAY COVERAGE) PHONE DISPLAYS: AWAY (Canada) or Partner Name (i.e. TMobile in USA) Signal strength and network coverage will vary. See WIND plan details for Away pay per use rates.

[igrigorik]

We don't need to get into carrier/region based ceiling discussions - these values change on a minute to minute basis based on network load, location, device in use, etc. However, regardless of all of these variables, the maximum bandwidth for a particular generation of network stays the same - e.g. EDGE will max out at 384kbps. In other words, regardless of how good or bad the current network weather is for any carrier or location, if you're on EDGE, I know you'll never go faster than 384kbps.

[marcoscaceres]

Not particularly representative, but here are my results sitting right in the middle of Toronto on 3G:

Download: 1.52 Mbps Upload: 0.22 Mbps Ping: 73 ms

http://www.speedtest.net/my-result/i/938818231

I'm also reading on Wikipedia that:

ITU has not provided a clear[citation needed][vague] definition of the data rate that users can expect from 3G equipment or providers. Thus users sold 3G service may not be able to point to a standard and say that the rates it specifies are not being met. While stating in commentary that "it is expected that IMT-2000 will provide higher transmission rates: a minimum data rate of 2 Mbit/s for stationary or walking users, and 384 kbit/s in a moving vehicle,"[25] the ITU does not actually clearly specify minimum required rates, nor required average rates, nor what modes[clarification needed] of the interfaces qualify as 3G, so various[vague] data rates are sold as '3G' in the market.

I'm worried that carriers might be annoyed if we start stating that things should be at some speed, but they are not.

Note from the above that my own carried in Canada doesn't even get to 2 Mbits/s on 3G.

It is also noted in Wikipedia that, at least, "In India, 3G is defined by telecom service providers as minimum 2 Mbit/s to maximum 28 Mbit/s".

[igrigorik]

Right, there are no minimums (I wish), but that's not what we're after with 'ceiling'. There are well-defined maximums, which are based on the underlying technology: coding algorithm, number of channels, etc. Hence the reason why I'm stressing that this should be "ceiling" to indicate that this is max possible throughput of last hop - e.g. Gigabit ethernet would return 1000Mbps.

[marcoscaceres]

@igrigorik where would we get the max speed list from? Let's build it up.

[igrigorik]

Original FF implementation is a good starting point: http://hg.mozilla.org/mozilla-central/rev/e652d673382f#l2.68. I'm not sure where @mounirlamouri got those numbers from, but on first pass, they are in the right ballpark.

• http://en.wikipedia.org/wiki/Mobile_broadband#Generations
• http://en.wikipedia.org/wiki/Comparison_of_wireless_data_standards

/cc @dougsillars

[marcoscaceres]

As the theoretical bandwidth can change independently of type, we might need an event. Over on public-script-coord, they currently discussing adding [Observable] to WebIDL. If we had that, we could make both bandwidthCeiling and type observable instead of requiring the event. However, it's looking like it's going to take ages, we can add a change event for bandwidthCeiling.

[dougsillars]

The numbers seem ballpark correct.

Any thoughts on providing a "latency minimum" along with the "Bandwidth Max"? This might be helpful for interactive sites, notifying the site to prefetch more data earlier?

[igrigorik]

@dougsillars which set of numbers? The ones in original FF implementation?

For latency, even if we don't include it in the spec, since we're deriving BW based on last hop type, it would be pretty easy to build a lookup map on your own. In other words, seems like a library?

[dougsillars]

The firefox numbers are close, but I think they tend to be a bit slower than your references and those in android docs. Here are the Android emulator speeds for various GSM networks. Value Description Comments gsm GSM/CSD (Up: 14.4, down: 14.4) hscsd HSCSD (Up: 14.4, down: 43.2) gprs GPRS (Up: 40.0, down: 80.0) edge EDGE/EGPRS (Up: 118.4, down: 236.8) umts UMTS/3G (Up: 128.0, down: 1920.0) hsdpa HSDPA (Up: 348.0, down: 14400.0) full no limit (Up: 0.0, down: 0.0) http://developer.android.com/tools/devices/emulator.html#netspeed

For HSPA+ and LTE we can look at your book :) HSPA+ (22 Mbit/s up 168 Mbit/s down LTE (75 300

i love the idea of a library to help devs create latency measurements. For browsers (and native!)

[mounirlamouri]

I'm not sure where @mounirlamouri got those numbers from, but on first pass, they are in the right ballpark.

Just did a bit of research.

[igrigorik]

Relevant references for Android:

• http://developer.android.com/training/basics/network-ops/managing.html#check-connection
• http://developer.android.com/reference/android/telephony/TelephonyManager.html

int NETWORK_TYPE_1xRTT  Current network is 1xRTT
int NETWORK_TYPE_CDMA   Current network is CDMA: Either IS95A or IS95B
int NETWORK_TYPE_EDGE   Current network is EDGE
int NETWORK_TYPE_EHRPD  Current network is eHRPD
int NETWORK_TYPE_EVDO_0 Current network is EVDO revision 0
int NETWORK_TYPE_EVDO_A Current network is EVDO revision A
int NETWORK_TYPE_EVDO_B Current network is EVDO revision B
int NETWORK_TYPE_GPRS   Current network is GPRS
int NETWORK_TYPE_HSDPA  Current network is HSDPA
int NETWORK_TYPE_HSPA   Current network is HSPA
int NETWORK_TYPE_HSPAP  Current network is HSPA+
int NETWORK_TYPE_HSUPA  Current network is HSUPA
int NETWORK_TYPE_IDEN   Current network is iDen
int NETWORK_TYPE_LTE    Current network is LTE
int NETWORK_TYPE_UMTS   Current network is UMTS

For iOS 7+:

The Core Telephony framework (CoreTelephony.framework) lets you get information about the type of radio technology in use by the device. Apps developed in conjunction with a carrier can also authenticate against a particular subscriber for that carrier... source, example

Added CTTelephonyNetworkInfo.currentRadioAccessTechnology
Added CTRadioAccessTechnologyCDMA1x
Added CTRadioAccessTechnologyCDMAEVDORev0
Added CTRadioAccessTechnologyCDMAEVDORevA
Added CTRadioAccessTechnologyCDMAEVDORevB
Added CTRadioAccessTechnologyDidChangeNotification
Added CTRadioAccessTechnologyEdge
Added CTRadioAccessTechnologyGPRS
Added CTRadioAccessTechnologyHSDPA
Added CTRadioAccessTechnologyHSUPA
Added CTRadioAccessTechnologyLTE
Added CTRadioAccessTechnologyWCDMA
Added CTRadioAccessTechnologyeHRPD

Other references:

• http://chimera.labs.oreilly.com/books/1230000000545/ch07.html#_3gpp_and_3gpp2_partnerships
• http://chimera.labs.oreilly.com/books/1230000000545/ch07.html#_evolution_of_3g_technologies
• http://hg.mozilla.org/mozilla-central/rev/e652d673382f#l2.68

[igrigorik]

First run at intersecting the Android and iOS network types: https://gist.github.com/igrigorik/cb94f9b83f5afb3f722a

Now we just need to fill in the downlink speeds.. easier to iterate in a gist. :)

[paulirish]

The network presets in DevTools emulation: https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit/Source/devtools/front_end/toolbox/OverridesUI.js&q=WebInspector.OverridesUI._networkConditionsPresets&sq=package:chromium&type=cs&l=366

I believe eustas did some research on these. I spoke to a few folks that indicate they may not be too accurate, but we haven't had any definitive data that disproves them.

I mentioned this to @bluesmoon as SOASTA might have some data here. Philip, do you have anything that could help?

[bluesmoon]

Let me have a look at my data.

[igrigorik]

@paulirish @bluesmoon note that we're not looking for estimated BW data based on current carriers, etc. The ceiling value is meant to indicate the maximum theoretical throughput of the connection type, regardless of carrier and all other factors.

[paulirish]

Well, I'm pretty interested in average/expected BW per connection type. Would help me as a dev to set expectations for what each of the ceiling values will map to.

[igrigorik]

@paulirish as an FYI, sure... I'm just saying that the intent of bandwidthCeiling is to expose the theoretical maximum, not some expected average - these values are highly variable and change with time. If / once / when we get to surfacing bandwidthEstimate, or some such, then these averages and end-to-end BW estimates start to become more relevant.

[paulirish]

​Sure, but I guess I'm saying I don't know how useful the bandwidthCeiling metric is to me if, say for instance, the Estimate values are actually 80-90% smaller.

This API is going to expose decision points for developers on what sort of assets to send down, so I want to use the Ceiling metric to pick something reasonable. And the gulf between Ceiling and Estimate is the difference between unreasonable and reasonable. :)

[bluesmoon]

I'd have to agree with @paulirish, the ceiling is rather useless from a web dev's point of view, just as useless as knowing the connection type.

[paulirish]

I definitely want the granularity in this API that Ilya has proposed. And I'm OK with ceiling being how we get there. (Definitely seems to be the best route to shipping this)

But when that API ships, I would be doing this estimate research work on my own understand how to use it. :)

[igrigorik]

@bluesmoon it's not useless, it's a cap - e.g. knowing that your cap is 384kbps (EDGE) can be incredibly useful. First, we know you won't ever exceed that limit, and second, it also allows you to estimate the RTT range... Once you get into higher G's, the BW range gets wider, and last mile latency range gets narrower. Yes, these are coarse signals, but they are still important, especially for delivering a better user experience for users on older generation networks, of which there are many... Something we often forget because most of us are walking around with latest generation devices that are connected to relatively fast networks (3.75G+).

@paulirish I do hope that, one day, we'll also expose bandwidthEstimate attribute, or some such, that would provide a situational end-to-end BW estimate.. However, that's a tricky area, and having the ceiling value does not conflict with this goal. If anything, if you want to rely on averages, you can map the ceiling values to some values that you determine are (more) reasonable for your location, etc.

[marcoscaceres]

I've started adding bandwidthCeiling based on what we've discussed so far (see bandwidthCeiling branch). I've tied it to "Mbps". I've defined a "underlying connection type" concept - which I will link to the table of maximums once we have it.

[marcoscaceres]

(we should bikeshed the bandwidthCeiling name later ... making people type addEventListener("bandwidthceilingchange") will not make us any friends :smile: )

[igrigorik]

@marcoscaceres ceil is a common API name... Perhaps bandwidthceil, bwceil? The latter is nice and short, but kinda cryptic.

[marcoscaceres]

@igrigorik, agree. Let's stick with bandwidthCeiling at least for now.

[igrigorik]

@marcoscaceres took a run at mapping different technologies to BW numbers... In the process, realized that some of the previous numbers in FF implementation were off: some types were mapped to wrong generations and some generations had wrong bitrates - bits vs bytes mixups, woops...

In https://github.com/w3c/netinfo/commit/19ec1af943767dbe78dd9b7b6fcf467267cd79b6#commitcomment-7498128, would it make sense to reference the suggested numbers? I want to make sure we end up reporting the same values across different UAs.

/cc @dougsillars can you do another quick sanity check on above spreadsheet?

[marcoscaceres]

@igrigorik looking good! thanks for doing the research. I'll start putting in the infrastructure into the spec to capture this once we triple check it.

I'll let Moz people know that their values are off. We should expect that we will make similar mistakes, so we should think of some system to reduce that risk. The citations you have are great - and most seem fairly authoritative.

In the spec, your ok for us to use kbits/s?

[marcoscaceres]

We need to decide what we are going to do for the other connection types.

If we add this data for "cellular", we probably need to do something for "wifi", "bluetooth", "ethernet", etc.

"other" is left to the implementation, with +Infinity as the default. Note that other + bw ceiling might form a fingerprint, so need to make a note there.

"unknown" will probably just return +Infinity.

[igrigorik]

I'm sure there is a lot of room to bikeshed the exact ceiling numbers. That said, the more important part to me is that I can map a particular value to a type, and given the type I can then add a layer of additional smarts - e.g. combine some Geo + carrier knowledge to get a localized expected peak, or some average value (as @paulirish wanted). On that note, one thing that bugs me a bit at the moment is that HSUPA/HSDPA return same BW values -- which is true for download, but HSUPA offers a significant bump in uplink speed, which may be useful to know. Wondering if we could/should offset those a bit to make the distinction clear.

In terms of getting feedback, fixing errors, and adding new values in the future: what if we convert it into a standalone JSON struct, place it in the same repo, and reference it in the spec? That would make it easy to consume by external parties, with revision history, and so on. WDYT?

We need to decide what we are going to do for the other connection types.

• wifi: http://en.wikipedia.org/wiki/IEEE_802.11#Protocol - take the max in data rate column.
• bluetooth: http://en.wikipedia.org/wiki/Bluetooth#Uses - see table on the right.
• ethernet: just map to link speed? I.e. 10, 100, 1000, 1000Mbps.
• other + unknown: nil? Infinity always seemed a bit too optimistic? :)

In the spec, your ok for us to use kbits/s?

I can be convinced either way, but 1000Mbps for Ethernet would result in a lot of trailing zeros. Perhaps Mbps is better? As a crazy thought, even though link speeds are typically reported in bits.. My experience shows that most developers (myself included) often miss this and think its bytes - e.g. 384kbps for EDGE is 48kB/s! What if we reported in MBps?

[dougsillars]

@igrigorik Are you proposing a a BandwidthUploadCeiling? :)

The numbers in the spreadsheet look accurate, and have great references (saving the link for future use).

[bmaurer]

Just wanted to chime in here and say that at Facebook, we already use this type of data in our native clients. http://techcrunch.com/2014/08/27/facebook-turns-on-bandwidth-targeting-to-match-mobile-ads-to-network-quality/ is an example of how this can be useful to help serve ads to the right audience -- you wouldn't want to serve a video ad to a 2G user.

To be honest, I'm not sure how we'd use the bandwidth ceiling information. I assume that the enumeration of possible connection types would not change much over time, so I think we might end up having our own map of connection type -> quality. That said, I think it's a good thing to make sure that browsers can add new connection types and clients that don't know about that connection type can parse them in some reasonable way.

[igrigorik]

@dougsillars I really hope we can avoid that :)

@bmaurer thanks, great use case, and from what I understand the current announcement does not apply to web, just native - right? It'd be great to enable this for the web.

Re, how to use: all browsers should report same ceiling values for each underlying type (e.g. 384kbps for 2G EDGE), which you can then use to do a reverse-map to a type, or use the value directly (e.g. if BW ceiling > 1mbps, run a video ad). The benefit of reporting a numeric value, instead of an enum, is that the API continues to work in the future when/if we add new connection types, and also across different technologies... Also, if you have additional data, like Geo-specific info on how a certain connection type behaves in a particular geo region, or some such, you can easily add that on top of this API.

[marcoscaceres]

@igrigorik so, we can do the JSON thing and keep the spec and JSON synchronized (i.e., we generate the table in the spec from the JSON file).

[marcoscaceres]

So, for the JSON, going with the following (hopefully calling it max-megabytes will help remind people to use MiB/s). We can, of course, bikeshed later - Any obvious problems with this structure?

{
  "cellular": [
    {"version": "", "max-megabytes":  0 }
  ],
  "bluetooth": [
    {"version": "", "max-megabytes":  0}
  ],
  "ethernet": [
    {"version": "", "max-megabytes":  0}
  ],
  "wifi": [
    {"version": "", "max-megabytes":  0}
  ]
}

[igrigorik]

@marcoscaceres on first pass, LGTM. I'll put together a pull request with the right data.

[marcoscaceres]

Ok, started adding stuff: https://github.com/w3c/netinfo/commit/89947032cbe7450f5464c82f8c93165ed70c58ec

• fixed mixed content issue
• added task source
• started adding dependencies
• switched to using megabytes
• switched megabytes to unrestricted double (for +Infinity)
• defined bandwidthCeiling and event
• added "steps for getting the bandwidth ceiling"
• defined more clearly "Underlying connection types"
• added empty "Table of maximum theoretical bandwidths"
• added "maxbws.json" file

[igrigorik]

GitHub didn't pick up the update, manual ping: see pull https://github.com/w3c/netinfo/pull/16.

There are a few nomenclature changes from what was discussed above (e.g. bandwidthCeiling is now downlinkMax) but the general idea and approach is the same. Please take a look!

[sicking]

I'd prefer to only expose the "bandwidth" for cellular connections. On for example wifi and ethernet connections the actual connection bandwidth is usually irrelevant. Back when the spec contained just .metered and .bandwidth this lead to endless rathole discussions about the evilness of the .bandwidth property.

So I'd prefer to call this new property something like .cellDownlinkMax or .cellBandwidth or something to that effect. And make it return null except when .type === 'cellular.

The only use case that I could see lost there would be getting the bandwidth of a current bluetooth connection. But this seems like a pretty rare edge case.

[jkarlin]

I'm in favor of this as an implementer as it's difficult information to obtain for each connection type. Restricting to cellular is much easier.

On Wed, Nov 12, 2014, 3:54 PM Jonas Sicking notifications@github.com wrote:

I'd prefer to only expose the "bandwidth" for cellular connections. On for example wifi and ethernet connections the actual connection bandwidth is usually irrelevant. Back when the spec contained just .metered and .bandwidth this lead to endless rathole discussions about the evilness of the .bandwidth property.

So I'd prefer to call this new property something like .cellDownlinkMax or .cellBandwidth or something to that effect. And make it return null except when .type === 'cellular.

The only use case that I could see lost there would be getting the bandwidth of a current bluetooth connection. But this seems like a pretty rare edge case.

— Reply to this email directly or view it on GitHub https://github.com/w3c/netinfo/issues/13#issuecomment-62791501.

[igrigorik]

@sicking playing devil's advocate... I want to build an application that does some really intensive transfers (e.g. streaming high resolution 4K video), and try as I might, I can't jam the stream over 802.1b... it would sure be nice to know that upfront, such that I can adapt my logic. Similarly, WiFi has its bad network weather periods also, just because you're on 802.11g doesn't mean you'll get the full pipe.

I don't think we should special case cellular in the spec. We should keep the door open for UAs to improve their estimates and reporting in the future - e.g. if someone plumbs through WiFi signal strength and interface speeds, instead of just mapping to family, then that's great and we should allow for that.

[sicking]

Exposing the bandwidth of the local connection is generally only useful if that local connection is the limiting factor.

If the user is on a wifi network, then it's almost certain that that wifi network is not the limiting factor. The cost of upgrading your wifi router is small enough compared to paying for a high quality internet connection that in practice people that pay for a faster internet connection will also ensure that they are on a wifi router that enables them to use that connection.

If 802.11b isn't fast enough for you, then netinfo likely won't provide you the data that you need. I.e. there's a high degree of risk that even if netinfo indicates that the user is on a fast ethernet or a fast wifi, that the actual internet connection on the other side of the local router will be the limiting factor.

The other situation when the wifi connection is the limiting factor is if you're on a crowded wifi network. But that too is not something that the current proposal will help with since it

I agree that it would be cool if the UA could estimate your actual internet connection speed. Back when the spec had a .bandwidth property we had a bunch of discussions at both W3C and mozilla about this. I think it's a good idea, but requires more research.

So my proposal is that we in the meantime just expose the local link speed. The simplest way to do that seems to be to just expose the cellular bandwidth. And explicitly define it as such rather than create a best-effort UA estimate that may or may not just look ad the connection type. I worry that anything beyond that will go the way of the previous netinfo spec, i.e. getting bogged down by allowing perfect to stand in the way of good.

Once we have experiments that show that we can provide useful bandwidth estimates then I'd be all for exposing that as well.

[igrigorik]

Exposing the bandwidth of the local connection is generally only useful if that local connection is the limiting factor.

Give me any link speed, and I'll give you a case where I can saturate it. downlinkMax is relevant regardless of interface. Introducing cellDownlinkMax would only break APIs down the road.

For example, say we add wifiDownlinkMax, applications would then need to update their code to check for different XDownlinkMax variables based connection type... that's broken. If all I care about is speed, I shouldn't have to check for interface type. As specced, I can simply use downlinkMax and:

• check for +Infinity as a signal that UA has no reasonable estimate - i.e. I'm on my own.
• check for interface type if I have conditional logic - e.g. only fetch large files over WiFi, or some such.

In short, I think we should keep downlinkMax as a consistent attribute, regardless of interface. We don't win anything by adding more specific attributes.

[jkarlin]

Ignore my previous comment, it's just implementer griping. I agree with Ilya that we should have a unified attribute for ease of coding. The more information the merrier.

That said, I think getting at the underlying connection technology for bluetooth/wifi will be challenging on a number of platforms, so expect a lot of +Infinitys.

[marcoscaceres]

I agree with @igrigorik. We should keep downlinkMax as a consistent attribute.

@jkarlin, I think lots of +Infinitys are fine. We can always keep updating implementations and the spec to best serve the people who make use of this information.

[fahadbaig123]

why not allow the data usage ceiling as a device level setting that the end user can setup.. I believe bandwidth is just one side of the coin the real issue lies in the data plan that the consumer has bought.. the control should be with the consumer in deciding how much data plan does he or she want the brands to use..