search

matomo-org / device-detector

The Universal Device Detection library will parse any User Agent and detect the browser, operating system, device used (desktop, tablet, mobile, tv, cars, console, etc.), brand and model.
http://devicedetector.net
GNU Lesser General Public License v3.0
2.93k stars 474 forks source link

TV UserAgents #3505

Closed anonymous-matomo-user closed 10 years ago

anonymous-matomo-user commented 11 years ago

I would like to see TVs and SetTopBoxes in the piwik stats seperated from "mobile and desktop". So far most of them are just "unknown" in the stats...

What is needed for that? "Only" a list of UserAgents? Keywords: TV

anonymous-matomo-user commented 11 years ago

Attachment: first list TVUAs.txt

sgiehl commented 11 years ago

I think we should at least add those useragents to have them recognized correctly. Do you have a list of useragents?

anonymous-matomo-user commented 11 years ago

Just a few so far, but I would like to request some by the vendors... What shell they provide? Only UserAgent or more Info? OS? Vendor? This is for instance is a "Medion TV": HbbTV/1.1.1 (;Vestel;MB70;1.0;1.0) NetFront/3.5 Not very obvious...

Maybe we could collect info on the currently watched TV channel, too

anonymous-matomo-user commented 11 years ago

for the Plugin-List: If User-Agent contains HbbTV the browser supports it :-) Shame on the people, who made this standard!

See: http://www.etsi.org/deliver/etsi_ts/102700_102799/102796/01.01.01_60/ts_102796v010101p.pdf Page 38, 7.3.2.4 HTTP User-Agent header

HbbTV/1.1.1 (<capabilities>; [<modelName>; [<softwareVersion>]; [<hardwareVersion>]; <reserved>)

mattab commented 11 years ago

If you need this, please get in touch with professional services to sponsor the feature: http://piwik.org/consulting/

WE probably should have a new report that would be a detailed breakdown of all devices used (phones by type, console by type, TV by type, computers, tablets, etc)

anonymous-matomo-user commented 11 years ago

I do not really need this. I am just interested to get it in - and if it takes a year, I am fine with that. I just like to contribute.

If you can point me into the right direction of what all has to be done, I will see, what I can do.

mattab commented 11 years ago

See the master ticket "improvements to devices detection #3612" for more information!

mattab commented 11 years ago

In 6804733ddcb9fcf72b2df3f796a0d5b228bb0ab2: Adding new DevicesDetection plugin, developed by Clearcode and sponsored by a client of Piwik Professional Services. A beautiful work of engineering, all released under GNU/GPL license!

Fixes piwik/piwik#3612

Refs #3505 There is some basic TV detection included and maybe you can help contribute better detections (see the .yml data files containing the regular expressions in YAML format)

HOW DOES IT WORK

This is quite beautiful system. It is a plugin that disabled by default. when enabled, it will create additional columns in the DB. Also at tracking, it will look at the user agent, and try to match it against one browser we know in the databases. The databases of user agent matching are composed by 3 YML files, parsed by spyc.php into php array.

These 3 YML took dozens of hours of work and testing with dozens of mobile phones and devices for accuracy. We are happy with the result as they should cover > 80% of the devices commonly used. We hope the community will help us build up these YML files and make them better, so we can track accurately 90% or 95% of requests.

The performance overhead is pretty small, but parsing the YML files + running dozens of regex will add some overhead. This is why it is still disabled by default. We will think about how to integrate it in core, in the next few months.

Please let me know if you find any problem with this new awesome code!

Solfire commented 11 years ago

@mattab

please add following browser engine for the TV devices: ANTGalio/3.1.1.23.04.09

If you need help for detecting TV - I've tons of log files with user agent strings for televisions. And a lot of regex for TV models are available too.

Best way to identify smart TVs is to check the HbbTV/1.1.1 or HbbTV/1.2.1 sting in the UA

mattab commented 11 years ago

@diwel we would be very interested in your user agent list of strings for TV devices.

Could you post list of user agent and the expected detection (see our existing test suite).

We'd like for more TV sets and smart TVs to be detected accurately within Piwik!

Solfire commented 11 years ago

@mattab

Yes I'll make a separate .yml file for TVs! Could take some time.

But the TVs have a lot more interesting information like the HbbTV version or the screen size in inch or the firmware version. And most of the devices have no real browser but a webkit engine. Additional to that we don't have a normal Opera, it's an embedded Opera and in future there will be webkit instead of presto inside...

mattab commented 11 years ago

TV detection would be a nice addition! If you can keep the number of regex to the smallest, it would be nice (for performance)...

Solfire commented 11 years ago

Based on the HbbTV standard for UAs

HbbTV/1.1.1 (<capabilities>; [<modelName>; [<softwareVersion>];[<hardwareVersion>]; <reserved>)

First you have to detect if it's a TV device 


preg_match('/HbbTV\/[1-9]{1}\.[0-9]{1}\.[0-9]{1}/i', $_SERVER["HTTP_USER_AGENT"]);

After that you can continue with the yml file for TVs I am working on. This will work for nearly 98% of all TVs

But if the UA contains 'firetv-firefox-plugin' it is not a TV, it's the FireHbbTV Plugin for Firefox and we have to ignore that - this needs to be count as a normal FF.

The reason why i did't use a complete regex are UAs like the Samsung TVs from 2011

 'HbbTV/1.1.1 (;;;;;) Maple_2011'

If we use a complete regex we will lose about 40% of the devices by getting vendor and model name!

May be it will work in the future. For example Samsung changed a lot since 2011 

 'HbbTV/1.1.1 (;Samsung;SmartTV2012;;;) WebKit'
 'HbbTV/1.1.1 (;Samsung;SmartTV2013;T-MST12DEUC-1111.1;;) WebKit'

I hope others like Toshiba will follow because they put the capabilities in the wrong position

Wrong: 'Opera/9.80 (Linux armv7l ; U; HbbTV/1.1.1 (; TOSHIBA; 46UL863; 19.3.61.14; 3; ) ; ToshibaTP/1.3.0 (+VIDEO_MP4+AUDIO_MPEG+AUDIO_MP4) ; de) Presto/2.6.33 Version/10.60'
Right: 'HbbTV/1.1.1 (+VIDEO_MP4+AUDIO_MPEG+AUDIO_MP4; TOSHIBA; 46UL863; 19.3.61.14; 3; )'

You can see that there are a lot of difficulties.

Did we want to differ between STBs and TVs? That would be another important decision. If we want that, we have to extend the yml structure.

We'll need to extend the browser vendors by ANTGalio, Maple, Netfront, ... to make it work.

To give you a better overview here some UAs

Sony: "Opera/9.80 (Linux armv7l;  HbbTV/1.1.1 (; Sony; KDL32W655A; PKG3.802EUA; 2013;); ) Presto/2.12.362 Version/12.11"
LG: "Mozilla/5.0 (DirectFB; Linux armv7l) AppleWebKit/534.26+ (KHTML, like Gecko) Version/5.0 Safari/534.26+ HbbTV/1.1.1 ( ;LGE ;NetCast 3.0 ;1.0 ;1.0M ;)"
Grundig: "Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.1 (KHTML, like Gecko) HbbTV/1.1.1 (+PVR;Mstar;OWB;;;)"
Humax: "HbbTV/1.1.1 (+PVR;Humax;iCord Cable;1.01.09;1.0)CE-HTML/1.0 ANTGalio/3.1.1.23.04.09"
Panasonic: "HbbTV/1.1.1 (;Panasonic;VIERA 2012;1.261;0071-3403 2000-0000;)"
Philips: "Opera/9.80 (Linux mips; HbbTV/1.2.1 (; Philips; ; ; ; ) CE-HTML/1.0 NETTV/4.2.0 PHILIPSTV/1.1.1 Firmware/171.56.0 (PhilipsTV, 1.1.1,) en) Presto/2.12.362 Version/12.11"
Technisat: "Opera/9.80 (Linux sh4; U; HbbTV/1.1.1 (;;;;;); CE-HTML; TechniSat Digit ISIO S; de) Presto/2.9.167 Version/11.50"
Inverto: "Mozilla/5.0 (;;;) AppleWebKit/534.6 HbbTV/1.1.1 (+DL+PVR; inverto; IDL-6651N Volksbox Web Edition; 1.0; 1.0;) hdplusinteraktiv/1.0 (NETRANGEMMH;)  CE-HTML/1.0"
TechnoTrend: "Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.4+ (KHTML, like Gecko) Version/5.0 Safari/535.4+ HbbTV/1.1.1 (+DL+PVR;;;;;) CE-HTML/1.0 hdplusinteraktiv/1.0 hdplusSmartTV/1.0 (NETRANGEMMH;) TechnoTrend Goerler/Kathrein"
Lwe: "Opera/9.80 (Linux sh4; U; HbbTV/1.1.1 (+PVR; Loewe; SL150; LOH/3.10;;) CE-HTML/1.0 Config(L:deu,CC:DEU); en) Presto/2.10.250 Version/11.60"
mattab commented 11 years ago

Replying to diwel:

Based on the HbbTV standard for UAs

Are there also other TV sets we could easily detect?

After that you can continue with the yml file for TVs I am working on. This will work for nearly 98% of all TVs

Nice.

Did we want to differ between STBs and TVs? That would be another important decision. If we want that, we have to extend the yml structure.

let's not differ between STB and TVs and consider them as TVs for now.

Thank you for your input, looking forward to your Pull request maybe (ideally you may also add/update the list of user agents that we test for)

Solfire commented 10 years ago

Replying to matt:

Thank you for your input, looking forward to your Pull request maybe (ideally you may also add/update the list of user agents that we test for)

Plugin updated and testet by myself. Pull request to the master is done. List for the tests was updated too.

Happy testing :)

sgiehl commented 10 years ago

In 121e1510fba1eab30f9ea87b0f71d889214b8f7f: refs #3505 (see PR #155) added detection of televisions added detection of 'LG Browser' and 'Espial TV Browser' added a bunch of test cases

sgiehl commented 10 years ago

In a9641b6bb0679eebd27576489bf88c1792c1faa9: Merge pull request #213 from piwik/devicedetectionimprovements

refs #4215 #3505 Improvements to DevicesDetection plugin

sgiehl commented 10 years ago

There are a lot of tvs detected by DeviceDetector. If there are still some missing, please create new issues for them.