Cannot select font by family name with subfamily feature in xelatex

Description

According to the documentation, fontspec should let fonts be specified both by family name and by filename, either on lualatex and xelatex. Apparently, specifying fonts by postscript name also works.

However, in xelatex there is an issue when a font is available in different subfamilies (e.g., bold, italic) in different files, as it is often the case, and you want to specify the font by family and subfamily name.

For instance, the following snippet works in lualatex, but not on xelatex:

setmainfont{Linux Libertine O}[Ligatures=TeX,
  ItalicFont={TeX Gyre Pagella Italic}]

Apparently, there is no way to tell xelatex that you want the "Italic" version of Gyre Pagella if you prefer specifying the font by name. Setting ItalicFont={TeX Gyre Pagella} works but obviously does not give you an italic font.

Interestingly, specifying the font by postscript name as

setmainfont{Linux Libertine O}[Ligatures=TeX,
  ItalicFont={TeXGyrePagella-Italic}]

works. Yet, it seems to me that the more "abstract" the naming the better for compatibility and portability reasons (e.g, when a font is to be instantiated in a class file to be used on different systems and setups).

Check/indicate

[X] Relevant for XeTeX
[ ] Relevant for LuaTeX
[ ] Issue tracker has been searched for similar issues?
[ ] Links to discussion if appropriate

Minimal example demonstrating the issue

See above

Further details

Seen on Linux (kubuntu 19.10 with TexLive 2019 - the upstream one, not the ubuntu packaged one - kept up to date with tlmgr).

latex3 / fontspec