Show styles from userstyles.org in the Stylus popup, with options to preview & install

[derv82]

• Browser: Firefox Quantum
• Operating System: Windows 10
• Screenshot: N/A

I think Stylish on Chrome has this feature. You click the "S" icon on the browser toolbar, and the popup shows styles available from userstyles.org that you can install immediately.

It would be really cool to show the search results in the popup (title/author/screenshot thumbnail), and give users the option to preview on the page or install permanently.

One of the problems with using Stylus/Stylish is the grueling process of finding the right style. For me, the process is:

1. Click {S}
2. Click Find more styles for this site (opens userstyles.org search results)
3. Look at list of styles on userstyles.org
4. Click one of the styles that looks OK, usually based purely on the screenshot (opens userstyles.org page for the style)
5. Click "Install style" & confirm
6. Go back to the page and see how it looks
7. If it doesn't look good / work, uninstall the style and start over from Step 3

---

I looked around and it seems like the userstyles.org site has an API:

• JSON Search results (for github): https://userstyles.org/api/v1/styles/search?search=github&page=1 (includes each style's id)
• JSON Style info (for id: 37035): https://userstyles.org/api/v1/styles/37035 (I assume this includes all information needed to install/preview the style, including CSS, title, author, etc).

[tophf]

I thought about it. If we implement it, the embedded list will appear only after clicking "find styles" action, not automatically.

[derv82]

I'm happy to implement this in my free time, but I need a quick pointer for where in the code base the change should be included (haven't actually looked yet).

---

Including some notes I took regarding the JSON APIs:

---

Sample SearchJSON: https://userstyles.org/api/v1/styles/search?search=reddit.com&page=1 Sample FullJSON: https://userstyles.org/api/v1/styles/70271

Identical in both SearchJSON & FullJSON:

• id (int)
• name (string)
• description (string)
• updated (string)
• weekly_install_count (int)
• total_install_count (int)
• user.id (int)
• user.name (string)
• rating (int)

Missing from SearchJSON:

• css (string)
• created (string)
• screenshots (list)
• additional_info (string)
• discussions (list)
• userjs_url (string)

Missing from FullJSON:

• url (string) -- path to userstyles.org page

Differences:

• after_screenshot_name (FullJSON string) vs. screenshot_url (SearchJSON string)

Screenshots

Screenshots are usually in the format: /^[0-9]+_after.png$/

Path to screenshots:

• Full size: https://userstyles.org/style_screenshots/118959_after.png
• Thumbnail: https://userstyles.org/style_screenshot_thumbnails/118959_after.png

---

Enhancing Search Results

Right now I'm looking at the search API to see if we can specify "subcategory" aka the domain (via styles_controller.rb which appears to be -- or was -- the search code for the site).

I already found per_page which can be set to 200: request for per_page=1

• per_page=200 + filtering irrelevant subcategory entries (i.e. all of the unrelated Google/Youtube/Etc results) would make the search results more relevant.

[tophf]

Our code doesn't have a proper architecture, there are lots of interconnected things, so you'll have to study it in order to be able to extend it. And I doubt it'll be easy. At this point I can only give some trivial pointers: you need to put the code in a separate file in popup/ directory and load it dynamically from popup.js when find-styles is clicked. I'd argue we should add an option to restore the old behavior or maybe there should be another action link or icon in the popup that starts the embedded browser. I also think there's no need to cache the fetched JSONs locally (in chrome.storage) for more than a couple of minutes.

[derv82]

Forked and made some changes: https://github.com/derv82/stylus/commits/master

GIF

What I've done:

• I figured out how to fetch search results and display them in the popup.html
• Load Styles link (above "Find styles for this site) shows search results.
  • Fetches 3 results at-a-time
  • Next/Prev navigation through search results
• The results show:
  • The screenshot thumbnail,
  • The style title (link to userstyles.org style page),
  • The author (link to userstyles.org user page),
  • One-line truncated description (title/hover text is the full description sans HTML)

TODO:

• There's an Install button, but it does not do anything.
  • It does fetch a JSON object representing the style (title, description, css), and logs/alerts the info.
  • I looked at content/install.js and a few other places, but I can't see an easy way to "install a style" using the existing code base
• Search results are not cached (contrary to what you mentioned). So results have to be re-fetched every time the {S} button is clicked.

I think the remaining work regarding installation & caching is a bit too complicated for me to figure out. Unless you have some easy-to-use pointers to an "installing" & "caching" library, I think I've hit a stopping point.

[tophf]

Installing is simply a call to saveStyleSafe

[narcolepticinsomniac]

Looks to me like you're on Github and it's returning results for Google. I realize that their shitty search does that, but should we choose to implement this, would ours have to also suck?

[derv82]

@tophf Thanks, I'll look at saveStyleSafe and see what I can come up with.

---

@narcolepticinsomniac Search would improve if there is a way to filter by subcategory in the search API. This might be possible but I haven't confirmed yet -- see Enhancing Search Results above.

Another (expensive and complicated) option is to parse and filter each result as it comes in:

1. Fetch the full style,
2. Look at @-moz-document to find URLs the style applies to,
3. Remove results that are irrelevant to the current page.

Some benefits to pre-loading all info (including CSS) for each search result is:

1. Installing is instantaneous,
2. Implementing a "Preview" option (e.g. when hovering a result) shouldn't be that hard.

[tophf]

Another (expensive and complicated) option is to parse and filter each result as it comes in

It's expensive only bandwidth-wise. If you fetch style JSONs there's no need to parse the entire code, you can simply use BG.getApplicableSections function to check the applies-to values.

[tophf]

Can you open a PR? It'd be more convenient to discuss the code-related stuff. One thing I'd like to mention right off the bat, our existing code uses global functions, but for the new features it makes sense to enclose everything in an object (all stuff is exposed, not a problem though) or IIFE to keep things encapsulated, only exposing a few methods to the outside via the standard return {method1, method2}.

[derv82]

I agree with encapsulating my changes.

See PR: https://github.com/openstyles/stylus/pull/251

Aside: I'm new to contributing to open source projects, so I apologize in advance if I'm not following the proper Contributing/PR etiquette. Let me know if I can make things easier.

[derv82]

Found more-relevant search endpoint which searches by subcategory when given:

/api/v1/styles/subcategory?search=<pagename>&page=1&country=NA

Where <pagename> is:

• If domain (not subdomain) is one of the 8,000+ "websites" listed here
  • Exclude the subdomain (no www.) and excluding TLD (no .com, .co.uk)
  • E.g. google instead of www.google.com
  • Regex to extract the "site name": /^(.*)\..*$/
• Otherwise:
  • Include TLD
  • E.g. somewebsite.org

Caveats

userstyles.org is in the list, but:

• userstyles.org: https://userstyles.org/api/v1/styles/subcategory?search=userstyles.org&page=1&country=NA
  • 391 results, all (?) relevant to userstyles.org
• userstyles: https://userstyles.org/api/v1/styles/subcategory?search=userstyles&page=1&country=NA
  • 4,290 results, mostly irrelevant to userstyles.org (i.e. themes that mention userstyles.org in the description)

Including the TLD can hide some results:

• washingtonpost: https://userstyles.org/api/v1/styles/subcategory?search=washingtonpost&page=1&country=NA
  • 15 results, all relevant
• washingtonpost.com: https://userstyles.org/api/v1/styles/subcategory?search=washingtonpost.com&page=1&country=NA
  • 6 results, all relevant

---

I'll write a script that scrapes these "sites" and generates a regex that can be used within search-results.js; that way we search using the appropriate search term (e.g. github.com -> github).

[tophf]

It's probable their API or whatever has a bug with a couple of terms like userstyles, for everything else you can probably just extract the site name without proper stripping of TLD from the Public Suffix List. But checking with a scraper is a good idea anyway.

[derv82]

My initial hope was to make the Stylus' search results more-relevant than userstyles.org. E.g. searching for userstyles.org would only show styles that apply to URLs containing userstyles.org.

But it (userstyle's API, database, search method) is much worse than I thought. Userstyles apparently has a 💩 subcategory naming scheme.

Going to /styles/browse/http://ex.ua redirects to /styles/browse/ex. Which is fine since all the styles listed in the ex category are for ex.au. :+1:

But going to /styles/browse/http://ex.com redirects to /styles/browse/ex. This is bad because all styles listed are for ex.ua. :-1:

So a subcategory (e.g. google or ex or any of the 8,000+ "sites") applies to all variations of that name + any TLD. E.g. /styles/browse/http://ex.derv82 redirects to /browse/ex because... ugh.

At this point, I may have to use /styles/browse/all/<encodedURL> to discover the category (via a 302 redirect) instead of /styles/browse/....

For example, /styles/browse/all/https://www.theregister.co.uk/ redirects to /styles/browse?category=theregister (5 results)

• The same results can be found via the API call to /api/v1/styles/subcategory?search=theregister
• The API shows 9 results. 4 of which have a different format of screenshot_url and might explain why they're excluded from the main site's search results.

A similar (but very different URL) /styles/browse/all/https://www.theregister.com.br/ redirects to the same theregister category.

So I guess the secret sauce of category detection is (using example www.theregister.co.uk):

1. Lop off the TLD from the end (hard-coding some list of TLDs like com, net, org, co.uk, .com.br).
   • (Leaving us with www.theregister)
   • If we can't lop off the TLD, force-search for the full domain name (/browse/all/https://www.theregister.faketld/) and stop
2. Grab the "domain" (sans TLD) after the remaining right-most . (e.g. theregister)
3. See if that "domain" is in the list of 8,000 "sites" (theregister is)
4. If so, redirect to that category (category=theregister)
   • Note: userstyles seems to stop here (e.g. searching /browse/all/https://www.notarealwebsite-shouldnothaveacategory.com.br redirects to category=notarealwebsite-shouldnothaveacategory).
5. If not, search using the full domain name (search=theregister.co.uk) -- not using "category=" (which is actually subcategory).

---

I'll write a script that scrapes these "sites" and generates a regex

I'm not wild about a regex containing 8,000+ entries, especially since a lot of them appear to be non-sanitized, non-curated user-input (e.g. forums.xbox.com)url-prefix(http://forums.xbox.com is in the list of 8,000).

Maybe we can just store the top 500 "sites" in a regex (sorted by most-available-styles) , and resort to searching the full domain if the site isn't in that list (i.e. follow through to Step 5 above).

[tophf]

Just launch several searches simultaneously using the above approaches with simple and short regexps. USO is not rocket science.

[tophf]

I've just used a naive regexp in 387193d347d717f1c0f3b635ea1846c28e7cea30, which I think should work for all sites that can be found by USO.

[derv82]

I opted to avoid trying to match USO's regex (which strips some TLDs but not others).

Instead, the latest version of #251 :

1. Makes a HEAD request to USO (/browse/all/<URL>) to get category=... from the redirect.
2. Searches by-category using /styles/browse?category=...

It seems to work well for the sites I've tried it on (userstyles.org, github.com, stackoverflow.com, metacritic.com). The search results only contain relevant styles, we no longer get Google/Youtube results from github.com.

[tophf]

Well, the naive regexp seems to be working for github, but if the HEAD request is fast enough then it's fine too.

[tophf]

IIRC I've tried the HEAD trick when I was working on supporting freestyler.ws search and I remember it added about ~500ms, which I didn't like (no one would), so I switched to a naive regexp in that PR too.

[derv82]

See latest commit to PR: https://github.com/openstyles/stylus/pull/251/commits/8ae669bd120f9e6d25e75ce8a1e5161939f4efdf#diff-b8d58048c3cb67b9add71ac13580c3c0R173

HEAD trick is performed when popup is first loaded. By the time user clicks Find styles for this site to load search results, the HEAD is already finished. This avoids the 500ms delay.

---

TBH I'm pretty happy with my PR:

Please take a look & provide feedback when you can.

[tophf]

Implemented in #251.