Investigating client-side search

[mbland]

Something along the lines of:

https://github.com/blog/1939-how-github-uses-github-to-document-github

[afeld]

https://github.com/PascalW/jekyll_indextank may be useful.

@jglovier @benbalter @gjtorikian We're looking to replicate how you implemented search (and for a similar purpose) as what you described in the blog post above...any chance the code that generates (and ideally, consumes) https://help.github.com/search/search.json is open source? If not, any interest in collaborating on a gem?

[gjtorikian]

It's all totally open-source!

We use lunrjs to build the search. A gem does exist to easily support it in Jekyll, but, it hasn't been updated for Jekyll 2.0, and is largely abandoned, I think.

But that's totally fine. Without the gem dependency, we can do all of this in native Jekyll, which makes it much easier to deploy to GHPages. Here's what we do:

• Include lunrjs and mustache.js as assets

• Create a blob of HTML that can be passed to lunrjs. This is an HTML template that's written with mustache:

  <script id="quicksearch-results-template" type="text/mustache">
  {% raw %}
  {{#entries}}
  <tr class="article">
    <td class="info">
      <a href="{{url}}" class="js-articles-quicksearch-link">{{title}}</a>
    </td>
  </tr>
  {{/entries}}
  {% endraw %}
  </script>

• Set lunrjs up (above the template definition):

  <script type="text/javascript">
  $(function() {
  $('#quicksearch-query').lunrSearch({
    indexUrl: '{{ site.baseurl }}/search/search.json',
    quickSearchResults:  '.autocomplete-results',
    quickSearchEntries:  '.result-group',
    quickSearchTemplate: '#quicksearch-results-template',
  
    searchResults: '.search-container',
    searchEntries: '.search-results',
    searchTemplate: '#search-results-template'
  });
  });
  </script>

• Define some sort of search.json file. Ours looks something like this:

  ---
  ---
  {
  
  "entries": [
  {% for article in site.dotcom_articles %}
  {
    "title": "{{ article.title }}",
    "category": "{{ page_category }}",
    "category_url": "{{ page_category_url }}",
    "url": "{{ site.baseurl }}/articles/{{ article.title | slugify }}",
    "body": "{{ article.content | strip_html | strip_newlines | replace: '\', '\\\\' | replace: '"', '\\"' }}",
    "excerpt": "{{ article.content | strip_html | strip_newlines | truncate: 140, '…' | replace: '\', '\\' | replace: '"', '\\"' }} "
  }{% unless forloop.last %},{% endunless %}
  {% endfor %}
  ]
  }

Basically, search.json is a collection of all your site's content. Lunrjs uses this to create its index. And the template defines the layour for the search results.

In order to keep Lunrjs's indexing off the main thread, I made a search worker to do it off-thread: https://gist.github.com/gjtorikian/aed86fec6b266c3e4d46

....In typing all this out, I realize how convoluted this all probably is. :worried: Lemme know if you need more info.

[afeld]

@gjtorikian Super helpful, thanks!

FYI, started playing around with (a couple approaches for) generating the JSON in the search branch, which I plan to eventually move into a Jekyll plugin. Ran into a handful of issues – will submit a PR soon to solicit feedback.

[jglovier]

which I plan to eventually move into a Jekyll plugin

:heart_eyes:

[afeld]

Working on the pages API here: https://github.com/18F/hub/pull/69

[afeld]

A 3rd-party option: http://www.digitalgov.gov/services/search/ via @gbinal

[gbinal]

Thanks, @afeld. If time comes, I'm happy to make the argument for dogfooding that shared service.

[mbland]

Just added https://18f.gsa.gov/hub/ to DigitalGov.

[gbinal]

Thanks, @mbland. There's a couple ways to integrate the resulting site search, through basically a page that they pull in or through their search API. Holler if you all need any help or want to connect with the team that runs this. They are one of the more agile and responsive teams I know in gov't so it's a good connection for us to deepen.