[Site/Search] Reduce Technical Debt, and Parameterize, the Pipeline

[pishoyg]

TODO:

• HTML code (which you write) is code, such as the Crum index page.
• Move CSS out of Crum's index.html file.
• Update the diagram to reflect the recent changes, such as the search engine implemented in #108.
• Split the pipelines at component boundaries, for example ts_transpile, and xooxle.
• Store the generated home page in the coptic repo, instead of constructing it and copying it in the site_publish pipeline, which doesn't write anything to coptic but goes to pishoyg.github.io straightaway. This is more consistent with what you do in your other pipeline. It will also allow you to look at the home before it's published.
• Load the search index file locally to support local testing as well.
• Site publishing copies the HEAD commit, along with a boolean indicating whether the repo was dirty, instead of always having a fixed commit message.

[pishoyg]

Status:

Done:

• HTML code (which you write) is code, such as the Crum index page.
• Move CSS out of Crum's index.html file.
• Site publishing copies the HEAD commit, along with a boolean indicating whether the repo was dirty, instead of always having a fixed commit message.

No longer relevant:

• Store the generated home page in the coptic repo, instead of constructing it and copying it in the site_publish pipeline, which doesn't write anything to coptic but goes to pishoyg.github.io straightaway. This is more consistent with what you do in your other pipeline. It will also allow you to look at the home before it's published. (No longer relevant, we don't write markdown anymore.)

TODO:

• Update the diagram to reflect the recent changes, such as the search engine implemented in #108.
• Move the HTML header to the HTML, not JavaScript, essentialY undoing e1547e048c281144814925b7b71eebfe948036e3. This is required because the header will differ from one instance of xooxle to another. Locality is not highly desirable.
• Split the pipelines at component boundaries, for example ts_transpile, and xooxle. Consider merging ts_transpile into other pipelines, and not saving the intermediate copy of the JavaScript.
• Load the search index file locally to support local testing as well.

[pishoyg]

Status:

Done:

• Move the HTML header to the HTML, not JavaScript, essentialY undoing e1547e048c281144814925b7b71eebfe948036e3. This is required because the header will differ from one instance of xooxle to another. Locality is not highly desirable. (Done in 746199dd87f119614308c29d217c1cae9ab5a61a.)
• Load the search index file locally to support local testing as well. (Done in 4b76480b4f12151e242f4b16d463ce548164cbbe.)

TODO:

• Clean up the raw HTML, e.g. by removing comments and excess space.
• Parameterize the search behavior utilizing the new metadata field. Stop giving the text field special treatment in xooxle.ts.
• Parameterize the highlighting behavior utilizing the new metadata field.
• Parameterize the order used to populate the fields in the results table. Right now, we don't do any ordering, and data just happens to fall in the right order!
• Update the diagram to reflect the recent changes, such as the search engine implemented in #108.
• Split the pipelines at component boundaries, for example ts_transpile, and xooxle. Consider merging ts_transpile into other pipelines, and not saving the intermediate copy of the JavaScript.
• Make it possible to strip some unused HTML classes, to reduce the size of the search index. Stripping refers to retaining their text but discarding their HTML tags. For the "marcion" (which is currently the only field that we retain in raw format), we could, for example, get rid of the tags bearing such classes as coptic, greek, and possibly others.

[pishoyg]

For HTML reduction: https://stackoverflow.com/questions/1765848/remove-a-tag-using-beautifulsoup-but-keep-its-contents

[pishoyg]

NOTE: We started experiencing some performance degradation, now that the size of our search index is 24MB. The page takes a few moments to load, and you can now feel it. While this happens only occasionally (because the file gets cached by the browser afterwards), we still shouldn't be wasteful and should keep the data optimal whenever possible.

[pishoyg]

Status:

Done:

• Update the diagram to reflect the recent changes, such as the search engine implemented in #108.
• Clean up the raw HTML, e.g. by removing comments and excess space.

Abandoned:

• Split the pipelines at component boundaries, for example ts_transpile, and xooxle. Consider merging ts_transpile into other pipelines, and not saving the intermediate copy of the JavaScript.

TODO:

Parameterization:

• Parameterize the search behavior utilizing the new metadata field. Stop giving the text field special treatment in xooxle.ts.
• Parameterize the highlighting behavior utilizing the new metadata field.
• Parameterize the order used to populate the fields in the results table. Right now, we don't do any ordering, and data just happens to fall in the right order!

Optimization:

• Make it possible to strip some unused HTML classes, to reduce the size of the search index. Stripping refers to retaining their text but discarding their HTML tags. For the "marcion" (which is currently the only field that we retain in raw format), we could, for example, get rid of the tags bearing such classes as coptic, greek, and possibly others.

[pishoyg]

I have an idea for HTML reduction. Let's add a retain_classes parameter to the selector. When we extract text from a tag, let's retain the elements belonging to classes in that list in raw format, and let's extract the text from the other elements.

[pishoyg]

Let's get rid of the raw parameter?

This whole thing is complicated and needs some brainstorming!

[pishoyg]

Update on this optimization suggestion: Initial experimentation shows that it's actually working great! We will adopt it!

[pishoyg]

Status:

Done:

• Make it possible to strip some unused HTML classes, to reduce the size of the search index. Stripping refers to retaining their text but discarding their HTML tags. For the "marcion" (which is currently the only field that we retain in raw format), we could, for example, get rid of the tags bearing such classes as coptic, greek, and possibly others.
  • This also reduced the size of the index from 24MB to 17MB.
  • Regarding #241, the new HTML is more mobile-friendly.

TODO:

• Parameterize the search behavior utilizing the new metadata field. Stop giving the text field special treatment in xooxle.ts.
• Parameterize the highlighting behavior utilizing the new metadata field.
• Parameterize the order used to populate the fields in the results table. Right now, we don't do any ordering, and data just happens to fall in the right order!

[pishoyg]

Xooxle has become our flagship product, so we prioritize the improvements.

[pishoyg]

Status:

No longer relevant:

• Parameterize the highlighting behavior utilizing the new metadata field.
• Parameterize the search behavior utilizing the new metadata field.

DONE:

• Parameterize the order used to populate the fields in the results table. Right now, we don't do any ordering, and data just happens to fall in the right order!
• Stop giving the text field special treatment in xooxle.ts.

TODO:

• Simplify the parameters for the view cell. Currently you have three different parameters (view, path_prefix, and retain_extension). It should be just one path_fmt field.
• Parameterize the name of the path / key field, instead of just assuming that it's always going to be called path. This will also make it possible for you to simplify the JSON parsing in xooxle.ts.

[pishoyg]

TODO: Document the simplicity of the HTML structure supported by Xooxle. Currently, you only allow the following tags:

• <span>
• <b>, <strong>
• <i>, <em>
• <br>

[pishoyg]

Status:

No longer wanted:

• Parameterize the name of the path / key field, instead of just assuming that it's always going to be called path. This will also make it possible for you to simplify the JSON parsing in xooxle.ts.

DONE:

• Simplify the parameters for the view cell. Currently you have three different parameters (view, path_prefix, and retain_extension). It should be just one path_fmt field.
• Document the simplicity of the HTML structure supported by Xooxle.

TODO:

• See in-code TODO's.

[pishoyg]

Status:

TODO:

• See in-code TODO's....

[pishoyg]

Reopened #234.