Create interactive web server

[kalekundert]

Curently, the lab notebook is hosted as a static site; it's basically just a graphical way to view notebook entries. However, it would be nice if the web interface was more interactive and more customized to the lab notebook application. Some specific features that would be nice:

• Reorganize experiments
  • Split into parent/child
  • Add tags/categories
• Browse experiments by tag/category
  • I like the idea of tags + arguments to create tables.
• Improved search:
  • Lab notebooking really benefits from powerful search features.
  • The search functionality in Sphinx is not very good.
  • One straight-forward way to improve the search would be to generate the index in advance. This would make the search faster, and allow me to be more sophisticated about stemming etc.
• Keep track of planned/current/complete expeiments (Kanban style)
  • Inspired by Kanobo.
  • A simple way to think about kanban:
  • Visualize work
  • Limit work-in-progress
  • Would probably also want to be able to add experiments via the web server.
  • Manage dependencies between experiments, e.g. expt #⁠2 can be done once I get expt #⁠1 to work.
  • This could allow me to automatically move experiments from "planned" to "current" once the precursor experiment is marked as "complete".
• Better experiment navigation:
  • With sphinx, there's a practical limit on how deep I can nest experiments before the navigation gets unmanageable.
  • This is a limit I'd really like to remove.
• Distinguish between goals and experiments:
  • I haven't thought about this much, but I've gotten the sense that I have two different kinds of notebook entries.
  • "Experiments" detail the results of a single protocol:
  • Pretty formulaic:
    • Intro: why I'm doing the experiment
    • Protocol: what exactly I did (thank you stepwise!)
    • Figure: the data I collected
    • Observations: my interpretation of the data
  • Doesn't matter if the experiment "worked" or not, I just report what happened.
  • "Goals":
  • A big picture thing that I want to accomplish, like "Get cDNA display working".
  • Often a planning area for new experiments/directions.
  • Tend to have "Considerations" sections, and TOCs of other experiments.
  • Have to keep adding subgoals/subexperiments until I have success (or give up).
  • Sometimes start as normal experiments, but turn into goals when the experiment doesn't work and I need to figure out why.
  • Is there a benefit to distinguishing these types of notebook entries?
  • Only goals can really be nested.
  • Goals are much longer term that experiments
  • Not clear.

The underlying directory structure would remain the single source of truth, and "edit" actions by the web server would directly operate on the file system. Some more thoughts about the implementation:

• My first instinct would be to use Django, although the fact that the server wouldn't have a database might make Django a poor fit. Perhaps a simpler framework like Flask would be better.
• I'd be able to develop this without interrupting my work at all, because this web server would just be a new interface to the same data I already have. The existing system should continue to work unchanged.
• Would I continue to use Sphinx/restructured text?
  • Compatible with what I already have.
  • I need support for cross-linking and footnotes, which Sphinx has.
  • Sphinx is extensible, even though plugins are not easy to write.
  • I might be able to run sphinx like normal, and have Flask serve the resulting _build directory.
  • Flask would have to translate Sphinx's links into its own. Not clear how hard/fragile this would be.
  • Since Sphinx would be the tool generating the final HTML for each experiment page, Flask would have to supply Sphinx with a custom theme to provide sidebars, etc.
  • Alternatively, I might be able to invoke Sphinx within the web server to generate the necessary HTML.
  • In pseudocode:
    • Parse the *.rst into nodes.
    • Convert the nodes into html, with hooks to handle cross-references.
    • Embed the resulting html in a Mako/Jinja template with sidebars etc. and return to client.
  • I've never done anything like this before, so I don't know how hard it would be, but at the very least it would involve really digging into the guts of sphinx.
  • Hooking into the cross-referencing directives/roles (e.g. .. toctree::, :expt:) might be hard.
• The routing would be pretty simple:
  • /: Dashboard-style views of current/planned/complete experiments.
  • /expt/<id>: Show the requested experiment.
  • Each experiment would have links to:
    • Child experiments, e.g. a TOC.
    • Parent experiments, perhaps as breadcrumbs.
    • Back-references, e.g. experiments that reference this one.
  • /tag/<name>: Show all the experiments associated with the given tag.
  • If the tag has arguments, this would be rendered as a table.
  • /search/<query>: Show a list of all the experiments matching the given query (with some context).
  • Every page would have a side bar with:
  • A search form.
  • Links to the top-level experiments.
  • Links to the current experiments.
• I'd probably want to add a .meta directory to each experiment to hold information about ids and tags. Right now the only metadata is the .id file in each experiment. Alternatively, I could keep the metadata in a single structured file (e.g. nested text) that contains ids and tags. A directory is probably more future-proof, though.
  • Decided to use a directory, with as much information as possible stored in different files. This should reduce the chance of merge conflicts.
• Frontend:
  • This is definitely something that could bog me down.
  • I could look for templates, but I'm unlikely to find anything that fits well.
  • There should be very little need for javascript, though. I should be able to get pretty far with CSS and semantic HTML, which are easier for me.
  • Use Furo theme for inspiration? I like that it has left and right side bars.
• Might be nice to monitor the filesystem, and automatically rebuild Sphinx/serach index in response to changes.

Probably the first thing to play with is the Flask/Sphinx linkage. If I can get that to work, the rest should be pretty straight-forward and incremental.

[kalekundert]

I can (and should) start this project by implementing the new kanban-inspired features (tags, planned/current/complete) in the CLI interface. This should be pretty straight-forward, except for having to replace the .id file with a .meta directory, and I'll be able to start using those features right away.

[kalekundert]

Experiment pages:

• Collapse all sections by default
  • This might require css/javascript, or changing how the builder outputs section nodes.
• Left sidebar:
  • Search
  • Top-level experiments
  • Current experiments.
• Right sidebar:
  • Download links for all files in experiment
  • Links to all child experiments
  • Button to add child experiment
  • Links to all parent experiments
  • Links to all tags
  • Button to add tag:
    • Click button → popup form
• Responsive design:

• Dashboard:

[kalekundert]

This article has some information on search engine packages available in python: https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xvi-full-text-search

Packages include: Elasticsearch, Apache Solr, Whoosh, Xapian, Sphinx

After briefly looking into this, I think whoosh is the best option. It's probably the least performant, and also doesn't seem to be that actively maintained. But all the other options would dramatically complicate installation, because they are written in other languages (e.g. java, c++) and require running/communicating with a server process. Whoosh is pure-python and has no separate server. Since exmemo is meant to be installed by end-users, having a straight-forward installation process is the most important consideration.

[kalekundert]

Kanban-style statuses I'd like to have:

• New/Conceived/Described/Planning
  • Default status after exmemo note new
• Ready/Planned
  • Once I'm ready to actually start working on the experiment (e.g. I have a protocol).
• Pending/Waiting/Blocked
  • Ready or New, but waiting on another experiment to finish.
  • Will automatically revert to previous state once prereqs are complete.
  • Not an explicit state; inferred from prereqs.
  • Not shown by default.
• In progress/Started
• Complete/Finished
• Abandoned/Deferred/Withdrawn/Paused/Canceled
  • Like Blocked, this is a meta status. If undone, automatically return to previous state.
  • Not shown by default

Worried I have too many states, some of which may not be very useful. But for now I'm just going to cater to my own needs, and worry about generalizing later. Ex memo is somehwat opinionated in general, so this isn't even inconsistent with the bigger picture.

Need to add --prereq argument to exmemo note new, that will automatically add the new experiment as a prerequisite to its parent.

---

Kanobo seems to have the following statuses:

• Upcoming
• Work in progress
• Done
  • Completed
  • Abandoned

[kalekundert]

New commands:

• ex memo note status <ready|in-progress|complete|abandon> [expt]
• ex memo note assign-ids
• ex memo note priority
• ex memo note tag
  • - prefix: remove tag
  • + prefix: add tag
  • how to specify expt?