ly http-server

[uliska]

I'm considering to write a ly HTTP server but wanted to get some feedback before diving in.

Use cases/benefits for such a server mode are/include:

• "live" syntax highlighting on websites
• implementing lilypond code editors in browser (extension of first item)
• integrating ly in non-Python applications
• batch processing

The latter item is why I'm considering this. I want to rewrite my plugin that does the highlighting in the LIlyPond Textbook, and instead of firing up the Python script for each LilyPond snippet I hope that it is more efficient to start a server at the beginning of that process and retrieve the highlighted data from that.
If that works out it should be an option for building the LilyPond docs as well (I really would love seeing LilyPond's docs with highlighted code examples).

Any ideas, caveats, objections at this point?

[wbsoft]

A nice idea! Probably Python already provides most of the needed infrastructure: https://docs.python.org/3/library/http.server.html A side note: for this server-mode it would really be nice if ly.music could be superseded by ly.xml so we can have an XML-representation of the music available for javascripts that provide more functionality to in-browser code editors.

[uliska]

Am 01.04.2016 um 07:14 schrieb Wilbert Berendsen:

A nice idea! Probably Python already provides most of the needed infrastructure: https://docs.python.org/3/library/http.server.html

Yes, that's about what I thought of.

A side note: for this server-mode it would really be nice if ly.music could be superceded by ly.xml so we can have an XML-representation of the music available for javascripts that provide more functionality to in-browser code editors.

OK, that sounds reasonable. However, I think I wouldn't tackle that initially, and I don't think it's necessary. It should be sufficient to keep it in mind so an initial implementation wouldn't build any obstacles for a later XML implementation.

What I thought of initially is that the server responds to GET requests and configures the command, config and data to be processed either by query params in the URL and raw data in the request body, or by accepting a JSON string in the message body.

That JSON string should (sketch:) contain "command", "config" and "data" fields. The "config" structure should depend on the command (e.g. "transpose" expects two pitches, while "highlight" can have quite different configuration parameters).

[uliska]

One more question. Would you prefer a new command

ly serve

in ly itself or rather a separate ly-server program?

[wbsoft]

I think the commands manipulate a document or documents (note that you can chain commands, like ly "transpose c d;indent" etc. So a separate server programm is better, which can be put in bin/ly-server and simply contain something like

#!/usr/bin/env python
import sys
from ly.server.main import main
sys.exit(main())

[uliska]

OK, I arrived at that solution as well.

I now think I know what the server should "look" like. If noone objects I'll implement an http server with a REST api, so clients will put the data (i.e. the "document" to process) as a plain text or JSON payload in the request body and do the configuration through the URL:

http://localhost:4001/highlight
http://localhost:4001/transpose/a/b
etc.

If the payload is plain text it is used as-is, if it is JSON it should have a config field with more information. An example for the highlight command:

{
  "config": {
    "css": "ext",
    "document": "body"
  },
  "data": "\relative c' {\n  c'\n}"
}

As a first step I'll only implement the highlight command as that is what I need personally. But through the REST approach that should not in any way interfere with future enhancements for full coverage.

As a proof-of-concept I intend to write a website/webpage where the user can get LilyPond code highlighted. Through a form they will enter the document text (or upload it) and configure the output. The highlighted result will be displayed in an output field. Depending on the actual speed of the process this may even be made "live".

Which leads me to a question that I had for quite some time: Do you think it's necessary to "sanitize" input that is fed into ly (be it for highlighting or any other command)? Or will the worst thing to happen be errors or useless results?

[uliska]

It seems the generally recommended way is to write sucha RESTful server using the Flask microframework. It seems this would create a new sort-of dependency for python-ly: Concretely, anybody who wants to run the ly-server would have to do pip install flask before. If I'm not mistaken any other use of python-ly (including Frescobaldi) would not be affected.

Is this acceptable?

[uliska]

Paddling back after some research: Flask is probably more expensive than necessary. Actually this ly-server is very clear about its goals and requires very little server functionality. So it really should be worth the effort avoiding additional dependencies.

https://gist.github.com/tliron/8e9757180506f25e46d9

Seems to indicate a solution that can easily be built upon.

[wbsoft]

If ly-server would require additional dependencies just for the server tasks, then please consider making ly-server a separate package that depends on python-ly and the other packages. Python-ly should be a powerful LilyPond-related python module and a simple commandline tool. Larger applications can depend on python-ly. (or copy the ly/ folder in their own module directory)

[uliska]

As written in my previous comment I've paddled back and started (today) implementing it with standard tools. There is absolutely no need for a "framework" here.

I think I'll be soon at a stage to share and have the code reviewed.

[uliska]

OK. I still think it would be a good idea to make the commands behave consistently and then reuse them, but nevertheless...

[uliska]

OK, I have now implemented an initial working version of a ly-client node module and tested it with a Gitbook plugin. So it goes:

• Gitbook uses gitbook-plugin-lilypond-highlight
• the plugin requires the ly-client node module
• ly-client calls ly-server through HTTP or ly

Both methods work to render the Gitbook correctly, but interestingly firing up the ly script is about 1/3 faster than sending a request over HTTP to localhost (compile time of the whole book is about 6.5 vs. 9.5 seconds). That seems to indicate that using ly-server on localhost is generally less efficient than directly invoking ly.

Ironically this means that ly-server isn't useful for the original use case. However, it will be useful then when using it on a real server, as a backend tool for interaction with a website. I think I'll go for also writing an Angular module angular-ly-client some day.