how should we treat Scheme code?

jwaldmann commented 4 years ago

LISP (Scheme) code starts with #, and this happens all the time, e.g.,

  42     \override Glissando.breakable = ##t

  70     systemStartDelimiter = #'SystemStartSquare

 138 modern =
 139 #`(Staff ,(make-accidental-rule 'same-octave 0)
 140   ,(make-accidental-rule 'any-octave 0)
 141   ,(make-accidental-rule 'same-octave 1))

Even if we want to ignore it - we must be able to skip it reliably, so the parser still has to analyze nesting of parens. Not trivial! We need to discard parens in strings and comments, etc.

We could avoid rolling our own, by using https://hackage.haskell.org/package/atto-lisp instead (they use attoparsec, not parsec) But - I'm not seeing whether that library can handle

# literals (see example ##t, I think first # is for lilypond to know that LISP follows, second # is LISP)
and unquotes (the , in line 140 above)

Another candidate is https://hackage.haskell.org/package/megalisp https://github.com/kuribas/megalisp (using megaparsec, not parsec)

jwaldmann commented 4 years ago

Ha, no, lilypond apparently can switch back to Ly from inside Scheme, as in

  47 moveDyn =
  48   #(define-event-function (x y event) (number? number? ly:event?)
  49     #{ \tweak DynamicLineSpanner.outside-staff-priority ##f
  50        \offset DynamicText.X-offset #x
  51        \offset DynamicLineSpanner.Y-offset #y
  52        #event #})

So, we have to roll our own LISP parser since it needs to call the Ly parser.

jwaldmann commented 4 years ago

And it's worse - if we want to extract notes, we have to interpret LISP to some extent, e.g., the above moveDyn could get used like

\tuplet 3/2 { d'4\noHorzSpace \moveDyn -1.5 -3.2 ^\ff r8 }

we need be able to see that \moveDyn has three arguments?

ejlilley commented 4 years ago

In practice it's uncommon for people to define new Lilypond commands in Scheme. I reckon we could get to 99% coverage of actually-existing Lilypond files without writing a lisp interpreter.

jwaldmann commented 4 years ago

Yes, let's hope for that. We should copy some more "actually-existing lilypond files" into our test data set. I made a start with ly-examples. Can you suggest other sources? It's gotta have the right licensing. Perhaps https://www.mutopiaproject.org/ ?

[EDIT] I copied a few (very few) files from https://github.com/MutopiaProject/MutopiaProject/tree/master/ftp It's no use maintaining a copy of their tree. Instead, write test automation to download mutopia and run lilypond-parse (I'll make a separate issue)

jwaldmann commented 4 years ago

we need be able to see that \moveDyn has three arguments?

I am reading this as "we need to know semantics (arity of functions) to decide about syntax (who is argument to whom". I'm pretty certain this makes parsing undecidable (there could be conditional definitions? you have to evaluate code before knowing arities) (as it is for Perl, http://www.jeffreykegler.com/Home/perl-and-undecidability9) It's (not) funny how often (total) mess occurs in parser.yy ...

jwaldmann commented 4 years ago

Briefly discussed in Goals of README. Closing for now, might re-open later, when looking at actual examples.

ejlilley / lilypond-parse

how should we treat Scheme code? #5