File structures and presentation for collections published in Mutopia

[dominicus]

Opening a discussion thread to collect & discuss proposals for alternative approaches to submit and publish collections in Mutopia. The baseline approach available is for the admin to define a collection. These are indexed in the "Browse" section. Each individual piece is listed in compact form See our existing listing of collections in the collection page.

[dominicus]

Proposal for file structure for a collection. Objective was to allow for definition of a separate "all" PDF including all individual pieces, avoiding dual-maintenance of music, and conforming to the existing file structure for Mutopia:

Aspects of the Mutopia file structure that impact consolidated PDF from existing, related pieces:

The assigned pieceID defines a unit of content. Each pieceID is assigned an exclusive primary folder within the corresponding ftp/composer/opus tree. The assigned folder holds one or more .ly or Makefile file(s) used to produce PDF and midi content associated with the pieceID. The source files need to compile in standalone fashion: referencing other files is OK, but these references need to be relative references to files found within the primary folder or its subfolders. If these were the only constraints, a viable file structure for a compilation could be: ftp/composer/opus/book.ly -> assigned a unique pieceID, \include's music from subfolders, produces book PDF and individual midi's for all scores ftp/composer/opus/opus-1/piece1.ly -> assigned a unique pieceID, produces piece1 individual PDF and midi ftp/composer/opus/opus-1/piece1-music.ily -> music to be \include'd into both piece1.ly and book.ly ftp/composer/opus/opus-2/piece2.ly -> assigned a unique pieceID, produces piece2 individual PDF and midi ftp/composer/opus/opus-2/piece2-music.ily -> music to be \include'd into both piece2.ly and book.ly With this nested folder structure, one can bundle the folder and subfolders under ../opus/ to get package all sources for the book, and each piece can be .zipped separately from the files found in /opus/opus-x/ folder/subfolders.

I wonder if there's a constraint to this I'm missing? Would this work within Mutopia?

There's downsides to this, of course: first is the file-count increase, second is linking in this fashion puts the book.ly at risk of breaking every time a version upgrade is done on the underlying individual scores. This will be the case for any configuration that leverages music for both individual and book PDF generation.

[ksnortum]

There is a definite upside that just be mentioned. When a change or correction need to happen to the single movement file, it flows to the suite or all movements file automatically. And using Frederico Bruni's suggestion [1] of using a \bookpart in the movement file, there is only one file for the movement and one file for the suite, so there is no increase in the number of files.

[1] https://mail.google.com/mail/u/0/#label/Mutopia/145f89fa526d38c0

[dominicus]

\bookpart blocks can't contain variable declarations and certain blocks within it, which generally requires individual files for the music, individual files to format and print each part, and one more to consolidate the collection. There's also pain-points regarding how LilyPond handles naming of multiple MIDI's generated from a single .ly. There's an enhancement to LilyPond in the works, but until then, workarounds involves yet more .lys or custom makefiles.

[fedelibre]

I don't know how a collection is defined in Mutopia, so I cannot comment on most of the discussion... ksnortum meant to link to this discussion (sadly, archives are private): http://lists.bcn.mythic-beasts.com/mailman/private/mutopia-discuss/2014-May/000858.html

You don't have to put the variable declarations inside \bookpart block. Is MIDI naming really an issue here? I think that the midi should be generated for each piece only, while we should find a way to not produce any midi when the book is compiled. See the solution at the bottom of this discussion: http://lilypond.1069038.n5.nabble.com/Lilypond-compile-with-no-midi-output-td140395.html

Piece structure:

\version "2.18.2"
music = \relative c' {
  c1
}

\bookpart {
  \header {
    title = "piece 1"
  }
  \score {
    \music
    \layout {}
    \midi {}
  }
}

Book.ly:

\version "2.18.2"
\include "piece1.ly"
%\include "piece2.ly"
% etc.

book should be compiled this way to avoid the generation of midi:

lilypond -e "(set! write-performances-midis (lambda (performances basename . rest) 0))" book.ly

I made a quick test and it works on 2.18.2 but not on 2.19.6

[dominicus]

Yes, my assumption is that several pieces collated in book form become another pieceID in Mutopia, and include PDFs/MIDIs/.lys (and maybe links to the individual pieces in "moreInfo"). Have you tested this structure with \bookpart? I'd be looking to see how it deals with the \header and \paper blocks, and whether you lose any flexibility of having different maintainers or licenses at the individual piece level.

We should also consider pros/cons of just collating the book PDF programmatically with open tools like PDFSam directly from the individual PDF's.

[fedelibre]

I don't like the idea of collating PDF using PDFSam or similar, because you lose the power of lilypond (page numbers, headers, etc.).

\bookpart deals well with header and paper blocks, it's very flexible. The only big problem is MIDI. However, as long as each piece is in a different directory, having two files (one for notes and one for the piece) is not a big nuisance. Maybe if/when LilyPond is able to deal better with MIDI exclusion, we may use \bookpart.

[fedelibre]

What I don't like is the use of a redundant -lys directory, like here: https://github.com/chrissawer/The-Mutopia-Project/tree/master/ftp/BeethovenLv/WoO30/Drei_Equali

All the files are in one directory deeper but they could be in the parent (main) directory, where is no file.

[dominicus]

Review/comment #410 in context of this discussion.

[ksnortum]

What I didn't like about using \bookpart for this compilation was that each movement had to start on a new page. I think one of the advantages to an "all movements" piece should be the ability to have shorter movements takes less than a whole page.

[dominicus]

Reviewed the output of #410: http://www.mutopiaproject.org/cgibin/piece-info.cgi?id=475

The published collection .zip includes the collated "all" version plus the individual pieces, but the individual movements are not indexed separately in Mutopia, nor are they available for independent download.

The objective is to identify a solution that would allow both (collection & individual) to be offered without the need for dual maintenance of individual pieces:

• Search results for the collection title (i.e. "Pictures at an Exhibition", "Mussorgsky"), should yield a listing for the collection.
• Search results for an individual piece (i.e. "Catacombae" or "Catacombae Mussorgsky"), should yield a listing for the individual piece.

Having each movement in a collection individually indexed (i.e. independently published with its own PieceID), allows users searching for a specific piece to find the individual pieces (same as current collections), and allows future download counts and popularity metrics to be tracked at the piece level.

[dominicus]

Trial collection book: see #554