Manifest Parsing

[musicus]

Currently when downloading files from the ELVIS database the user has two download options: (1) a zip file (flat) without directory structure, containing a meta data (manifest) file, and (2) a directory structure where the meta data file sits in the furthest branch of a directory tree (download some file, to see what I mean).

We need to be able to parse these meta data files, and assign the meta data with the appropriate VIS functions to the pieces.

[ahankinson]

I'm confused -- what part of parsing the metadata would go into the VIS framework itself?

[musicus]

The meta data will not go into the vis-framework, only the parser. We have this module http://vis-framework.readthedocs.io/en/latest/vis.models.html#module-vis.models.indexed_piece and perhaps we should be able to read the meta data from the downloaded files from the ELVIS database, which includes a meta file, with this module by extending/refactoring it.

This is useful for analysts using the vis-framework by itself. But we could also just make it an independent extension or simply a script within the new tutorial/documentation.

[musicus]

@mborsodi In addition, the meta data can also be attached to the music21 stream. The following questions need also be investigated:

• Is the vis mechanism separate from the music21 process?
• Can the meta data, once assigned with the vis mechanism, read in at any time, as is the case with the music21 stream?
• Is meta data from music21 streams added to vis indexed pieces?
• Does the meta data persist throughout all indexers?

[alexandermorgan]

@musicus didn't you do a bunch of work on importing folders of files several months ago? It seems like this task would go hand in hand with that one. As I understand it, VIS collects metadata at the import stage, so it is generally not something that would be done at any time. A user could manually assign metadata at any time though. Concerning your last two questions, metadata in VIS is assigned as attributes of an indexed piece so it will persist throughout all indexers because none of the indexers overwrite or interact with this metadata.

[minamouse]

If we have the option for a meta-data file at the import stage, though, we could do it all at once, right?

[musicus]

@alexandermorgan No, I did not (I worked on something else), because we did not have the same type of meta data files generated by the ELVIS database. The last time I worked with this issue, meta data was not properly assigned, and therefore was not available at any stages. The questions were directed @mborsodi, since she is working at a solution.

[alexandermorgan]

I chimed in here because I was temporarily assigned. @mborsodi it seems like all the metadata should be able to be assigned at the same time, but if you work on this make sure that whoever is in charge of the database now is going to keep the metadata file the same because it would be a shame if you worked out a solution and then the format of that file changed.

[minamouse]

Yes! Thank you @alexandermorgan, I will definitely look into that.

[crantila]

Responding to @musicus' questions above:

• Is the vis mechanism separate from the music21 process?

All of the VIS metadata come from music21.

• Can the meta data, once assigned with the vis mechanism, read in at any time, as is the case with the music21 stream?
• Is meta data from music21 streams added to vis indexed pieces?

Metadata are (always, only) stored in IndexedPiece instances, and are available for as long as the instance is accessible, regardless of what happens to the Stream from which it was obtained. They're imported by the _import_score() method, which should be the only way a music21 Score is ever imported into VIS.

• Does the meta data persist throughout all indexers?

Because the metadata are stored in the IndexedPiece, analyzers can't actually access them. But yes, analyzer instances are ephemeral, and the IndexedPiece is created before and survives after any analyzers run on a piece, therefore the metadata persists through all indexers.

I'll also point out that you want to use the often-overlooked AggregatedPieces class here, and that it already has some interesting metadata-handling functionality.

[musicus]

@crantila thanks for answering questions for @mborsodi. I'm sure this will make it easier for @mborsodi to parse the meta data file(s) included in the zipped downloads from the ELVIS database.