Nuances of the bundle format

[bradenmacdonald]

Warning: brain dump.

At this time, the evolution of the bundle format used for content libraries in blockstore-relay and my draft runtime PR is described in this docstring.

One aspect of this design that has proven a bit more complex than I hoped is the following:

        Any OLX file that represents an XBlock type that can have children may
        contain <xblock-include> nodes, like this:
            <unit>
                <xblock-include definition="html/intro" />
                <xblock-include definition="html/intro" usage="alernate" />
                <xblock-include definition="linked_bundle/html/intro" usage="linked_intro" />
            </unit>
...
        Any block which exists in the main bundle (not linked in to the bundle)
        and which is not a child of other blocks (no <xblock-include /> nodes
        reference it) is called a "top-level block". Its usage ID always matches
        its definition ID.

Another way of phrasing the existing design is:

• Every definition.xml file represents a definition
• All definitions have one or more usage
• If an <xblock-include ...> directive exists anywhere in the bundle referencing a particular definition, it's an explicit usage of that definition
• If no <xblock-include ...> directives in the bundle reference a particular definition, that definition is called a "top level block" and gets an implicit usage with the same usage ID as its definition ID. Top level blocks are the blocks that one sees when listing the contents of a library.

This design has these nice properties:

• It's impossible to get "orphaned" definitions: either a definition has one or more parents, or it's a "top level" block.
• There is no need to maintain a separate data file / structure that lists top level blocks, because top-level-ness can be computed by scanning the bundle.
• A usage can be moved from a top-level usage to a child of another block in the bundle without changing its usage ID, and vice versa. (This is a design requirement.)

But it has these problems:

• A block cannot be a top-level block in the library and also used as a child usage of some other block. (Most problematic is this situation: library bundle contains two top-level blocks A & B. Now a user adds a new usage of block B as a child of A. Instead of now having A & B in the library with a second usage of B as a child of A, the library now contains only A as a top level block, with a usage of B as its child. Creating a "second" usage of B has demoted B and made it no longer a top level block. This may be very surprising behavior for users because the user was not trying to move the block B underneath A but simply create a second usage of B)
  • A workaround if you really understand what's going on and want to keep B as a top level block is to create another empty top level block C and add another usage of B as a child, creating a wrapper around B. But that's messy.
  • This issue doesn't affect blocks in linked bundles, because the "top level" status of blocks is not affected by usages in other bundles. If we expect (or require?) that the vast majority of multiple-usages-per-definition happens via inter-bundle links (e.g. courses linking to library content) and not intra-bundle includes (e.g. course with the same leaf html block in two places), then this is largely a non-issue.
• Looking up a usage ID and verifying that usage IDs are unique can require scanning all the OLX files in the bundle (to check <xblock-include usage="..." ...> elements). This can be slow, although cacheable. (Also, in practice this is optimized considerably since the runtime knows not to bother reading definition files of types like html that cannot have children.)

So although those may not be big problems, they don't feel like a particularly elegant or clean design. I would be open to better ideas before anything gets too entrenched.

Some ideas I considered but discarded:

• Having a manifest file that explicitly lists the usages of top-level blocks in the bundle. (Rejected because we need to be able to publish changes to individual blocks at a time, and there's no way to atomically publish changes to only one line in a manifest file using the blockstore drafts API while leaving other changed lines in the draft unaffected. Also because this can created orphaned definitions that are neither top-level blocks nor children of any other blocks. Also because IDs in the manifest can become out of sync with the definitions if there are any bugs or write conflicts.)
• Identifying top level blocks via the presence of an empty file like .top-level alongside the definition.xml file: this allows draft/publish of individual blocks and marking blocks as top-level or not regardless of how many usages they have (nice), and it can never have IDs that are out of sync, but it still allows orphaned blocks. It's also a bit inelegant to have these extra files.
  • An alternative is allowing the definition files to be named either definition.xml or child-definition.xml depending on whether they are "top level with optional other usages" (definition) or "not top level" (child-definition). Still allows orphans though.
  • Perhaps we don't care if orphans exist, or we still scan need to scan through the bundle's various <xblock-include directives anyways, so we can detect them and display them as top-level blocks along with a warning.

[SSPJ]

A quick thought: you talk about the need to cache results of computing top level blocks. If this computation is cached by means of a flag in the block itself, it can become "sticky", i.e. once a block is found to be a top level block, it remains so thereafter until a user explicitly removes the flag.

In that way, a block stays in the library even if it is later added as a child to other blocks.

[symbolist]

@bradenmacdonald @ormsbee

A couple of questions:

1. I understand in general most blocks that are children of others in a library may not be for re-use but which blocks are for re-use seems to me to be orthogonal to where they are in the block graph in the bundle? The problems noted above are because of complecting the two?
2. To take this one step further should re-usability status of blocks in a library even be declared in or derived from the definition of blocks?

[bradenmacdonald]

I guess I was assuming that in general, blocks designed for re-use should be put at the top of the bundle/library. Though it's certainly possible to re-use child blocks.

Do you think we need a separate mechanism for flagging blocks as "suitable for re-use"?

[symbolist]

@bradenmacdonald

Do you think we need a separate mechanism for flagging blocks as "suitable for re-use"?

That certainly is more flexible but whether we need that flexibility or can work around it depends on product requirements. I don't think any users will be using the library 2 interface in Studio immediately so maybe we come back to this later?

[bradenmacdonald]

Closing as this should be resolved by #53.