After building and working with this initial version @saulshanabrook and I talked more about the underlying data model. We have a few observations about the nesting of datasets:
The current approach of allowing datasets to be nested introduces a lot of extra complexity without a commensurate amount of value to the user.
There are other types of relationships that datasets may have than strict nesting.
The more general case is that of a full knowledge graph of people, datasets, notebooks, etc.
Based on these factors, our sense is that this extension and its API should focus on the core dataset interface and API and not expose an abstraction around nested datasets. Instead, in cases where more complex relationship between datasets and other entities are required, additional, separate extensions should be built that use the unique dataset identifiers to build the needed knowledge graphs.
At the end of the day, we are proposing to simplify the data model by removing dataset nesting.
Description
After building and working with this initial version @saulshanabrook and I talked more about the underlying data model. We have a few observations about the nesting of datasets:
Based on these factors, our sense is that this extension and its API should focus on the core dataset interface and API and not expose an abstraction around nested datasets. Instead, in cases where more complex relationship between datasets and other entities are required, additional, separate extensions should be built that use the unique dataset identifiers to build the needed knowledge graphs.
At the end of the day, we are proposing to simplify the data model by removing dataset nesting.