BigBang now includes data as part of its repository: tables of labeled organizations, labeled email domains, lists of mailing list URLs, etc.
(I'm not talking about PII and email data here. I'm talking about the other stuff.)
Currently, this material is spread ad hoc through the examples/ directory, in a directory associated with a particular project or notebook.
But as integrate these data sets, the directory distinctions in the examples/ directory start to not make sense.
Moreover, it is best practice to, in these cases, provide a programatic interface to the data (not just a flat CSV file) so that it can be more easily documented.
In that project, datasets is a submodule of the Python package. That submodule has several further submodules for types of data and methods that relate to programmatic access. These python methods can then all be documented using docstrings and that documentation shows up on ReadTheDocs as API documentation.
BigBang now includes data as part of its repository: tables of labeled organizations, labeled email domains, lists of mailing list URLs, etc.
(I'm not talking about PII and email data here. I'm talking about the other stuff.)
Currently, this material is spread ad hoc through the
examples/
directory, in a directory associated with a particular project or notebook.But as integrate these data sets, the directory distinctions in the
examples/
directory start to not make sense.Moreover, it is best practice to, in these cases, provide a programatic interface to the data (not just a flat CSV file) so that it can be more easily documented.
Consider this way of handling data within a git repository: https://github.com/econ-ark/HARK/tree/master/HARK/datasets
In that project,
datasets
is a submodule of the Python package. That submodule has several further submodules for types of data and methods that relate to programmatic access. These python methods can then all be documented using docstrings and that documentation shows up on ReadTheDocs as API documentation.