bbrowning
commented
1 month ago A few high level items that need some thought and/or discussion:
- How do we keep track of
ilab data generateruns (sessions? not sure what to call these) that we might want to be able to resume? For example, if I run data generation on taxonomy A, taxonomy B, different branch/commit of taxonomy B, taxonomy C with the simple pipeline or taxonomy C with the full pipeline - each of these would have different saves that we'd need to keep track of. Or, is the assumption that the user can only ever resume the most recently runilab data generateand must use the exact same set of args, taxonomy, taxonomy branch/commit, etc as when they started it? Is this even a concern of SDG, or do we task the CLI with this type of bookkeeping? - How do we handle documents (qna.yaml or the knowledge docs) that have changed in some way since we started processing the first time? Do we guard against this, with some kind of per-document (or more likely per-chunk) hashes we save to ensure that we're not resuming against a document that has been modified?
- What types of user action cause us to invalidate our saved checkpoints and refuse to resume? This is kind of the inverse of the first bullet - what are the things that we know would break resuming?
- What type of durable storage should we assume is available to us for any state tracking required here? Our limited checkpoint state today is exclusively filesystem-based - is that what we should continue to use, encoding any state we need into files in our datasets directory?
- When (if ever) do we clean up these intermediate states? Can we assume that once a specific SDG session runs to completion that any of the interim bookkeeping data we've written out can be deleted?
Some of the above is complex. If that's too complex to get started, what's our happy path case that we want to ensure works? Is it good enough to assume a user runs a single ilab data generate ..., it gets interrupted, and then they run it again and we continue from where they left off automatically? And if they change any params, docs, config, etc they have to manually blow away the entire state we've saved to start from a clean slate? We've already seen requiring users to manually clean SDG checkpoints between runs can be confusing for them and lead to errors if they forget - are we ok continuing down that path, or is this the right time to make this more friendly?
ktam3
Feature Overview (aka. Goal Summary) When running the SDG cycle ("ilab generate") the execution continues until all the provided documents are processed and the number of requested samples is generated.
This Feature card is to enhance InstructLab:
Goals (aka. expected user outcomes) The observable functionality that the user now has due to receiving this feature. Include the anticipated primary user type/persona and which existing features will be expanded. Complete during New status.
Requirements (aka. Acceptance Criteria): A list of specific needs or objectives the feature must deliver to be considered complete. Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc. Initial completion during Refinement status.