Closed danbraunai-apollo closed 9 months ago
TODO:
Generally these arguments are confusing to me, I'm not sure what is supported where.
I update these a little bit:
tokenizer_name: str = Field(
...,
description="The HuggingFace name for the tokenizer. Please check whether the tokenizer is "
"compatible with the model you are using.",
)
return_set: Literal["train", "test"] = Field(
..., description="The dataset split to return from HuggingFace."
)
What do you think? I don't think we can or want to list a full set of tokenizers the users can load, if that's what you were pointing at?
Generally these arguments are confusing to me, I'm not sure what is supported where.
Sorry I meant the return_set
and which dataset types supported both, etc. But it's much clearer now!
Support return_set_frac and n_samples for all datasets
NOTE: Commented out
tests.test_build_graph.test_modular_arithmetic_rotate_final_layer_invariance
must be addressed before merging. @stefan-apollo would you mind looking at this, it doesn't seem to be passing for most of the configurations.Description
@overload
calls, yayy.Note, it's possible that I can and should also define
source
,name
andreturn_set
inside the DatasetConfig base class, but I don't know how this works with pydantic when all the subclasses have different types and they must be defined in all subclasses.Related Issue
Closes #234; Closes #229; Closes #238
How Has This Been Tested?
tests.test_loader.test_get_data_subset
.Does this PR introduce a breaking change?
Yes.
return_set
argument.