NVIDIA / NeMo-Curator

Scalable toolkit for data curation
Apache License 2.0
331 stars 33 forks source link

Remove argparse from get_client function signature #12

Closed ryantwolf closed 1 month ago

ryantwolf commented 3 months ago

Main items:

Other changes:

Maghoumi commented 2 months ago

@ryantwolf I think we should have this functionality. I was trying to use the Curator through a notebook and had to do some gymnastics to create a client using get_client().

If this is too much work for now, maybe a simple workaround would be to have an equivalent of add_distributed_args(), but return a dictionary of the required arguments. This way the user can pass that dict instance to get_client() and still be able to use the code in a notebook environment.