Support for parquet datasets?

[tjthejuggler]

Reminder

• [X] I have read the README and searched the existing issues.

Reproduction

I am trying to finetune with a parquet dataset like this one:

https://huggingface.co/datasets/MadVoyager/stable_diffusion_instructional_dataset

Expected behavior

when i put it into the /data folder i expected the dataset to show up in the list of datasets, but it didn't.

System Info

• transformers version: 4.37.2
• Platform: Linux-6.5.0-15-generic-x86_64-with-glibc2.35
• Python version: 3.11.7
• Huggingface_hub version: 0.20.3
• Safetensors version: 0.4.2
• Accelerate version: 0.27.0
• Accelerate config: - compute_environment: LOCAL_MACHINE
  • distributed_type: NO
  • mixed_precision: bf16
  • use_cpu: False
  • debug: False
  • num_processes: 1
  • machine_rank: 0
  • num_machines: 1
  • gpu_ids: 0
  • rdzv_backend: static
  • same_network: True
  • main_training_function: main
  • downcast_bf16: no
  • tpu_use_cluster: False
  • tpu_use_sudo: False
  • tpu_env: []
• PyTorch version (GPU?): 2.2.0+cu121 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Others

Sorry if this is something simple or unrealistic or something!

[hiyouga]

Parquet is supported, see https://github.com/hiyouga/LLaMA-Factory?tab=readme-ov-file#data-preparation-optional

[tjthejuggler]

Thankyou!