Missing sanity/unit test for create_loader - Githubissues

huggingface / pixparse

Pixel Parsing. A reproduction of OCR-free end-to-end document understanding models with open data

11 stars 3 forks source link

Missing sanity/unit test for create_loader #4

Open molbap opened 1 year ago

molbap commented 1 year ago

Function create_loader https://github.com/huggingface/pixparse/blob/64257d721fb88600def794382f1736f36d98878c/src/pixparse/data/loader.py#L7C1-L34C18 is responsible for the shape of batches and overall training sanity. It should be tested

With a small test dataset, ensure a full enumerate gets us what is expected (tensor shapes, etc) This could be relevant for https://github.com/huggingface/chug/blob/cfb16882e1058b37871b61fe8f76830cef3d8750/src/chug/webdataset/loader.py#L272C11-L272C11as well