anas-awadalla
commented
1 month ago Hi Han-Cheol! Thanks for your interest. I am not too sure why the huggingface default way of loading the dataset is returning this error. It seems that some samples might be missing some metadata. I have managed to load the data using the following code, hope this is helpful! Feel free to reopen if the issue persists.
from datasets import load_dataset
from tqdm import tqdm
from PIL import Image
import io
def read_mint_dataset(num_samples=None):
# Load the dataset
dataset = load_dataset("mlfoundations/MINT-1T-PDF-CC-2024-18", split="train", streaming=True)
# Cast columns to the correct type
dataset = dataset.cast_column('json', 'string')
# Create a progress bar
pbar = tqdm(desc="Reading samples", unit="sample")
# Iterate through the dataset
for i, sample in enumerate(dataset):
# Update the progress bar
pbar.update(1)
# Read the TIFF image as a list of PIL Images
tiff_image = sample['tiff']
if isinstance(tiff_image, Image.Image):
tiff = tiff_image
else:
tiff = Image.open(io.BytesIO(tiff_image))
images = []
for page in range(getattr(tiff, 'n_frames', 1)):
if hasattr(tiff, 'seek'):
tiff.seek(page)
image = tiff.copy()
if image.mode != 'RGB':
image = image.convert('RGB')
images.append(image)
# Print some information about the sample
print(f"Sample {i}:")
print(f" Text data: {sample['json']['texts']}")
print(f" Number of images: {len(images)}")
print("---")
if num_samples is not None and i + 1 >= num_samples:
break
if not isinstance(tiff_image, Image.Image):
tiff.close()
pbar.close()
if __name__ == "__main__":
# Read 10 samples from the dataset
read_mint_dataset(num_samples=100)
hancheolcho
Hi, First of all, thank you for releasing MINT-1T dataset :-)
I loaded one of MINT-1T datasets (MINT-1T-PDF-2023-06) but encountered the following error.
Error message shows that the output data has two additional fields: language_id_whole_page_fasttext and previous_word_count.
Do you have any idea how to fix it?
Best regards, Han-Cheol