search

huggingface / datasets

🤗 The largest hub of ready-to-use datasets for ML models with fast, easy-to-use and efficient data manipulation tools
https://huggingface.co/docs/datasets
Apache License 2.0
18.65k stars 2.56k forks source link

Require Pillow >= 9.4.0 to avoid AttributeError when loading image dataset #6883

Closed albertvillanova closed 4 weeks ago

albertvillanova commented 1 month ago

Require Pillow >= 9.4.0 to avoid AttributeError when loading image dataset.

The PIL.Image.ExifTags that we use in our code was implemented in Pillow-9.4.0: https://github.com/python-pillow/Pillow/commit/24a5405a9f7ea22f28f9c98b3e407292ea5ee1d3

The bug #6881 was introduced in datasets-2.19.0 by this PR:

Fix #6881.

HuggingFaceDocBuilderDev commented 1 month ago

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

albertvillanova commented 4 weeks ago

Do you think this is worth making a patch release for? CC: @huggingface/datasets

github-actions[bot] commented 4 weeks ago
Show benchmarks PyArrow==8.0.0
Show updated benchmarks! ### Benchmark: benchmark_array_xd.json | metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence | |--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| | new / old (diff) | 0.005764 / 0.011353 (-0.005589) | 0.004182 / 0.011008 (-0.006826) | 0.064520 / 0.038508 (0.026012) | 0.034260 / 0.023109 (0.011151) | 0.245677 / 0.275898 (-0.030221) | 0.277889 / 0.323480 (-0.045591) | 0.004569 / 0.007986 (-0.003417) | 0.002905 / 0.004328 (-0.001423) | 0.049346 / 0.004250 (0.045095) | 0.050529 / 0.037052 (0.013476) | 0.264718 / 0.258489 (0.006229) | 0.295705 / 0.293841 (0.001864) | 0.028144 / 0.128546 (-0.100402) | 0.011048 / 0.075646 (-0.064598) | 0.206290 / 0.419271 (-0.212982) | 0.035886 / 0.043533 (-0.007647) | 0.245038 / 0.255139 (-0.010101) | 0.269835 / 0.283200 (-0.013365) | 0.018927 / 0.141683 (-0.122756) | 1.136536 / 1.452155 (-0.315619) | 1.183256 / 1.492716 (-0.309460) | ### Benchmark: benchmark_getitem\_100B.json | metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row | |--------|---|---|---|---| | new / old (diff) | 0.115372 / 0.018006 (0.097366) | 0.315471 / 0.000490 (0.314982) | 0.000238 / 0.000200 (0.000038) | 0.000043 / 0.000054 (-0.000012) | ### Benchmark: benchmark_indices_mapping.json | metric | select | shard | shuffle | sort | train_test_split | |--------|---|---|---|---|---| | new / old (diff) | 0.021201 / 0.037411 (-0.016210) | 0.070374 / 0.014526 (0.055848) | 0.077557 / 0.176557 (-0.099000) | 0.124713 / 0.737135 (-0.612423) | 0.078850 / 0.296338 (-0.217489) | ### Benchmark: benchmark_iterating.json | metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 | |--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| | new / old (diff) | 0.278674 / 0.215209 (0.063465) | 2.739597 / 2.077655 (0.661942) | 1.438214 / 1.504120 (-0.065906) | 1.326373 / 1.541195 (-0.214822) | 1.370961 / 1.468490 (-0.097529) | 0.569160 / 4.584777 (-4.015617) | 2.411890 / 3.745712 (-1.333822) | 2.954073 / 5.269862 (-2.315788) | 1.816883 / 4.565676 (-2.748794) | 0.063123 / 0.424275 (-0.361152) | 0.005531 / 0.007607 (-0.002076) | 0.328184 / 0.226044 (0.102140) | 3.263083 / 2.268929 (0.994155) | 1.809159 / 55.444624 (-53.635465) | 1.535257 / 6.876477 (-5.341220) | 1.583428 / 2.142072 (-0.558644) | 0.642950 / 4.805227 (-4.162277) | 0.122240 / 6.500664 (-6.378424) | 0.044596 / 0.075469 (-0.030873) | ### Benchmark: benchmark_map_filter.json | metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow | |--------|---|---|---|---|---|---|---|---|---| | new / old (diff) | 0.999993 / 1.841788 (-0.841795) | 12.941508 / 8.074308 (4.867200) | 10.417519 / 10.191392 (0.226127) | 0.134345 / 0.680424 (-0.546079) | 0.014651 / 0.534201 (-0.519550) | 0.288660 / 0.579283 (-0.290623) | 0.274550 / 0.434364 (-0.159814) | 0.327785 / 0.540337 (-0.212553) | 0.422954 / 1.386936 (-0.963982) |
PyArrow==latest
Show updated benchmarks! ### Benchmark: benchmark_array_xd.json | metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence | |--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| | new / old (diff) | 0.006051 / 0.011353 (-0.005302) | 0.003926 / 0.011008 (-0.007082) | 0.051480 / 0.038508 (0.012972) | 0.036102 / 0.023109 (0.012992) | 0.273358 / 0.275898 (-0.002540) | 0.293261 / 0.323480 (-0.030219) | 0.004562 / 0.007986 (-0.003424) | 0.002918 / 0.004328 (-0.001410) | 0.050386 / 0.004250 (0.046135) | 0.048427 / 0.037052 (0.011375) | 0.280178 / 0.258489 (0.021689) | 0.314599 / 0.293841 (0.020758) | 0.030876 / 0.128546 (-0.097670) | 0.010571 / 0.075646 (-0.065076) | 0.058555 / 0.419271 (-0.360717) | 0.034974 / 0.043533 (-0.008559) | 0.266604 / 0.255139 (0.011465) | 0.284712 / 0.283200 (0.001512) | 0.020296 / 0.141683 (-0.121387) | 1.116760 / 1.452155 (-0.335395) | 1.157794 / 1.492716 (-0.334922) | ### Benchmark: benchmark_getitem\_100B.json | metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row | |--------|---|---|---|---| | new / old (diff) | 0.103777 / 0.018006 (0.085771) | 0.314267 / 0.000490 (0.313778) | 0.000226 / 0.000200 (0.000026) | 0.000047 / 0.000054 (-0.000008) | ### Benchmark: benchmark_indices_mapping.json | metric | select | shard | shuffle | sort | train_test_split | |--------|---|---|---|---|---| | new / old (diff) | 0.023837 / 0.037411 (-0.013574) | 0.082145 / 0.014526 (0.067619) | 0.090434 / 0.176557 (-0.086123) | 0.132096 / 0.737135 (-0.605040) | 0.092426 / 0.296338 (-0.203913) | ### Benchmark: benchmark_iterating.json | metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 | |--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| | new / old (diff) | 0.299554 / 0.215209 (0.084345) | 2.932382 / 2.077655 (0.854727) | 1.549994 / 1.504120 (0.045874) | 1.454944 / 1.541195 (-0.086251) | 1.474987 / 1.468490 (0.006497) | 0.586149 / 4.584777 (-3.998628) | 0.972118 / 3.745712 (-2.773594) | 2.991719 / 5.269862 (-2.278142) | 1.876365 / 4.565676 (-2.689311) | 0.065178 / 0.424275 (-0.359098) | 0.005114 / 0.007607 (-0.002493) | 0.353704 / 0.226044 (0.127660) | 3.500940 / 2.268929 (1.232012) | 1.965581 / 55.444624 (-53.479043) | 1.662594 / 6.876477 (-5.213883) | 1.702761 / 2.142072 (-0.439311) | 0.663879 / 4.805227 (-4.141348) | 0.120036 / 6.500664 (-6.380628) | 0.043195 / 0.075469 (-0.032274) | ### Benchmark: benchmark_map_filter.json | metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow | |--------|---|---|---|---|---|---|---|---|---| | new / old (diff) | 0.997690 / 1.841788 (-0.844098) | 13.448914 / 8.074308 (5.374606) | 10.132469 / 10.191392 (-0.058923) | 0.148493 / 0.680424 (-0.531930) | 0.016670 / 0.534201 (-0.517531) | 0.289708 / 0.579283 (-0.289575) | 0.132938 / 0.434364 (-0.301425) | 0.411425 / 0.540337 (-0.128913) | 0.430748 / 1.386936 (-0.956188) |

lhoestq commented 4 weeks ago

maybe not super important since it was not reported by users, this can be included in the next release

Eric2i commented 3 weeks ago

I observed the same AttributeError with Pillow == 10.3.0, while 9.4.0 works for me.

lhoestq commented 3 weeks ago

What's the error you're getting @Eric2i ?

On my side on 10.3.0 I could run this without errors:

import PIL.Image
PIL.Image.ExifTags.Base.Orientation is not None  # True
Eric2i commented 3 weeks ago

Sorry, false alarm. I double-checked that 10.3.0 is also good on my side. Thanks for your sample codes.

MaxHeuillet commented 1 day ago

I just faced the same bug after installing recent versions of Huggingface and datasets in a new environment. I solved it by uninstalling the recent version of Pillow and sticking to 9.4.0. pip uninstall Pillow pip install Pillow==9.4.0