Descriptive stats: `duckdb.IOException: IO Error: No files found that match the pattern ...`

[lhoestq]

Happens for imagenet-1k:

"Traceback (most recent call last):\n",
" File \"/src/services/worker/src/worker/job_manager.py\", line 168, in process\n job_result = self.job_runner.compute()\n",
" File \"/src/services/worker/src/worker/job_runners/split/descriptive_statistics.py\", line 467, in compute\n compute_descriptive_statistics_response(\n", "
File \"/src/services/worker/src/worker/job_runners/split/descriptive_statistics.py\", line 396, in compute_descriptive_statistics_response\n cat_column_stats: CategoricalStatisticsItem = compute_categorical_statistics(\n",
" File \"/src/services/worker/src/worker/job_runners/split/descriptive_statistics.py\", line 232, in compute_categorical_statistics\n categories: List[Tuple[int, int]] = con.sql(\n",
"duckdb.IOException: IO Error: No files found that match the pattern \"/storage/stats-cache/48010914651841-split-descriptive-statistics-imagenet-1k-3acf5c2b/default/test/*.parquet\"\n"

cc @polinaeterna

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[severo]

https://datasets-server.huggingface.co/statistics?dataset=imagenet-1k&config=default&split=train

gives

{"error":"IO Error: No files found that match the pattern \"/storage/stats-cache/43040372179808-split-descriptive-statistics-imagenet-1k-d5e07eb7/default/train/*.parquet\""}

[polinaeterna]

i think it's the case for partial datasets, i can work on this