search

huggingface / safetensors

Simple, safe way to store and distribute tensors
https://huggingface.co/docs/safetensors
Apache License 2.0
2.79k stars 189 forks source link

README is inaccurate #487

Closed judahrand closed 2 months ago

judahrand commented 3 months ago

System Info

NA

Information

Reproduction

The README states that Arrow requires decoding and then links to a page which is about Parquet (which does require decoding). Arrow and Parquet are two independent formats. Arrow's IPC format is not the same as Parquet. Could this be corrected?

Expected behavior

The README should refer to the Arrow format. The point about lack of (native)bf16 support is, I believe, still valid but the point about requiring decoding doesn't hold water I don't think.

julien-c commented 3 months ago

maybe cc @lhoestq too

lhoestq commented 3 months ago

Arrow doesn't require decoding indeed, I opened https://github.com/huggingface/safetensors/pull/488 to fix this passage in the readme

github-actions[bot] commented 2 months ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.