search

inducer / loopy

A code generator for array-based code on CPUs and GPUs
http://mathema.tician.de/software/loopy
MIT License
565 stars 71 forks source link

`memoize_on_disk` can produce filenames that are too long #803

Open wence- opened 10 months ago

wence- commented 10 months ago

Since https://github.com/inducer/loopy/pull/710 (perhaps), memoize_on_disk produces quite long filenames, for example:

.../.cache/pytools/pdict-v4-loopy-memoize-cache-buffer_array-LoopyKeyBuilder.LoopyKeyBuilder-v0-2022.1-islpy2023.1.2-cgen2020.1-8158afdb18f100fb21b6cd86c6178e3c5cd19058-v1-py3.10.12.final.0

Unfortunately, some filesystems have quite short name restrictions. e.g. we had a bug report where this failed because the cache was on an encrypted filesystem where the max filename length is 144 characters (rather than the usual ext4 limit of 255).

Would it make sense to hash the (long) prefix as well as the contents?

inducer commented 10 months ago

I'd be open to moving a bunch of that stuff from the directory name to the cache key. Perhaps it would make sense to retain pdict-v4-loopy-buffer_array-3.10? Want to cook up a PR?