search

kowainik / hash-store

Hash as cache
https://kowainik.github.io/projects/hash-store
Mozilla Public License 2.0
2 stars 1 forks source link

openBinaryFile: invalid argument (File name too long) #10

Open kfigiela opened 5 years ago

kfigiela commented 5 years ago

Smuggler fails with the error, but I narrowed the issue to this package. Looks like there is a limit on file name length (at least on macOS). In an example case below file name of cache file is 263 characters that exceeds 255 characters limit (see discussion on quora)

[141 of 614] Compiling Aaaaaaaaaaa.Aaaaaaaaaaaa.AAAaaaaaa.Aaaaaaaa.Aaaaaa.AaaaaaaaAaaaaaa ( /Users/kfigiela/restaumatic/restaumatic/apps/aaaaa/aaa/aaaaaaaaa/Aaaaaaaaaaa/Aaaaaaaaaaaa/Aaaaaaaaa/Aaaaaaaa/Aaaaaa/AaaaaaaaAaaaaaa.hs, /Users/kfigiela/restaumatic/restaumatic/apps/aaaaa/.stack-work/odir/Aaaaaaaaaaa/Iaaaaaaaaaaa/Aaaaaaaaa/Aaaaaaaa/Aaaaaa/AaaaaaaaAaaaaaa.o )
*** Exception: .smuggler/1299bb5828223884900ba5c6724cd43427bbe9eafcfb50d0930697f2e8747873bf82de40b18f3c35111fb9ab9885215cf775ba8703bbcae0abe6c739b4c6aca1-.Users.kfigiela.restaumatic.restaumatic.apps.aaaaa.aaa.aaaaaaaaa.Aaaaaaaaaaa.Aaaaaaaaaaaa.Aaaaaaaaa.Aaaaaaaa.Aaaaaa.AaaaaaaaAaaaaaa.hs: openBinaryFile: invalid argument (File name too long)

Obviously, i redacted module names, but the lengths were preserved.

chshersh commented 5 years ago

@kfigiela That's an interesting issue :thinking: I'm not sure about the proper solution for the problem. We're using Blake2b hash because it's the fastest hash and base16 encoding to have file names that work on every file system without problems. I would like to keep the focus of hash-store on performance and efficiency, so switching to another library will require some benchmarks.

If you're not using hash-store directly and you're interested in using smuggler only, then there are some plans on how to improve smuggler workflow and remove local cache (which includes removal of hash-store usage as a consequence). And this should solve your problem.

kfigiela commented 5 years ago

Hi! As a quick and dirty workaround I changed the algorithm to hash filenames also and use base58 encoding (but now as I think of it – this may cause unlikely issues on case-insensitive filesystems). Maybe it'd be sufficient to have a single hash filename <> file content and do not include original file name in hash? I don't think switching to another hash would help as there still may be a longer file name.

Anyway, I am mostly interested in Smuggler, so any solution works for me :)

chshersh commented 5 years ago

@kfigiela There is a benefit of keeping file names: it helps further readability and debugging various issues.

Regarding Smuggler: keep an eye on this issue, it should help with this problem :slightly_smiling_face: