search

pierrec / xxHash

Pure Go implementation of xxHash (32 and 64 bits versions)
BSD 3-Clause "New" or "Revised" License
58 stars 17 forks source link

Reg : xxhsum (file and folder) #3

Closed chellasundar closed 7 years ago

chellasundar commented 8 years ago

Hi Yann,

I have tried using below mentioned command xxhsum directory/* But it will give the same checksum value entire directory.

We have so many sub folders and files in a single source directory.

Thanks, Chellasundar SR

pierrec commented 8 years ago

Hello,

Can you give an example of what you are doing? A quick check gives me:

./xxhsum/xxhsum ./*/*
0623fd8ad18e31fa ./xxHash32/example_test.go
f7bb327c75333a06 ./xxHash32/xxHash32.go
97afcb2a9c5241d2 ./xxHash32/xxHash32_test.go
abf27ee263b56927 ./xxHash64/example_test.go
1dcfaa51a6e134f4 ./xxHash64/xxHash64.go
7822463fccb01aa1 ./xxHash64/xxHash64_test.go
e4868b9de60a0612 ./xxhsum/main.go
db881b6476c45ff8 ./xxhsum/xxhsum
chellasundar commented 8 years ago

Hi Pierrec,

Hope all is well.

We are doing the test checksum for every files and folders.

We have so many subfolders and files is there. The folder size is example - 1 TB.

Already we done the test using md5deep command. But it will take at maximun 2 to 2.30 Hrs taken for 1 TB files and folder.

So we find out the solution from internet (xxHash - Speed 5.4 GB/s). Then we have installed and test it for 100 GB it will taken 8 Min only.

Only the dropback is we can't run the script subfolders and files. every foler we need to run the script. We can't run the script each folder and files.

Again we try to find out the solution SMHasher speed test, compiled using GCC 4.8.2, on Linux Mint 64-bits. It is (XXH64 - 13.8 GB/s) speed.

So only we are asking how to install SMHasher. What is the system and OS requirment to install.

The URL is : https://cyan4973.github.io/xxHash/

I have attached screen shot.

Kindly advice on this.

Thanks & Regards, Chellasundar SR System Administrator EFX - Prasad Corporation Ltd. ----- Original Message ----- From: Pierre Curto To: pierrec/xxHash Cc: chellasundar ; Author Sent: Wednesday, August 24, 2016 12:31 AM Subject: Re: [pierrec/xxHash] Reg : xxhsum (file and folder) (#3)

Hello,

Can you give an example of what you are doing? A quick check gives me:

./xxhsum/xxhsum .// 0623fd8ad18e31fa ./xxHash32/example_test.go f7bb327c75333a06 ./xxHash32/xxHash32.go 97afcb2a9c5241d2 ./xxHash32/xxHash32_test.go abf27ee263b56927 ./xxHash64/example_test.go 1dcfaa51a6e134f4 ./xxHash64/xxHash64.go 7822463fccb01aa1 ./xxHash64/xxHash64_test.go e4868b9de60a0612 ./xxhsum/main.go db881b6476c45ff8 ./xxhsum/xxhsum — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

pierrec commented 8 years ago

This repository is a Go (http://golang.org) implementation of the xxhash hashing algorithm. If you have questions specific to the C implementation, ask here: https://github.com/Cyan4973/xxHash/issues.

As for the Go implementation, it does support subfolders and will walk them to hash any file present there. It is slower than the C implementation, so experiment with it to see if the performance is acceptable for your use case.