Hibb fails when executed: Invalid version bit (should be 1)

[jarirajari]

Hi! Thanks for this great piece of work - wouldn't be comfortable sending my users' passwords to anyone's API - not even as hashed to Troy's API. So I am trying to contribute by putting this to Docker format and sharing it via Docker hub. But I have noticed two things, one of which I could resolve on my own but the second one persists. When I try to run "hibb" the following error is produced (Docker image is based on Ubuntu 18.04): ` /root/go/bin/hibb

Loading Bloom filter from pwned-passwords-2.0.bloom... Invalid version bit (should be 1) ` Can't really figure this one out because I am not that familiar with go - could you help? This might not be a bug, but at this time, but still... Below is what I could find from cache. Thanks!

BR, Jari

--- SNIP ---

// Read loads a filter from a reader object.
func (s *BloomFilter) Read(input io.Reader) error {GoCover_0_656131393636393837346564.Count[0] = 1;
        bs8 := make([]byte, 8)

        if _, err := io.ReadFull(input, bs8); err != nil {GoCover_0_656131393636393837346564.Count[10] = 1;
                return err
        }

        GoCover_0_656131393636393837346564.Count[1] = 1;flags := binary.LittleEndian.Uint64(bs8)

        if flags&0xFF != 1 {GoCover_0_656131393636393837346564.Count[11] = 1;
                return fmt.Errorf("Invalid version bit (should be 1)")
        }

[adewes]

It seems there was a flag missing @jarirajari, I fixed that and also switched the bloom filter to the "official" version (I used my own fork while waiting for upstream changes). Should work now! Please close this if it does.

[jarirajari]

Thanks for the quick reply! I removed hibb server, then updated the code and re-did make for tool and server. I noticed changes in the build phase but still I get the same error: "Invalid version bit (should be 1)" One thing though, I am still using golang 1.11 currently as rebuilding to 10GB images takes quite a while, but I am updating it to 1.13 in the Dockerfile. For this version I noticed error " go get: warning: modules disabled by GO111MODULE=auto in GOPATH/src; " which I tackled with " export GO111MODULE="on" ". So to my eyes everything seems okay, but the error persists unfortunately.

[adewes]

Yes go modules don't work by default in Go 1.11, so if possible update to the latest go version, it should then work.

[jarirajari]

Switched to 1.13, verified with "go version", and did make again. I tried in many different ways, still the same error.

Another question, should I use "bloom" in /root/go/bin or the one that I installed from the Ubuntu repository?

[adewes]

Don't use the Ubuntu version of Bloom, it is out of date. You need to install the most recent version from master, then it will work. Hibb uses the new Bloom version, the CLI tool you use is the old version, hence the file formats are not compatible.

[adewes]

Does this work @jarirajari ?

[jarirajari]

Hi sorry, didn't catch your reply and then I was away for a while. Switched to Go 1.15 and now using built bloom and hibb. Redid everything from scratch and got exactly the same error response when executing hibb unfortunately.

[adewes]

No worries! That's weird, maybe you still have an old bloom version in your path that is being used instead of the new one? Can you run the bloom tool (without any arguments) and tell me the version it displays?

[jarirajari]

# bloom -v
Bloom Filter version 0.2.2

Procedure of setting up the image is in Dockerfile in https://github.com/jarirajari/hibp-bloomed-docker although in my tests I use test data (instead of real 10 GB data) as an input to the filter which seems to go okay.

[adewes]

Hey @jarirajari, I found the issue (the hibb server assumed the Bloom filter would be uncompressed) and fixed it. I also improved the README and Makefile and added a test method that won't need to build the entire filter. Let me know if that works for you now!

[jarirajari]

Hi, it works after two additional changes: where it reads "{print $$1}" there should be only one dollar sign.

Secondly, hibb default file name is not working:

# hibb
Loading Bloom filter from pwned-passwords-2.0.bloom...
open pwned-passwords-2.0.bloom: no such file or directory

But working with command "hibb -f pwned-passwords-2.0.bloom.gz". So default filename should be changed.

After that I got it working! Thank you!

[adewes]

I fixed the filename. The two $$ are correct though, as Make would otherwise treat $1 as the value of variable 1 (which is undefined).