search

DataDog / kvexpress

## Auto-archived due to inactivity. ## Go program to move data in and out of Consul's KV store.
Apache License 2.0
129 stars 13 forks source link

The SHA256 of a blank key is always the same. #87

Closed darron closed 8 years ago

darron commented 8 years ago

Maybe I should add a warning that says:

"Hey - there's nothing at that location - make sure you're actually inserting."

See:

https://twitter.com/davidneudorfer/status/707228434590769152 https://gist.github.com/davidneudorfer/26467e742357a5d5c840

davidvasandani commented 8 years ago

@darron I'm trying another key and I get the same error.

# curl "http://localhost:8500/v1/kv/product-catalog/builds/latest?raw"
691

# KVEXPRESS_DEBUG=1 ./kvexpress out --verbose=true --key product-catalog/builds/latest --file test.json
2016-03-08T16:36:30Z: out: Checking cli flags.
2016-03-08T16:36:30Z: out: Required cli flags present.
2016-03-08T16:36:30Z: out: username='root'
2016-03-08T16:36:30Z: out: path='data' fullPath='kvexpress/product-catalog/builds/latest/data'
2016-03-08T16:36:30Z: out: path='checksum' fullPath='kvexpress/product-catalog/builds/latest/checksum'
2016-03-08T16:36:30Z: out: path='stop' fullPath='kvexpress/product-catalog/builds/latest/stop'
2016-03-08T16:36:30Z: out: path='kvexpress/locks/ccabc787fb1dff12fcd2c75b9dfe064266f670317b1efd1072cb6508822c3782/awsstage-helios-app1b00'
2016-03-08T16:36:30Z: out: server='localhost:8500' token='anonymous'
2016-03-08T16:36:30Z: out: action='consulGet' key='kvexpress/locks/ccabc787fb1dff12fcd2c75b9dfe064266f670317b1efd1072cb6508822c3782/awsstage-helios-app1b00'
2016-03-08T16:36:30Z: out: action='consulGet' key='kvexpress/product-catalog/builds/latest/stop'
2016-03-08T16:36:30Z: out: Stop Key is NOT present - continuing.
2016-03-08T16:36:30Z: out: action='consulGet' key='kvexpress/product-catalog/builds/latest/data'
2016-03-08T16:36:30Z: out: action='consulGet' key='kvexpress/product-catalog/builds/latest/checksum'
2016-03-08T16:36:30Z: out: length='1' minLength='10'
2016-03-08T16:36:30Z: out: longEnough='false'
2016-03-08T16:36:30Z: out: computedChecksum='e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
2016-03-08T16:36:30Z: out: checksum='' computedChecksum='e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
2016-03-08T16:36:30Z: out: checksumMatch='false'
2016-03-08T16:36:30Z: out: longEnough='no'
2016-03-08T16:36:30Z: out: checksumMismatch='yes'
davidvasandani commented 8 years ago

Make sure you're inserting with kvexpress in first.

OH! I didn't realize that. So it's not possible to export first. I'm not fully understanding the use case for kvexpress I guess.

darron commented 8 years ago

OK - you're trying to pull something from a key that's never been inserted with kvexpress - which is why you're getting an empty response.

If you look closely - you'll see that kvexpress is looking at the kvexpress/product-catalog/builds/latest/data key - and you're showing me the output of product-catalog/builds/latest - which is not a key that's inserted by kvexpress.

The key structure is detailed here.

The workflow is normally:

  1. kvexpress in - which prepares the keys - data and checksum
  2. kvexpress out - which pulls from that key.

You can pull from random keys by using kvexpress raw which is detailed here.

That workflow is then just:

  1. kvexpress raw

Make sense?

darron commented 8 years ago

NOTE: In this case - this actually wouldn't work because copy assumes an existing kvexpress key with data and checksum - just leaving it here so the thread makes sense.

Another thing you could do is:

  1. kvexpress copy the key from the original location to a kvexpress one - it will create the data and checksum keys. Command is detailed here.
  2. kvexpress out with the new location.

That way you get the benefit of checksum protection - length protection - etc.

You'd need to setup a watch on the initial key - so that if it ever changed it would be updated in the kvexpress location.

How are you inserting these items into the KV store?

davidvasandani commented 8 years ago

Manually at the moment.

While Iooking for a way to manage the KVs, I thought exporting as json, making changes, and uploading to Consul made the most sense for this use case.

I was confused how/why you would upload a file like https://git.io/vzrXH to consul, but its all beginning to make sense now. Thank you for walking me though this.

btw this worked:

KVEXPRESS_DEBUG=1 ./kvexpress raw --verbose=true --key product-catalog/builds/latest -l 1 --file test.json
darron commented 8 years ago

Perfect - yes - that's it.

We are using kvexpress to send larger files around. In our case we're getting data inserted via git2consul.

Here's how it works for us:

  1. git2consul pulls in our configuration repo and merges changes.
  2. We're using consul-template to watch that key - which pulls the data out, writes a file and runs kvexpress in.
  3. That changes the key - which all the nodes that need the file are watching and redownload.

We're going to probably adjust that to remove step 2 - but that's being done slowly in this branch. The idea is that step 2 will be done by a watch on the KV store and will do the steps of kvexpress in with the raw key.

I'll leave this open and add some logging around the particular checksum we see when a key's empty so it's less confusing to the next person.

darron commented 8 years ago

NOTE: We will be fixing this - the wontfix is a bug we're experiencing with github at the moment.