Fix invalid blobstore credentials

[mrdavidlaing]

Michael Tekel says,

Blobstore error: Failed to find object 'b9e969e6-3ad5-4c6b-b39e-a03b471f4ea0', S3 response error: The AWS Access Key Id you provided does not exist in our records. Could you please check if the credentials in the release are correct and if not, replace them with ones that have access to blobstore?

[mrdavidlaing]

I have a suspicion that the credentials currently being used

---
blobstore:
  provider: s3
  options:
    bucket_name: logsearch-boshrelease
    access_key_id: AKIAIISVJZNKHAZZX3JQ
    secret_access_key: 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
    encryption_key: SHkpJ5MFx2PASpp0LhVh7e1O2V0Bv2
final_name: logsearch

might have been expired during our recent password changes.

Checking...

[sopel]

Sorry, I had inactivated that key just yesterday due to a resp. inquiry by @ryanholder on behalf of Toby - my analysis of those being not in use turns out to be wrong apparently, I'll reactivate them.

[mrdavidlaing]

Thanks - AWS send moaning emails when you publish AWS credentials on github; even when they are "read only" ones for a specific S3 bucket like this.

@wdneto - you have an idea whether you can publish a BOSH release without having to publish AWS credentials?

[sopel]

The key is active again - ironically I stumbled over the instructions for BOSH maintainers to provide AWS credentials later that day and shortly considered those might be used for this, but didn't follow up, sorry for the disruption.

[mrdavidlaing]

Michael,

Could you check that you can access the logsearch-boshrelease blobs now that the key has been reactivated, and reply to the Github issue with your results.

Thanks

David

On 8 May 2014 15:56, Steffen Opel notifications@github.com wrote:

The key is activated again - ironically I stumbled over the instructions for BOSH maintainers to provide AWS credentials later that day and shortly considered those might be used for this, but didn't follow up, sorry for the disruption.

— Reply to this email directly or view it on GitHubhttps://github.com/logsearch/logsearch-boshrelease/issues/22#issuecomment-42559947 .

David Laing logsearch.io - build your own open source cloud logging cluster http://davidlaing.com

[sopel]

For reference, the BOSH credential exposure issue/topic is tracked internally in https://github.com/cityindex/labs-operations/issues/174.

[mrdavidlaing]

I asked about this on the BOSH mailing list - https://groups.google.com/a/cloudfoundry.org/forum/#!topic/bosh-users/x1Pyb_CZxeM

[mrdavidlaing]

Hi David,

I can confirm it works now. Thanks for the quick fix.

Michael

[aristotelesneto]

You use a private.yml file in the config folder of the release, and add that to .gitignore (so its not published to github). Then you move some of the contents of final.yml / dev.yml into it, like so:

final.yml

---
final_name: logsearch
blobstore:
  provider: s3
  options:
    bucket_name: logsearch-boshrelease
    encryption_key: <snip>

private.yml

blobstore:
  s3:
    access_key_id: <snip>
    secret_access_key: <snip>

[aristotelesneto]

It's worth noting that encryption_keys would need to be published, as the client also needs to be able to decrypt the files - so not sure how valuable that is.

[mrdavidlaing]

I really just want all the blobs to be public (so encryption doesn't really make sense).

Is there any way to avoid having to publish any keys / encrytion or aws?

[aristotelesneto]

The example above will do just that - you will have a copy of the private.yml file, but only you (or whoever else has "publishing new release rights") - then you publish the rest of the bosh release to github (without the private.yml file), and everyone else should be able to consume the blobs from the blobstore (provided the bucket is publicly accessible).

If you don't want encryption you can just remove 'encryption_keys' from the final.yml file.

[mrdavidlaing]

I shall:

[dpb587]

This issue has been hanging around for a while...

I tried adding the following bucket policy to enable public access:

{
  "Sid": "PublicRead",
  "Effect": "Allow",
  "Principal": {
    "AWS": "*"
  },
  "Action": "s3:GetObject",
  "Resource": "arn:aws:s3:::logsearch-boshrelease/*"
}

I tested it out by cloning the repo and dropping the access/secret key from config/final.yml. It complained that encryption_key can only be used with access/secret key. After getting rid of encryption_key, it was able to successfully download the files, but checksums would not match, presumably because existing ones are encrypted.

So, the bucket policy will work fine for us, but seems like we need to switch away from the encryption key. To do that, I tried downloading the blobs with the existing config, then dropping the credentials and encryption key, then attempting to re-sync/upload [to a separate test bucket]. But bosh kept telling me they were all synced. I tried trashing the config/blob.yml metadata, but that didn't help. Then I tried going through the process of re-bosh add blob for everything, and then re-uploading all that. The config/blobs.yml file obviously changed with all the blobs getting new Object IDs (none were reused, which could cause problems for prior release w/ the encryption key).

With all that, I think we should do a single release switching from the access/secret/encryption keys. So the process would look something like:

git checkout -b v12 v11
vim config/final.yml # remove keys
rm config/blobs.yml
for DIR in $(ls ~/code/logsearch-boshrelease/blobs) ; do
  for NAME in $(ls ~/code/logsearch-boshrelease/blobs/$DIR) ; do
    bosh add blob ~/code/logsearch-boshrelease/blobs/$DIR/$NAME $DIR
  done
done
bosh upload blobs
git add -u
git commit -m 'switch to public access w/o encryption /issue #22'
bosh create release --final
git push origin v12

I think that would let us close out this issue. @mrdavidlaing, @wdneto - does this seem appropriate, or is my bosh ignorance getting the better of me?

[dpb587]

Actually, I don't think this accounted for package blobs. So, it's incomplete.

Also, I did try finding an article walking through this sort of key change... but didn't find anything.

[mrdavidlaing]

@dpb587 - thanks for all the legwork - I think we should proceed; favouring getting it done rather than keeping backwards compatibility.

@wdneto - any tips you could offer here to make the process easier would be much appreciated.

[aristotelesneto]

Can't say that I've tried doing this before, but the process sounds plausible - I'm not sure what the contents of ~/code is, but I'd assume its all the tarballs required for all the packages (there shouldn't be any other blob, other than package ones+filters?).

This will break backward compatibility only if the access keys are changed, or if trying to upload older releases from the "wrong tag". One could still do a git checkout v10, which means the config will still have the access/encryption keys, allowing that release version to be uploaded.

Having said that though, if it was me I'd be inclined to change aws keys, and pull all releases pre-key-removal.

[dpb587]

Okay, so I've rewritten the configs and blob references to non-encrypted, non-authenticated objects. I've released it as v13. I've tested it with a clean bosh-lite VM and clone of the repository. Additionally I've tested it with an existing repository which requires git cleaning the .final_builds to avoid checksum mismatch errors (documented in the commit message and release notes.

[mrdavidlaing]

@dpb587,

in the release notes you say:

Blobs from prior releases are still available in the S3 blobstore. If you need to switch to a prior release, you'll need to run through the same cleaning process.

I'd quite like to delete those old S3 keys that were published.

Is that possible?

[dpb587]

Through the bucket policy all objects are now publicly readable, so the old, encrypted blobs are still technically available. I'd imagine AWS will error if it's given an invalid credential, even if their credential is unnecessary for accessing the file.

Given the low priority for supporting access to those releases, I think we should just revoke the key and move along. Highly doubtful somebody will care for an older release, at this point. Or, perhaps we want a couple weeks to revoke it just to be on the safe side.

[dpb587]

Status report... I missed a step and am getting checksum errors when trying to upload the release to our boshes. I thought I verified that scenario on my local VM, but it's missing from my notes and I must not have.

Currently working on a fix and then a subsequent release...

[mrdavidlaing]

Agreed. We'll wait a few weeks and then remove the old releases.

[dpb587]

Okay... I couldn't figure this out today. With Release 13, the blobs were updated with new blob IDs and new checksums, but the object IDs remained the same. Locally this was fine because we could clear the cache and it could happily re-download and checksum. In the bosh director though, it already has the object ID, but sees that the checksum changed (between encrypted and non-encrypted) so it errors with no apparent option to clear the director's blob cache or overwrite it. That makes sense - it really should never happen, nor, for idempotency, should objects ever be changing. But we need a way around it. In my mind, the easiest solution for this one time event would be to make the bosh director "accidentally lose" its cached blobs so we can re-upload them. The only official way I've been able to find to do that would be to delete the release from director and recreate it, but in our case that would significantly disrupt a running deployment.

Today I also tried creating a new release, but using new IDs for the objects in an attempt to force unique objects into the system (same blobstore ID, but new object ID in .final_builds and referenced by releases/release-14.yml). I thought this way it would let both blobs live happily within the same repository even. However, when given an object ID, bosh regenerates the fingerprint to get the object ID and since the blob is the same, it goes back to the old object ID which just gets back to the checksum mismatch issues. So that idea didn't work out.

I don't have other ideas yet, but am still thinking and will get back to it tomorrow...

[aristotelesneto]

You can possibly get around pre-existing objects in the director by pushing a new update to the job/packages (ie a newline at the end of a file thats part of the job/package) - this should force bosh to create a new "version". Ugly, but I think it'd work.

[mrdavidlaing]

46 Is another option - I've manually removed all old blob references; and made a new release name - logsearch-1.

The advantages to this approach are:

1. It makes the "cannot rollback" nature clear
2. It reduces the number of blobs that need to be downloaded for a fresh setup

[dpb587]

So, I touched all the packages/jobs (thank you for that suggestion, @wdneto) to coerce new blob references and committed it as v14 and successfully pushed it out to my VM and both the boshes. I think we should close this and #46.

[aristotelesneto]

:+1: Glad I could help! :D

[aristotelesneto]

So is release 13 broken? If so, would it be better to reset numbering and re-create/re-tag release 13?

[dpb587]

It's technically only broken if you're trying to go from <13 to 13. It works fine as a clean, initial deployment. I agree it'd be a bit nicer if it didn't exist at all, but I'd prefer not rewriting tags or history.

[mrdavidlaing]

To clarrify;

1. You have a logsearch-N deployment, where N < 13.
   • trying to updated to logsearch-13 will fail with blob checksum complaints.
   • skipping logsearch-13 and going to logsearch-14 will work fine
2. Starting with a fresh BOSH, deploying logsearch-13 works fine, as does upgrading to logsearch-14
3. Starting with a fresh BOSH, deploying logsearch-14 works fine

If this is true, then the simplest advice is to just skip logsearch-13 (since its clearly unlucky). We should remove the logsearch-13 release; or at least update its release notes to say "don't use"

You agree?

[mrdavidlaing]

I see that is exactly what @dpb587 has done already in the logsearch-13 release notes:

Note: this release is broken, you should skip to v14.

[dpb587]

Closing this after finally having a fully successful release without blob issues.

[mrdavidlaing]

The logsearch-boshrelease-public AWS access keys used by logsearch-boshrelease < v12 have been disabled.

See https://github.com/logsearch/logsearch-boshrelease/issues/70