`SHA1 mismatch` with non-ruby CLI `bosh create-release --force`

[cunnie]

I plan to submit a PR to fix.

bosh create-release --force --with-tarball
Blob download 'golang_1.6.2/go1.6.2.linux-amd64.tar.gz' (85 MB) (id: 4af848c7-c895-43cd-a9cd-b75a6bdea79a sha1: YjgzMThiMDlkZTA2MDc2ZDUzOTdlNmVjMThlYmVmM2I0NWNkMzE1ZA==) started

Blob download 'golang_1.6.2/go1.6.2.linux-amd64.tar.gz' (id: 4af848c7-c895-43cd-a9cd-b75a6bdea79a) failed

Getting blob '4af848c7-c895-43cd-a9cd-b75a6bdea79a' for path 'golang_1.6.2/go1.6.2.linux-amd64.tar.gz':
  Getting blob from inner blobstore:
    SHA1 mismatch. Expected YjgzMThiMDlkZTA2MDc2ZDUzOTdlNmVjMThlYmVmM2I0NWNkMzE1ZA==, got b8318b09de06076d5397e6ec18ebef3b45cd315d for blob /Users/cunnie/.bosh/tmp/bosh-s3-blob200517173

Exit code 1

Contents of blobs.yml

golang_1.6.2/go1.6.2.linux-amd64.tar.gz:
  object_id: 4af848c7-c895-43cd-a9cd-b75a6bdea79a
  sha: !binary |-
    YjgzMThiMDlkZTA2MDc2ZDUzOTdlNmVjMThlYmVmM2I0NWNkMzE1ZA==
  size: 84840658

[jianqiu]

@cunnie Hi, Why the sha1 value generated by non-ruby CLI is so different with that of ruby CLI? I am not sure whether the sha1 value generated by non-ruby CLI is compatible with ruby CLI? Thanks!

[cunnie]

@jianqiu :

Hi, Why the sha1 value generated by non-ruby CLI is so different with that of ruby CLI?

I am not sure why the Ruby CLI, in the BOSH SoftLayer's case, generated a Binary Data Language-Independent Type for YAML for the sha property. Until I reviewed the BOSH SoftLayer's CPI's blobs.yml, I was unaware that binary-encoding was possible (here are some releases I've worked on recently, and none of them have binary encoding: [0] [1] [2] [3]).

Others have struggled with Ruby's YAML being encoded as binary, and it appears that the locale has much to play with how Ruby chooses to encode its strings, which in turn affects the YAML output.

An interesting test would be to type the following command on the machine that generated blobs.yml:

ruby -e 'require "yaml"; puts YAML.dump(YAML.load(File.read("config/blobs.yml")))'

If the YAML output is encoded as binary, then a follow-on would be to determine the locale of the machine, e.g.:

$ locale
LANG=
LC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

I have attempted to replicate the binary-encoding within blobs.yml by setting my LANG variable, but have been unable to succeed — it is always generated as a string.

I am not sure whether the sha1 value generated by non-ruby CLI is compatible with ruby CLI?

The sha1 value is compatible, for two reasons:

1. The file/value was generated programmatically, not by hand: ruby -e 'require "yaml"; puts YAML.dump(YAML.load(File.read("config/blobs.yml")))'
2. It was tested: I built a development release using the Ruby CLI and using the non-binary-encoded blobs.yml that forced the download of the blob and there was no SHA1 mismatch:

± bc |master → origin ✓| → git co PR-blob-SHA1-mismatch
Switched to branch 'PR-blob-SHA1-mismatch'
± bc |PR-blob-SHA1-mismatch ✓| → cat config/blobs.yml
---
golang_1.6.2/go1.6.2.linux-amd64.tar.gz:
  object_id: 4af848c7-c895-43cd-a9cd-b75a6bdea79a
  sha: b8318b09de06076d5397e6ec18ebef3b45cd315d
  size: 84840658
± bc |PR-blob-SHA1-mismatch ✓| → git clean -xfd # remove any lingering artifacts
± bc |PR-blob-SHA1-mismatch ✓| → /Users/cunnie/.gem/ruby/2.2.3/bin/bosh create release --force --with-tarball
Syncing blobs...
golang_1.6.2/go1.6.2.linux-amd64.tar.gz downloaded
Please enter development release name: |bosh-softlayer-cpi|
...
Release name: bosh-softlayer-cpi
Release version: 2.3.18+dev.1
Release manifest: /Users/cunnie/workspace/bosh-softlayer-cpi-release/dev_releases/bosh-softlayer-cpi/bosh-softlayer-cpi-2.3.18+dev.1.yml
Release tarball (81.0M): /Users/cunnie/workspace/bosh-softlayer-cpi-release/dev_releases/bosh-softlayer-cpi/bosh-softlayer-cpi-2.3.18+dev.1.tgz

Thanks!

[jianqiu]

@cunnie @cppforlife

Thanks for your detailed information!

I remembered that blob.yml was generated by bosh upload blobs, but I forgot the version of bosh cli I used. Are you familiar with this command? Thanks!

[dpb587-pivotal]

@jianqiu: A while ago the YAML library used by the CLI was dumping binary values (binary values being the same as the decoded values, just difficult for us humans to parse). Later versions of the ruby CLI addressed that, and the upcoming CLI changes are expecting non-binary values.

[cunnie]

I remembered that blob.yml was generated by bosh upload blobs, but I forgot the version of bosh cli I used. Are you familiar with this command? Thanks!

Based on what @dpb587-pivotal said, I'm assuming the version of the Ruby CLI used to bosh upload blobs was one that inserted binary-encoded data into the blobs.yml file. @jianqiu : Is that enough to merge the pull request, or is there anything else you'd like to know?

[cunnie]

Closing. It's been lingering for four years, and it's either been fixed or will never get merged.