bbl-up task doesn't work with specified Docker image

[Amit-PivotalLabs]

We tried to use the bbl-up task with the Docker image it specifies, and got a huge error message when bbl drove bosh-init to do something involving Ruby.

Adding:

RUN apt-get install -y build-essential zlibc zlib1g-dev \
        ruby ruby-dev openssl libxslt-dev libxml2-dev \
        libssl-dev libreadline6 libreadline6-dev libyaml-dev \
        libsqlite3-dev sqlite3

to the Dockerfile and using that custom image fixed things. That apt-get command matches the bosh-init installation docs.

Since the pipeline jobs in runtime-ci are not triggered by anything, including daily or weekly resources, it appears this error may never have been caught.

/cc @fushewokunze-pivotal @dsabeti @anEXPer

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/140205611

The labels on this github issue will be updated when the story is started.

[jaresty]

Hey @Amit-PivotalLabs,

Can you please share the error message you saw? We just re-ran the task against one of our environments and it succeeded at running bbl up against the existing environment (which I thought should fail if bosh-init weren't properly installed).

Thanks! Josh

[Amit-PivotalLabs]

https://gist.github.com/Amit-PivotalLabs/bfbae8b57686bedf3ddb0e821d1dc48c

On Sun, Feb 19, 2017 at 5:35 PM, Josh Aresty notifications@github.com wrote:

Hey @Amit-PivotalLabs https://github.com/Amit-PivotalLabs,

Can you please share the error message you saw? We just re-ran the task https://runtime.ci.cf-app.com/teams/main/pipelines/cf-deployment/jobs/setup-infrastructure-fresh/builds/53 against one of our environments and it succeeded at running bbl up against the existing environment (which I thought should fail if bosh-init weren't properly installed).

Thanks! Josh

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cloudfoundry/cf-deployment-concourse-tasks/issues/7#issuecomment-280968086, or mute the thread https://github.com/notifications/unsubscribe-auth/AC9t3veyIUYKKXVpgUCooGXmGFK8zN_iks5reO37gaJpZM4MFpNP .

[dsabeti]

Ok, I just tried this from the docker image on my own and it worked just fine. However, I did notice that most local package compilation gets skipped, including the ruby package:

Started validating
  Downloading release 'bosh'... Finished (00:00:27)
  Validating release 'bosh'... Finished (00:00:12)
  Downloading release 'bosh-google-cpi'... Finished (00:00:13)
  Validating release 'bosh-google-cpi'... Finished (00:00:05)
  Validating cpi release... Finished (00:00:00)
  Validating deployment manifest... Finished (00:00:00)
  Downloading stemcell... Finished (00:00:00)
  Validating stemcell... Finished (00:00:00)
Finished validating (00:01:00)

Started installing CPI
  Compiling package 'golang/b7c20853a73ad56fad8d60b980503e22d3ac7f43'... Finished (00:00:30)
  Compiling package 'bosh-google-cpi/2334a22dc4536f3bd86a8cd97e0127a5dfb86249'... Finished (00:00:23)
  Installing packages... Finished (00:00:04)
  Rendering job templates... Finished (00:00:00)
  Installing job 'google_cpi'... Finished (00:00:00)
Finished installing CPI (00:00:59)

Starting registry... Finished (00:00:00)
Uploading stemcell 'bosh-google-kvm-ubuntu-trusty-go_agent/3312.15'... Finished (00:01:05)

Started deploying
  Creating VM for instance 'bosh/0' from stemcell 'stemcell-6634af8a-ef43-40d4-4a66-e46d5f187590'... Finished (00:01:00)
  Waiting for the agent on VM 'vm-ec4e42a1-b6ab-498b-498c-017f6776bf19' to be ready... Finished (00:00:22)
  Creating disk... Finished (00:00:03)
  Attaching disk 'disk-dc58dcd1-cf70-43c0-6c09-dfd4926e4ed0' to VM 'vm-ec4e42a1-b6ab-498b-498c-017f6776bf19'... Finished (00:00:23)
  Rendering job templates... Finished (00:00:05)
  Compiling package 'ruby/589d4b05b422ac6c92ee7094fc2a402db1f2d731'... Skipped [Package already compiled] (00:00:02)
  Compiling package 'golang/b7c20853a73ad56fad8d60b980503e22d3ac7f43'... Finished (00:00:55)
  Compiling package 'mysql/b7e73acc0bfe05f1c6cbfd97bf92d39b0d3155d5'... Skipped [Package already compiled] (00:00:02)
  Compiling package 'libpq/09c8f60b87c9bd41b37b0f62159c9d77163f52b8'... Skipped [Package already compiled] (00:00:02)
  Compiling package 's3cli/8cbc6ee1b5acaac18c63fafc5989bd6911c9be83'... Skipped [Package already compiled] (00:01:20)
  Compiling package 'davcli/5f08f8d5ab3addd0e11171f739f072b107b30b8c'... Skipped [Package already compiled] (00:00:01)
  Compiling package 'nats/0155cf6be0305c9f98ba2e9e2503cd72da7c05c3'... Skipped [Package already compiled] (00:00:05)
  Compiling package 'health_monitor/6f71f76cfa6e85f14fe22a2114f6b3de73fe531d'... Skipped [Package already compiled] (00:00:07)
  Compiling package 'nginx/21e909d27fa69b3b2be036cdf5b8b293c6800158'... Skipped [Package already compiled] (00:00:01)
  Compiling package 'bosh-google-cpi/2334a22dc4536f3bd86a8cd97e0127a5dfb86249'... Finished (00:00:21)
  Compiling package 'postgres/4b9f6514001f7c3f7d4394920d6aced9435a3bbd'... Skipped [Package already compiled] (00:00:02)
  Compiling package 'registry/cf4e7bab4ca7d90bb18efa1451d7fa264b0b015b'... Skipped [Package already compiled] (00:00:06)
  Compiling package 'director/3b9f26bdc74c7b6b439577c1e4b2cf1941fe58c3'... Skipped [Package already compiled] (00:00:08)
  Updating instance 'bosh/0'... Finished (00:00:18)
  Waiting for instance 'bosh/0' to be running... Finished (00:00:16)
  Running the post-start scripts 'bosh/0'... Finished (00:00:00)
Finished deploying (00:05:51)

It seems to me that it's totally possible that we have a regression in this image, but that we haven't seen this in our pipelines because we end up skipping ruby compilation.

@cppforlife: When bosh-init says Skipped [Package already compiled], how can I find out where the package is cached? Keep in mind, this inside of a fresh docker image, so it seems like it couldn't be on the local file system. Is there a way to force bosh-init to re-compile the package?

Also, @Amit-PivotalLabs, what IaaS were you using? AWS? Another thing I'm noticing is that the GCP CPI is written golang, but the AWS CPI is written in Ruby. It could be an important factor as well. I'll take another stab at this tomorrow.

[Amit-PivotalLabs]

AWS. Good catch, that may be it. Do any cf-deployment jobs happen to be on AWS?

Amit

On Sun, Feb 19, 2017 at 7:26 PM David Sabeti notifications@github.com wrote:

Ok, I just tried this from the docker image on my own and it worked just fine. However, I did notice that most local package compilation gets skipped, including the ruby package:

Started validating Downloading release 'bosh'... Finished (00:00:27) Validating release 'bosh'... Finished (00:00:12) Downloading release 'bosh-google-cpi'... Finished (00:00:13) Validating release 'bosh-google-cpi'... Finished (00:00:05) Validating cpi release... Finished (00:00:00) Validating deployment manifest... Finished (00:00:00) Downloading stemcell... Finished (00:00:00) Validating stemcell... Finished (00:00:00) Finished validating (00:01:00)

Started installing CPI Compiling package 'golang/b7c20853a73ad56fad8d60b980503e22d3ac7f43'... Finished (00:00:30) Compiling package 'bosh-google-cpi/2334a22dc4536f3bd86a8cd97e0127a5dfb86249'... Finished (00:00:23) Installing packages... Finished (00:00:04) Rendering job templates... Finished (00:00:00) Installing job 'google_cpi'... Finished (00:00:00) Finished installing CPI (00:00:59)

Starting registry... Finished (00:00:00) Uploading stemcell 'bosh-google-kvm-ubuntu-trusty-go_agent/3312.15'... Finished (00:01:05)

Started deploying Creating VM for instance 'bosh/0' from stemcell 'stemcell-6634af8a-ef43-40d4-4a66-e46d5f187590'... Finished (00:01:00) Waiting for the agent on VM 'vm-ec4e42a1-b6ab-498b-498c-017f6776bf19' to be ready... Finished (00:00:22) Creating disk... Finished (00:00:03) Attaching disk 'disk-dc58dcd1-cf70-43c0-6c09-dfd4926e4ed0' to VM 'vm-ec4e42a1-b6ab-498b-498c-017f6776bf19'... Finished (00:00:23) Rendering job templates... Finished (00:00:05) Compiling package 'ruby/589d4b05b422ac6c92ee7094fc2a402db1f2d731'... Skipped [Package already compiled] (00:00:02) Compiling package 'golang/b7c20853a73ad56fad8d60b980503e22d3ac7f43'... Finished (00:00:55) Compiling package 'mysql/b7e73acc0bfe05f1c6cbfd97bf92d39b0d3155d5'... Skipped [Package already compiled] (00:00:02) Compiling package 'libpq/09c8f60b87c9bd41b37b0f62159c9d77163f52b8'... Skipped [Package already compiled] (00:00:02) Compiling package 's3cli/8cbc6ee1b5acaac18c63fafc5989bd6911c9be83'... Skipped [Package already compiled] (00:01:20) Compiling package 'davcli/5f08f8d5ab3addd0e11171f739f072b107b30b8c'... Skipped [Package already compiled] (00:00:01) Compiling package 'nats/0155cf6be0305c9f98ba2e9e2503cd72da7c05c3'... Skipped [Package already compiled] (00:00:05) Compiling package 'health_monitor/6f71f76cfa6e85f14fe22a2114f6b3de73fe531d'... Skipped [Package already compiled] (00:00:07) Compiling package 'nginx/21e909d27fa69b3b2be036cdf5b8b293c6800158'... Skipped [Package already compiled] (00:00:01) Compiling package 'bosh-google-cpi/2334a22dc4536f3bd86a8cd97e0127a5dfb86249'... Finished (00:00:21) Compiling package 'postgres/4b9f6514001f7c3f7d4394920d6aced9435a3bbd'... Skipped [Package already compiled] (00:00:02) Compiling package 'registry/cf4e7bab4ca7d90bb18efa1451d7fa264b0b015b'... Skipped [Package already compiled] (00:00:06) Compiling package 'director/3b9f26bdc74c7b6b439577c1e4b2cf1941fe58c3'... Skipped [Package already compiled] (00:00:08) Updating instance 'bosh/0'... Finished (00:00:18) Waiting for instance 'bosh/0' to be running... Finished (00:00:16) Running the post-start scripts 'bosh/0'... Finished (00:00:00) Finished deploying (00:05:51)

It seems to me that it's totally possible that we have a regression in this image, but that we haven't seen this in our pipelines because we end up skipping ruby compilation.

@cppforlife https://github.com/cppforlife: When bosh-init says Skipped [Package already compiled], how can I find out where the package is cached? Keep in mind, this inside of a fresh docker image, so it seems like it couldn't be on the local file system. Is there a way to force bosh-init to re-compile the package?

Also, @Amit-PivotalLabs https://github.com/Amit-PivotalLabs, what IaaS were you using? AWS? Another thing I'm noticing is that the GCP CPI is written golang, but the AWS CPI is written in Ruby. It could be an important factor as well. I'll take another stab at this tomorrow.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cloudfoundry/cf-deployment-concourse-tasks/issues/7#issuecomment-280979532, or mute the thread https://github.com/notifications/unsubscribe-auth/AC9t3oZUSXW--1zl_ey8G93fdCR7tJ-Lks5reQfRgaJpZM4MFpNP .

[dsabeti]

Yeah, but we haven't run that job in a month: https://runtime.ci.cf-app.com/teams/main/pipelines/cf-deployment/jobs/setup-infrastructure-upgrade/builds/15.

My guess is that if we re-bootstrapped that environment, it would fail.

[jaresty]

I just built a test pipeline for this and it does fall on AWS. 2017年2月19日(日) 19:29 David Sabeti notifications@github.com:

Yeah, but we haven't run that job in a month: https://runtime.ci.cf-app.com/teams/main/pipelines/cf-deployment/jobs/setup-infrastructure-upgrade/builds/15 .

My guess is that if we re-bootstrapped that environment, it would fail.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/cloudfoundry/cf-deployment-concourse-tasks/issues/7#issuecomment-280979938, or mute the thread https://github.com/notifications/unsubscribe-auth/ABQMNEAZKVzlOpddpPnYfweunYEWb2uwks5reQingaJpZM4MFpNP .

[dsabeti]

This story was accepted a while ago, so I'll close this issue.