Open kipharris opened 5 years ago
hassenius
commented
5 years ago Thank you for reporting. Would you be able to share the tfvars you are deploying with (redacting any sensitive information of course) se we can use that to validate any fixes with? Also can you point us to the documentation you referred to? Ideally it should point to a tag in the repo so any changes (even improvements) will not render the document invalid...
kipharris
commented
5 years ago The following terraform.tfvars is the only thing that is unique in my environment. Otherwise I used a straight clone of the repo. Here's my vbls: (See attached file: terraform.tfvars)
The referenced Redbook document, just published on 01 May, is the "ICp System Admin's Guide" at http://www.redbooks.ibm.com/abstracts/sg248440.html?Open. Deployment using this repo's Terraform script is the topic of sections 2.4.3 and 2.4.4. The Redbook team has already jumped on this and is refreshing the publication. The instructions will direct the reader to clone the commit tagged with redbook_3.1.2, rather than the latest. They said it would happen with a few hours, so that update may already be in the text.
Feel free to let me know if I can be of further help.
Kip Harris IBM Cloud Labs +1 512-294-4731 (M) hkip@us.ibm.com
From: hassenius notifications@github.com To: ibm-cloud-architecture/terraform-icp-ibmcloud terraform-icp-ibmcloud@noreply.github.com Cc: Kip Harris hkip@us.ibm.com, Author author@noreply.github.com Date: 05/09/2019 04:21 PM Subject: Re: [ibm-cloud-architecture/terraform-icp-ibmcloud] Commit 55bfccd (remote registries) breaks deployment (#22)
Thank you for reporting. Would you be able to share the tfvars you are deploying with (redacting any sensitive information of course) se we can use that to validate any fixes with? Also can you point us to the documentation you referred to? Ideally it should point to a tag in the repo so any changes (even improvements) will not render the document invalid...
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.
kipharris
commented
5 years ago I just realized that this email which I sent earlier may be useful to you. I have a complete console log attached in my note to Drew Douglass, at bottom.
As i mentioned in my other note, the Redbook team has already implemented a documentation fix, by adding to their publication, instructions to clone the commit level that they tested with.
Kip Harris IBM Cloud Labs +1 512-294-4731 (M) hkip@us.ibm.com ----- Forwarded by Kip Harris/Austin/IBM on 05/09/2019 06:23 PM -----
From: Kip Harris/Austin/IBM To: Redbooks at IBM/Raleigh/IBM@IBMUS, Vasfi Gucer/Austin/IBM@IBMUS Cc: Richard Osowski/Durham/IBM@IBMUS, Jeffrey Kwong/Ontario/IBM@IBMCA Date: 05/09/2019 03:08 PM Subject: Potentially high imact defect affecting Redbook guidance and customer ability to deploy ICP
Hi guys,
I just opened issue #22 on the Terraform asset for ICP deployment onto the IBM Cloud here:
https://github.com/ibm-cloud-architecture/terraform-icp-ibmcloud/issues/22
I'm putting extra energy into communicating this, due to my concern about potential impact to the business. To recap, the automated Terraform deployment for ICP-ee on IBM Cloud is breaking, at least for me. I was able to work-around the issue by by backing up the repo to the last commit prior to the "remote registries" commit on 3/25. If I'm right, that "remote registries" commit (55bfccd) breaks the deployment (at least for the EE template).
If the defect is valid, then the guidance that our Redbook team just published in Chapter 2 of the "ICp System Admin's Guide" (http://www.redbooks.ibm.com/abstracts/sg248440.html?Open) is also broken. This would be painful, as we're trying like crazy to get people to adopt ICP. Our customers can't start working with ICP before they deploy it. Manual deployment is a long, expensive, error-prone process. Needless to say, automated deployment vastly lowers the pain + cost of installing the product.
I've written the defect up in github, per the URL for issue #22 above. A more verbose explanation of the issue, along with a complete console log, is in the note below. I wrote the note below before isolating the issue.
Kip Harris IBM Cloud Labs +1 512-294-4731 (M) hkip@us.ibm.com ----- Forwarded by Kip Harris/Austin/IBM on 05/09/2019 02:41 PM -----
From: Kip Harris/Austin/IBM To: Drew Douglass/Durham/IBM@IBMUS Date: 05/08/2019 12:47 PM Subject: Re: Your experience with deploying ICp-ee 3.1.2 using terraform?
Really appreciate the offer to look at the logs. See attached Word file. I use Word to preserve the color formatting, pls let me know if there's a better way to do that.
(See attached file: runlog.0506_1800.docx)
You'll see the errors about the /var storage towards the end. But you're right, the script continues execution past that, and for me eventually goes south on this:
module.icpprovision.null_resource.icp-install (remote-exec): TASK
[private-registry-pull-image : docker login registry] *****
module.icpprovision.null_resource.icp-install (remote-exec): Wednesday 08
May 2019 13:48:24 +0000 (0:00:00.317) 0:02:46.679 *
module.icpprovision.null_resource.icp-install (remote-exec): fatal:
[10.209.126.27]: FAILED! => changed=true
module.icpprovision.null_resource.icp-install (remote-exec): cmd: echo
'placeholder' | docker login -u placeholder --password-stdin
icpee1-mastr-23e4f5d0-1672377-dal13.lb.bluemix.net:8500 || docker login -u
placeholder -p placeholder
icpee1-mastr-23e4f5d0-1672377-dal13.lb.bluemix.net:8500
module.icpprovision.null_resource.icp-install (remote-exec): delta:
'0:00:00.132456'
module.icpprovision.null_resource.icp-install (remote-exec): end:
'2019-05-08 13:48:25.609477'
module.icpprovision.null_resource.icp-install (remote-exec): msg:
non-zero return code
module.icpprovision.null_resource.icp-install (remote-exec): rc: 1
module.icpprovision.null_resource.icp-install (remote-exec): start:
'2019-05-08 13:48:25.477021'
module.icpprovision.null_resource.icp-install (remote-exec): stderr: |-
module.icpprovision.null_resource.icp-install (remote-exec): Error
response from daemon: Get
https://icpee1-mastr-23e4f5d0-1672377-dal13.lb.bluemix.net:8500/v2/: EOF
module.icpprovision.null_resource.icp-install (remote-exec): WARNING!
Using --password via the CLI is insecure. Use --password-stdin.
module.icpprovision.null_resource.icp-install (remote-exec): Error
response from daemon: Get
https://icpee1-mastr-23e4f5d0-1672377-dal13.lb.bluemix.net:8500/v2/: EOF
module.icpprovision.null_resource.icp-install (remote-exec):
stderr_lines:
[other discussion deleted]
Kip Harris IBM Cloud Labs +1 512-294-4731 (M) hkip@us.ibm.com
I get the following failure from running the 55bfccd level (and presumably, later) of the repo (code which includes Jeff's 3/25 "Install from remote registries" commit).
module.icpprovision.null_resource.icp-install (remote-exec): cmd: echo 'placeholder' | docker login -u placeholder --password-stdin icpee1-mastr-23e4f5d0-1672377-dal13.lb.bluemix.net:8500 || docker login -u placeholder -p placeholder icpee1-mastr-23e4f5d0-1672377-dal13.lb.bluemix.net:8500
I am not installing from remote registries. I'm using the 'icp-ee' template.
If I backout to commit 62eb61a (3/18 commit by ososki "312 fix", also tagged as "redbook_312"), the terraform deployment runs to success.
If this defect is valid, I'll claim that it has high impact. The issue blocks ICP adopters from installing the product using the guidance in chapter 2 of the ICP Sys admin redbook sg248440.