search

pivotal-cf / bbr-pcf-pipeline-tasks

Collection of Concourse tasks for backing up a Tanzu Application Service (TAS) installation using BOSH Backup and Restore (BBR)
Apache License 2.0
37 stars 63 forks source link

failed to find instances for deployment cf-XXXXX: failed to check os: ssh.Run failed: ssh.Stream failed: ssh.Dial failed: dial tcp 192.168.1.59:22 #41

Closed akamalov closed 5 years ago

akamalov commented 5 years ago

Greetings,

Having a very strange behavior with Concourse. So, I’ve deployed Concourse though BOSH as part of deployment inside BOSH director. I was able to launch BBR job on it as well. So far so good. Now, out of three components of Concourse BBR pipeline, two finish without a problem:

export-om-installation
bbr-backup-director

However, bbr-backup-ert fails. Very curious error:

Using Ops Manager credentials
Status: 200 OK
Cache-Control: no-cache, no-store
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Fri, 21 Dec 2018 21:48:16 GMT
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: nginx/1.4.6 (Ubuntu)
Strict-Transport-Security: max-age=15552000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Request-Id: 29966c32-bac4-4c24-a014-f568cbad8821
X-Runtime: 1.019136
X-Xss-Protection: 1; mode=block
Retreving BBR client credentials for BOSH Director
Status: 200 OK
Cache-Control: no-cache, no-store
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Fri, 21 Dec 2018 21:48:17 GMT
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: nginx/1.4.6 (Ubuntu)
Strict-Transport-Security: max-age=15552000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Request-Id: 49f3c396-e933-498a-bff7-d530a147e5fd
X-Runtime: 1.341792
X-Xss-Protection: 1; mode=block
Status: 200 OK
Cache-Control: no-cache, no-store
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Fri, 21 Dec 2018 21:48:20 GMT
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: nginx/1.4.6 (Ubuntu)
Strict-Transport-Security: max-age=15552000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Request-Id: fcddd37e-d160-406b-9201-fed33bfb2df2
X-Runtime: 2.298233
X-Xss-Protection: 1; mode=block
Status: 200 OK
Cache-Control: no-cache, no-store
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Fri, 21 Dec 2018 21:48:22 GMT
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: nginx/1.4.6 (Ubuntu)
Strict-Transport-Security: max-age=15552000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Request-Id: 69c49dfb-c877-45dc-8bfa-b4b5593850c1
X-Runtime: 1.836832
X-Xss-Protection: 1; mode=block
Status: 200 OK
Cache-Control: no-cache, no-store
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Fri, 21 Dec 2018 21:48:23 GMT
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: nginx/1.4.6 (Ubuntu)
Strict-Transport-Security: max-age=15552000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Request-Id: 60474c65-effb-4f6c-bb1a-27b545e92f37
X-Runtime: 0.077863
X-Xss-Protection: 1; mode=block
Status: 200 OK
Cache-Control: no-cache, no-store
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Fri, 21 Dec 2018 21:48:25 GMT
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: nginx/1.4.6 (Ubuntu)
Strict-Transport-Security: max-age=15552000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Request-Id: 9d8e6205-82d7-44be-9803-b9c320961b61
X-Runtime: 1.924620
X-Xss-Protection: 1; mode=block
Status: 200 OK
Cache-Control: no-cache, no-store
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Fri, 21 Dec 2018 21:48:28 GMT
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: nginx/1.4.6 (Ubuntu)
Strict-Transport-Security: max-age=15552000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Request-Id: 3c4120c1-9e82-4d40-9208-a8dff6f7051e
X-Runtime: 2.847638
X-Xss-Protection: 1; mode=block
/tmp/build/e4b91406/ert-backup-artifact /tmp/build/e4b91406
[bbr] 2018/12/21 21:48:28 INFO - Looking for scripts
1 error occurred:
error 1:
failed to find instances for deployment cf-bd9479cdb9dee3d26d39: failed to check os: ssh.Run failed: ssh.Stream failed: ssh.Dial failed: dial tcp 192.168.1.59:22: connect: connection timed out

I don’t know where this IP 192.168.1.59 comes from. But with every backup it changes, while staying within the same RFC1918 address space (192.168.1.x). I hijacked into a container, couldn’t find anything but this little snippet:

$ fly42 -t boshcc hijack -j bbr-backup-CFLAB01-CG/bbr-backup-ert                                  
1: build #1, step: bbr-backup-ert, type: task
2: build #1, step: bbr-release, type: get
3: build #1, step: extract-binary, type: task
4: build #1, step: slack-alert, type: get
5: build #1, step: slack-alert, type: put
choose a container: 1
root@6a7867a6-08b8-4447-746c-90ab5b6e025c:/tmp/build/e4b91406# ls
bbr-pipeline-tasks-repo  bbr_client.json  bbr_keys.json  binary  bosh.crt  deployed_products.json  ert-backup-artifact
root@6a7867a6-08b8-4447-746c-90ab5b6e025c:/tmp/build/e4b91406# cd ert-backup-artifact/

root@6a7867a6-08b8-4447-746c-90ab5b6e025c:/tmp/build/e4b91406/ert-backup-artifact# ls
bbr-2018-12-21T21:50:45Z.err.log

root@6a7867a6-08b8-4447-746c-90ab5b6e025c:/tmp/build/e4b91406/ert-backup-artifact# cat bbr-2018-12-21T21\:50\:45Z.err.log
1 error occurred:
error 1:
dial tcp 192.168.1.59:22: connect: connection timed out
ssh.Dial failed
github.com/cloudfoundry-incubator/bosh-backup-and-restore/ssh.Connection.runInSession
       /tmp/build/80754af9/src/github.com/cloudfoundry-incubator/bosh-backup-and-restore/ssh/connection.go:148
github.com/cloudfoundry-incubator/bosh-backup-and-restore/ssh.Connection.Stream
       /tmp/build/80754af9/src/github.com/cloudfoundry-incubator/bosh-backup-and-restore/ssh/connection.go:85
github.com/cloudfoundry-incubator/bosh-backup-and-restore/ssh.Connection.Run

I don’t know what is it’s trying to do. It does not happen on any other foundations, however, it is important to tell that the other foundation Concourse ecosystem is not deployed using BOSH. That’s the difference. Other than that, all the pipeline settings are identical.

Here is my pipeline:

jobs:
- name: export-om-installation
  serial: true
  plan:
  - aggregate:
    - get: bbr-pipeline-tasks-repo
      tags:
      - ((concourse-worker-tag))
    - get: run-schedule
      tags:
      - ((concourse-worker-tag))
      trigger: true
  - task: export-om-installation
    tags:
    - ((concourse-worker-tag))
    file: bbr-pipeline-tasks-repo/tasks/export-om-installation/task.yml
    params:
      SKIP_SSL_VALIDATION: ((skip-ssl-validation))
      OPSMAN_URL: ((opsman-url))
      OPSMAN_USERNAME: {{opsman-username}}
      OPSMAN_PASSWORD: {{opsman-password}}
  - put: om-backup-artifact
    tags:
    - ((concourse-worker-tag))
    params:
      file: om-installation/installation.zip
      inputs:
        - name: export-om-installation
      outputs:
        - name: notify_message
  on_success:
    put: slack-alert
    tags:
    - ((concourse-worker-tag))
    params:
      channel: '#cloudeng'
      text: |
        The `BOSH BBR Export OM Installation` pipeline has Succeeded. Check it out at:
        $ATC_EXTERNAL_URL/teams/$BUILD_TEAM_NAME/pipelines/$BUILD_PIPELINE_NAME/jobs/$BUILD_JOB_NAME/builds/$BUILD_NAME
      silent: false
  on_failure:
    put: slack-alert
    tags:
    - ((concourse-worker-tag))
    params:
      channel: '#cloudeng'
      text: |
        The `BOSH BBR Export OM Installation` pipeline has FAILED. Please resolve any issues and ensure the pipeline lock was released. Check it out at:
        $ATC_EXTERNAL_URL/teams/$BUILD_TEAM_NAME/pipelines/$BUILD_PIPELINE_NAME/jobs/$BUILD_JOB_NAME/builds/$BUILD_NAME
      silent: false

- name: bbr-backup-ert
  serial: true
  plan:
  - get: run-schedule
    tags:
    - ((concourse-worker-tag))
    trigger: true
  - aggregate:
    - get: bbr-pipeline-tasks-repo
      tags:
      - ((concourse-worker-tag))
      trigger: false
    - get: bbr-release
      tags:
      - ((concourse-worker-tag))
      trigger: false
  - task: extract-binary
    tags:
    - ((concourse-worker-tag))
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          repository: cloudfoundrylondon/bbr-pipeline
          tag: release-candidate
      inputs:
      - name: bbr-release
      outputs:
      - name: binary
      run:
        path: sh
        args:
        - -c
        - |
          tar -xvf bbr-release/bbr*.tar
          cp releases/bbr binary/
  - task: bbr-backup-ert
    tags:
    - ((concourse-worker-tag))
    file: bbr-pipeline-tasks-repo/tasks/bbr-backup-ert/task.yml
    params:
      SKIP_SSL_VALIDATION: ((skip-ssl-validation))
      OPSMAN_URL: ((opsman-url))
      OPSMAN_USERNAME: ((opsman-username))
      OPSMAN_PASSWORD: ((opsman-password))
  - put: ert-backup-bucket
    tags:
    - ((concourse-worker-tag))
    params:
      file: ert-backup-artifact/ert-backup.tar
  on_success:
    put: slack-alert
    tags:
    - ((concourse-worker-tag))
    params:
      channel: '#cloudeng'
      text: |
        The `BOSH BBR ERT/PAS Backup` pipeline has Succeeded. Check it out at:
        $ATC_EXTERNAL_URL/teams/$BUILD_TEAM_NAME/pipelines/$BUILD_PIPELINE_NAME/jobs/$BUILD_JOB_NAME/builds/$BUILD_NAME
      silent: false
  on_failure:
    put: slack-alert
    tags:
    - ((concourse-worker-tag))
    params:
      channel: '#cloudeng'
      The `BOSH BBR ERT/PAS Backup` pipeline has FAILED. Please resolve any issues and ensure the pipeline lock was released. Check it out at:
        $ATC_EXTERNAL_URL/teams/$BUILD_TEAM_NAME/pipelines/$BUILD_PIPELINE_NAME/jobs/$BUILD_JOB_NAME/builds/$BUILD_NAME
      silent: false

- name: bbr-backup-director
  serial: true
  plan:
  - get: run-schedule
    tags:
    - ((concourse-worker-tag))
    trigger: true
  - aggregate:
    - get: bbr-pipeline-tasks-repo
      tags:
      - ((concourse-worker-tag))
      trigger: false
    - get: bbr-release
      tags:
      - ((concourse-worker-tag))
      trigger: true
  - task: extract-binary
    tags:
    - ((concourse-worker-tag))
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          repository: cloudfoundrylondon/bbr-pipeline
          tag: release-candidate
      inputs:
      - name: bbr-release
      outputs:
      - name: binary
      run:
        path: sh
        args:
        - -c
        - |
          tar -xvf bbr-release/bbr*.tar
          cp releases/bbr binary/

  - task: bbr-backup-director
    tags:
    - ((concourse-worker-tag))
    file: bbr-pipeline-tasks-repo/tasks/bbr-backup-director/task.yml
    params:
      SKIP_SSL_VALIDATION: ((skip-ssl-validation))
      OPSMAN_URL: ((opsman-url))
      OPSMAN_USERNAME: ((opsman-username))
      OPSMAN_PASSWORD: ((opsman-password))
  - put: director-backup-bucket
    tags:
    - ((concourse-worker-tag))
    params:
      file: director-backup-artifact/director-backup.tar
  on_success:
    put: slack-alert
    tags:
    - ((concourse-worker-tag))
    params:
      channel: '#cloudeng'
      text: |
        The `BOSH BBR Director Backup` pipeline has Succeeded. Check it out at:
        $ATC_EXTERNAL_URL/teams/$BUILD_TEAM_NAME/pipelines/$BUILD_PIPELINE_NAME/jobs/$BUILD_JOB_NAME/builds/$BUILD_NAME
      silent: false
  on_failure:
    put: slack-alert
    tags:
    - ((concourse-worker-tag))
    params:
      channel: '#cloudeng'
      The `BOSH BBR Director BAckup` pipeline has FAILED. Please resolve any issues and ensure the pipeline lock was released. Check it out at:
        $ATC_EXTERNAL_URL/teams/$BUILD_TEAM_NAME/pipelines/$BUILD_PIPELINE_NAME/jobs/$BUILD_JOB_NAME/builds/$BUILD_NAME
      silent: false

resource_types:
- name: pivnet
  tags:
  - ((concourse-worker-tag))
  type: docker-image
  source:
    repository: pivotalcf/pivnet-resource
    tag: latest-final
- name: slack-notification
  tags:
  - ((concourse-worker-tag))
  type: docker-image
  source:
    repository: cfcommunity/slack-notification-resource

resources:
- name: slack-alert
  tags:
  - ((concourse-worker-tag))
  type: slack-notification
  source:
    url: ((slack-webhook))

- name: bbr-pipeline-tasks-repo
  type: git
  tags:
  - ((concourse-worker-tag))
  source:
    uri: ssh://git@gitlab.localnet.local/cf/bbr-pcf-pipeline-tasks.git
    private_key: {{private_key}}
    branch: master

- name: om-backup-artifact
  tags:
  - ((concourse-worker-tag))
  type: s3
  source:
    bucket: ((backup-artifact-bucket))
    region_name: ((storage-region))
    endpoint: ((storage-endpoint))
    access_key_id: ((storage-access-key-id))
    secret_access_key: {{storage-secret-access-key}}
    versioned_file: installation.zip
    use_v2_signing: ((storage-use-v2-signing))
    disable_ssl: ((disable_ssl))

- name: ert-backup-bucket
  type: s3
  tags:
  - ((concourse-worker-tag))
  source:
    bucket: ((backup-artifact-bucket))
    region_name: ((storage-region))
    endpoint: ((storage-endpoint))
    access_key_id: ((storage-access-key-id))
    secret_access_key: {{storage-secret-access-key}}
    versioned_file: ert-backup.tar
    use_v2_signing: ((storage-use-v2-signing))
    disable_ssl: ((disable_ssl))

- name: director-backup-bucket
  tags:
  - ((concourse-worker-tag))
  type: s3
  source:
    bucket: ((backup-artifact-bucket))
    region_name: ((storage-region))
    endpoint: ((storage-endpoint))
    access_key_id: ((storage-access-key-id))
    secret_access_key: {{storage-secret-access-key}}
    versioned_file: director-backup.tar
    use_v2_signing: ((storage-use-v2-signing))
    disable_ssl: ((disable_ssl))

- name: run-schedule
  type: time
  tags:
  - ((concourse-worker-tag))
  source:
     location: America/New_York
     days: [Friday]
     start: 4:45 PM
     stop: 5:45 PM

- name: bbr-release
  tags:
  - ((concourse-worker-tag))
  type: pivnet
  source:
    api_token: {{pivnet-api-token}}
    product_slug: p-bosh-backup-and-restore
    product_version: ((BBR_major_minor_version))

Any thoughts ?

Thanks!!

cf-gitbot commented 5 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/162825044

The labels on this github issue will be updated when the story is started.

sattan66 commented 5 years ago

I am also facing the same issue now. Can you please let me know, how was this issue fixed.

Narenbommireddy commented 4 years ago

We are having the exact same issue. @akamalov can you please help with the solution if you have?

akamalov commented 4 years ago

I believe you need to resize your volume to larger volume size on your workers.

On Wed, Feb 5, 2020, 7:44 PM Narenbommireddy notifications@github.com wrote:

We are having the exact same issue. @akamalov https://github.com/akamalov can you please help with the solution if you have?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pivotal-cf/bbr-pcf-pipeline-tasks/issues/41?email_source=notifications&email_token=AB4FDSO5ZZLLPCYZQSPIEULRBNMPXA5CNFSM4GL6BMU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEK5QPEA#issuecomment-582682512, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4FDSMC7WDL7OZIJO5BLFDRBNMPXANCNFSM4GL6BMUQ .

Narenbommireddy commented 4 years ago

Thank you @akamalov .. Our BBR workers already has 500GB disk, will try to increase and see if that helps.

akamalov commented 4 years ago

I had to configure it to about 800GB to get it to work.

Alex

On Thu, Feb 6, 2020 at 8:44 AM Narenbommireddy notifications@github.com wrote:

Thank you @akamalov https://github.com/akamalov .. Our BBR workers already has 500GB disk, will try to increase and see if that helps.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pivotal-cf/bbr-pcf-pipeline-tasks/issues/41?email_source=notifications&email_token=AB4FDSJ2MC6TAZXA2SFUMJ3RBQH37A5CNFSM4GL6BMU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEK7INUY#issuecomment-582911699, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4FDSKTAI3AT57I3IP34XDRBQH37ANCNFSM4GL6BMUQ .

SergeyMuha commented 4 years ago

Hi I doesn't work for me. I have made disk 1TB ██fly -t rnd i -j bbr_backup/bbr_backup -b 2 1: build #2, step: bbr-backup-director, type: task 2: build #2, step: bbr-pipeline-tasks-repo, type: get 3: build #2, step: bbr-release, type: get 4: build #2, step: bbr_image, type: get 5: build #2, step: extract-bbr, type: task choose a container: 1 root@79297396-9b90-472e-72f1-9aedc3d84d7d:/tmp/build/cdcc260a# df -h Filesystem Size Used Avail Use% Mounted on /dev/loop2 967G 1.3G 964G 1% / tmpfs 16G 0 16G 0% /dev/shm /dev/sdb2 977G 4.0G 924G 1% /tmp/garden-init tmpfs 16G 0 16G 0% /sys/fs/cgroup /dev/loop2 967G 1.3G 964G 1% /scratch /dev/loop2 967G 1.3G 964G 1% /tmp/build/cdcc260a /dev/loop2 967G 1.3G 964G 1% /tmp/build/cdcc260a/bbr-pipeline-tasks-repo /dev/loop2 967G 1.3G 964G 1% /tmp/build/cdcc260a/binary /dev/loop2 967G 1.3G 964G 1% /tmp/build/cdcc260a/director-backup-artifact devtmpfs 16G 0 16G 0% /dev/tty tmpfs 16G 0 16G 0% /proc/scsi tmpfs 16G 0 16G 0% /sys/firmware root@79297396-9b90-472e-72f1-9aedc3d84d7d:/tmp/build/cdcc260a#

And still error root@79297396-9b90-472e-72f1-9aedc3d84d7d:/tmp/build/cdcc260a# cat director-backup-artifact/bbr-2020-02-26T04\:54\:08Z.err.log 1 error occurred: error 1: socks connect tcp localhost:5000->10.100.8.2:22: dial tcp 127.0.0.1:5000: connect: connection refused ssh.Dial failed github.com/cloudfoundry-incubator/bosh-backup-and-restore/ssh.Connection.runInSession /tmp/build/80754af9/bosh-backup-and-restore/ssh/connection.go:148 github.com/cloudfoundry-incubator/bosh-backup-and-restore/ssh.Connection.Stream /tmp/build/80754af9/bosh-backup-and-restore/ssh/connection.go:85 github.com/cloudfoundry-incubator/bosh-backup-and-restore/ssh.Connection.Run /tmp/build/80754af9/bosh-backup-and-restore/ssh/connection.go:77 github.com/cloudfoundry-incubator/bosh-backup-and-restore/ssh.SshRemoteRunner.FindFiles /tmp/build/80754af9/bosh-backup-and-restore/ssh/remote_runner.go:123 github.com/cloudfoundry-incubator/bosh-backup-and-restore/instance.(JobFinderFromScripts).findBBRScripts /tmp/build/80754af9/bosh-backup-and-restore/instance/job_finder.go:95 github.com/cloudfoundry-incubator/bosh-backup-and-restore/instance.(JobFinderFromScripts).FindJobs /tmp/build/80754af9/bosh-backup-and-restore/instance/job_finder.go:52 github.com/cloudfoundry-incubator/bosh-backup-and-restore/standalone.DeploymentManager.Find /tmp/build/80754af9/bosh-backup-and-restore/standalone/deployment_manager.go:53 github.com/cloudfoundry-incubator/bosh-backup-and-restore/orchestrator.(FindDeploymentStep).Run /tmp/build/80754af9/bosh-backup-and-restore/orchestrator/find_deployment_step.go:14 github.com/cloudfoundry-incubator/bosh-backup-and-restore/orchestrator.(Workflow).Run /tmp/build/80754af9/bosh-backup-and-restore/orchestrator/workflow.go:17 github.com/cloudfoundry-incubator/bosh-backup-and-restore/orchestrator.Backuper.Backup /tmp/build/80754af9/bosh-backup-and-restore/orchestrator/backuper.go:55 github.com/cloudfoundry-incubator/bosh-backup-and-restore/cli/command.DirectorBackupCommand.Action /tmp/build/80754af9/bosh-backup-and-restore/cli/command/director_backup.go:47 github.com/urfave/cli.HandleAction /go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:490 github.com/urfave/cli.Command.Run /go/pkg/mod/github.com/urfave/cli@v1.20.0/command.go:210 github.com/urfave/cli.(App).RunAsSubcommand /go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:379 github.com/urfave/cli.Command.startApp /go/pkg/mod/github.com/urfave/cli@v1.20.0/command.go:298 github.com/urfave/cli.Command.Run /go/pkg/mod/github.com/urfave/cli@v1.20.0/command.go:98 github.com/urfave/cli.(App).Run /go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:255 main.main /tmp/build/80754af9/bosh-backup-and-restore/cmd/bbr/main.go:73 runtime.main /usr/local/go/src/runtime/proc.go:203 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1357 ssh.Stream failed github.com/cloudfoundry-incubator/bosh-backup-and-restore/ssh.Connection.Stream /tmp/build/80754af9/bosh-backup-and-restore/ssh/connection.go:87 github.com/cloudfoundry-incubator/bosh-backup-and-restore/ssh.Connection.Run /tmp/build/80754af9/bosh-backup-and-restore/ssh/connection.go:77 github.com/cloudfoundry-incubator/bosh-backup-and-restore/ssh.SshRemoteRunner.FindFiles /tmp/build/80754af9/bosh-backup-and-restore/ssh/remote_runner.go:123 github.com/cloudfoundry-incubator/bosh-backup-and-restore/instance.(JobFinderFromScripts).findBBRScripts /tmp/build/80754af9/bosh-backup-and-restore/instance/job_finder.go:95 github.com/cloudfoundry-incubator/bosh-backup-and-restore/instance.(JobFinderFromScripts).FindJobs /tmp/build/80754af9/bosh-backup-and-restore/instance/job_finder.go:52 github.com/cloudfoundry-incubator/bosh-backup-and-restore/standalone.DeploymentManager.Find /tmp/build/80754af9/bosh-backup-and-restore/standalone/deployment_manager.go:53 github.com/cloudfoundry-incubator/bosh-backup-and-restore/orchestrator.(FindDeploymentStep).Run /tmp/build/80754af9/bosh-backup-and-restore/orchestrator/find_deployment_step.go:14 github.com/cloudfoundry-incubator/bosh-backup-and-restore/orchestrator.(Workflow).Run /tmp/build/80754af9/bosh-backup-and-restore/orchestrator/workflow.go:17 github.com/cloudfoundry-incubator/bosh-backup-and-restore/orchestrator.Backuper.Backup /tmp/build/80754af9/bosh-backup-and-restore/orchestrator/backuper.go:55 github.com/cloudfoundry-incubator/bosh-backup-and-restore/cli/command.DirectorBackupCommand.Action /tmp/build/80754af9/bosh-backup-and-restore/cli/command/director_backup.go:47 github.com/urfave/cli.HandleAction /go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:490 github.com/urfave/cli.Command.Run /go/pkg/mod/github.com/urfave/cli@v1.20.0/command.go:210 github.com/urfave/cli.(App).RunAsSubcommand /go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:379 github.com/urfave/cli.Command.startApp /go/pkg/mod/github.com/urfave/cli@v1.20.0/command.go:298 github.com/urfave/cli.Command.Run /go/pkg/mod/github.com/urfave/cli@v1.20.0/command.go:98 github.com/urfave/cli.(App).Run /go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:255 main.main /tmp/build/80754af9/bosh-backup-and-restore/cmd/bbr/main.go:73 runtime.main /usr/local/go/src/runtime/proc.go:203 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1357 ssh.Run failed github.com/cloudfoundry-incubator/bosh-backup-and-restore/ssh.Connection.Run /tmp/build/80754af9/bosh-backup-and-restore/ssh/connection.go:79 github.com/cloudfoundry-incubator/bosh-backup-and-restore/ssh.SshRemoteRunner.FindFiles /tmp/build/80754af9/bosh-backup-and-restore/ssh/remote_runner.go:123 github.com/cloudfoundry-incubator/bosh-backup-and-restore/instance.(JobFinderFromScripts).findBBRScripts /tmp/build/80754af9/bosh-backup-and-restore/instance/job_finder.go:95 github.com/cloudfoundry-incubator/bosh-backup-and-restore/instance.(JobFinderFromScripts).FindJobs /tmp/build/80754af9/bosh-backup-and-restore/instance/job_finder.go:52 github.com/cloudfoundry-incubator/bosh-backup-and-restore/standalone.DeploymentManager.Find /tmp/build/80754af9/bosh-backup-and-restore/standalone/deployment_manager.go:53 github.com/cloudfoundry-incubator/bosh-backup-and-restore/orchestrator.(FindDeploymentStep).Run /tmp/build/80754af9/bosh-backup-and-restore/orchestrator/find_deployment_step.go:14 github.com/cloudfoundry-incubator/bosh-backup-and-restore/orchestrator.(Workflow).Run /tmp/build/80754af9/bosh-backup-and-restore/orchestrator/workflow.go:17 github.com/cloudfoundry-incubator/bosh-backup-and-restore/orchestrator.Backuper.Backup /tmp/build/80754af9/bosh-backup-and-restore/orchestrator/backuper.go:55 github.com/cloudfoundry-incubator/bosh-backup-and-restore/cli/command.DirectorBackupCommand.Action /tmp/build/80754af9/bosh-backup-and-restore/cli/command/director_backup.go:47 github.com/urfave/cli.HandleAction /go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:490 github.com/urfave/cli.Command.Run /go/pkg/mod/github.com/urfave/cli@v1.20.0/command.go:210 github.com/urfave/cli.(App).RunAsSubcommand /go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:379 github.com/urfave/cli.Command.startApp /go/pkg/mod/github.com/urfave/cli@v1.20.0/command.go:298 github.com/urfave/cli.Command.Run /go/pkg/mod/github.com/urfave/cli@v1.20.0/command.go:98 github.com/urfave/cli.(App).Run /go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:255 main.main /tmp/build/80754af9/bosh-backup-and-restore/cmd/bbr/main.go:73 runtime.main /usr/local/go/src/runtime/proc.go:203 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1357 finding scripts failed on bosh/0 github.com/cloudfoundry-incubator/bosh-backup-and-restore/instance.(JobFinderFromScripts).findBBRScripts /tmp/build/80754af9/bosh-backup-and-restore/instance/job_finder.go:97 github.com/cloudfoundry-incubator/bosh-backup-and-restore/instance.(JobFinderFromScripts).FindJobs /tmp/build/80754af9/bosh-backup-and-restore/instance/job_finder.go:52 github.com/cloudfoundry-incubator/bosh-backup-and-restore/standalone.DeploymentManager.Find /tmp/build/80754af9/bosh-backup-and-restore/standalone/deployment_manager.go:53 github.com/cloudfoundry-incubator/bosh-backup-and-restore/orchestrator.(FindDeploymentStep).Run /tmp/build/80754af9/bosh-backup-and-restore/orchestrator/find_deployment_step.go:14 github.com/cloudfoundry-incubator/bosh-backup-and-restore/orchestrator.(Workflow).Run /tmp/build/80754af9/bosh-backup-and-restore/orchestrator/workflow.go:17 github.com/cloudfoundry-incubator/bosh-backup-and-restore/orchestrator.Backuper.Backup /tmp/build/80754af9/bosh-backup-and-restore/orchestrator/backuper.go:55 github.com/cloudfoundry-incubator/bosh-backup-and-restore/cli/command.DirectorBackupCommand.Action /tmp/build/80754af9/bosh-backup-and-restore/cli/command/director_backup.go:47 github.com/urfave/cli.HandleAction /go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:490 github.com/urfave/cli.Command.Run /go/pkg/mod/github.com/urfave/cli@v1.20.0/command.go:210 github.com/urfave/cli.(App).RunAsSubcommand /go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:379 github.com/urfave/cli.Command.startApp /go/pkg/mod/github.com/urfave/cli@v1.20.0/command.go:298 github.com/urfave/cli.Command.Run /go/pkg/mod/github.com/urfave/cli@v1.20.0/command.go:98 github.com/urfave/cli.(App).Run /go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:255 main.main /tmp/build/80754af9/bosh-backup-and-restore/cmd/bbr/main.go:73 runtime.main /usr/local/go/src/runtime/proc.go:203 runtime.goexit

akamalov commented 4 years ago

You can't just make disk bigger. You also need to make the volume that Concorse accesses (worker node) via docker, bigger as well. Try to restart your worker. If it does not work, you need to delete that volume and re-create the volume to expand to that size of the new disk size.

On Wed, Feb 26, 2020 at 12:09 AM Sergey Muha notifications@github.com wrote:

Hi I doesn't work for me. I have made disk 1TB ██fly -t rnd i -j bbr_backup/bbr_backup -b 2 1: build #2 https://github.com/pivotal-cf/bbr-pcf-pipeline-tasks/issues/2, step: bbr-backup-director, type: task 2: build #2 https://github.com/pivotal-cf/bbr-pcf-pipeline-tasks/issues/2, step: bbr-pipeline-tasks-repo, type: get 3: build #2 https://github.com/pivotal-cf/bbr-pcf-pipeline-tasks/issues/2, step: bbr-release, type: get 4: build #2 https://github.com/pivotal-cf/bbr-pcf-pipeline-tasks/issues/2, step: bbr_image, type: get 5: build #2 https://github.com/pivotal-cf/bbr-pcf-pipeline-tasks/issues/2, step: extract-bbr, type: task choose a container: 1 root@79297396-9b90-472e-72f1-9aedc3d84d7d:/tmp/build/cdcc260a# df -h Filesystem Size Used Avail Use% Mounted on /dev/loop2 967G 1.3G 964G 1% / tmpfs 16G 0 16G 0% /dev/shm /dev/sdb2 977G 4.0G 924G 1% /tmp/garden-init tmpfs 16G 0 16G 0% /sys/fs/cgroup /dev/loop2 967G 1.3G 964G 1% /scratch /dev/loop2 967G 1.3G 964G 1% /tmp/build/cdcc260a /dev/loop2 967G 1.3G 964G 1% /tmp/build/cdcc260a/bbr-pipeline-tasks-repo /dev/loop2 967G 1.3G 964G 1% /tmp/build/cdcc260a/binary /dev/loop2 967G 1.3G 964G 1% /tmp/build/cdcc260a/director-backup-artifact devtmpfs 16G 0 16G 0% /dev/tty tmpfs 16G 0 16G 0% /proc/scsi tmpfs 16G 0 16G 0% /sys/firmware root@79297396-9b90-472e-72f1-9aedc3d84d7d:/tmp/build/cdcc260a#

And still error root@79297396-9b90-472e-72f1-9aedc3d84d7d:/tmp/build/cdcc260a# cat director-backup-artifact/bbr-2020-02-26T04:54:08Z.err.log 1 error occurred: error 1: socks connect tcp localhost:5000->10.100.8.2:22: dial tcp 127.0.0.1:5000: connect: connection refused ssh.Dial failed

github.com/cloudfoundry-incubator/bosh-backup-and-restore/ssh.Connection.runInSession /tmp/build/80754af9/bosh-backup-and-restore/ssh/connection.go:148

github.com/cloudfoundry-incubator/bosh-backup-and-restore/ssh.Connection.Stream /tmp/build/80754af9/bosh-backup-and-restore/ssh/connection.go:85

github.com/cloudfoundry-incubator/bosh-backup-and-restore/ssh.Connection.Run /tmp/build/80754af9/bosh-backup-and-restore/ssh/connection.go:77

github.com/cloudfoundry-incubator/bosh-backup-and-restore/ssh.SshRemoteRunner.FindFiles /tmp/build/80754af9/bosh-backup-and-restore/ssh/remote_runner.go:123

github.com/cloudfoundry-incubator/bosh-backup-and-restore/instance.(*JobFinderFromScripts).findBBRScripts /tmp/build/80754af9/bosh-backup-and-restore/instance/job_finder.go:95

github.com/cloudfoundry-incubator/bosh-backup-and-restore/instance.(*JobFinderFromScripts).FindJobs /tmp/build/80754af9/bosh-backup-and-restore/instance/job_finder.go:52

github.com/cloudfoundry-incubator/bosh-backup-and-restore/standalone.DeploymentManager.Find

/tmp/build/80754af9/bosh-backup-and-restore/standalone/deployment_manager.go:53

github.com/cloudfoundry-incubator/bosh-backup-and-restore/orchestrator.(*FindDeploymentStep).Run

/tmp/build/80754af9/bosh-backup-and-restore/orchestrator/find_deployment_step.go:14

github.com/cloudfoundry-incubator/bosh-backup-and-restore/orchestrator.(*Workflow).Run /tmp/build/80754af9/bosh-backup-and-restore/orchestrator/workflow.go:17

github.com/cloudfoundry-incubator/bosh-backup-and-restore/orchestrator.Backuper.Backup /tmp/build/80754af9/bosh-backup-and-restore/orchestrator/backuper.go:55

github.com/cloudfoundry-incubator/bosh-backup-and-restore/cli/command.DirectorBackupCommand.Action

/tmp/build/80754af9/bosh-backup-and-restore/cli/command/director_backup.go:47 github.com/urfave/cli.HandleAction /go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:490 github.com/urfave/cli.Command.Run /go/pkg/mod/github.com/urfave/cli@v1.20.0/command.go:210 github.com/urfave/cli.(App).RunAsSubcommand /go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:379 github.com/urfave/cli.Command.startApp /go/pkg/mod/github.com/urfave/cli@v1.20.0/command.go:298 github.com/urfave/cli.Command.Run /go/pkg/mod/github.com/urfave/cli@v1.20.0/command.go:98 github.com/urfave/cli.(App).Run /go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:255 main.main /tmp/build/80754af9/bosh-backup-and-restore/cmd/bbr/main.go:73 runtime.main /usr/local/go/src/runtime/proc.go:203 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1357 ssh.Stream failed

github.com/cloudfoundry-incubator/bosh-backup-and-restore/ssh.Connection.Stream /tmp/build/80754af9/bosh-backup-and-restore/ssh/connection.go:87

github.com/cloudfoundry-incubator/bosh-backup-and-restore/ssh.Connection.Run /tmp/build/80754af9/bosh-backup-and-restore/ssh/connection.go:77

github.com/cloudfoundry-incubator/bosh-backup-and-restore/ssh.SshRemoteRunner.FindFiles /tmp/build/80754af9/bosh-backup-and-restore/ssh/remote_runner.go:123

github.com/cloudfoundry-incubator/bosh-backup-and-restore/instance.(*JobFinderFromScripts).findBBRScripts /tmp/build/80754af9/bosh-backup-and-restore/instance/job_finder.go:95

github.com/cloudfoundry-incubator/bosh-backup-and-restore/instance.(*JobFinderFromScripts).FindJobs /tmp/build/80754af9/bosh-backup-and-restore/instance/job_finder.go:52

github.com/cloudfoundry-incubator/bosh-backup-and-restore/standalone.DeploymentManager.Find

/tmp/build/80754af9/bosh-backup-and-restore/standalone/deployment_manager.go:53

github.com/cloudfoundry-incubator/bosh-backup-and-restore/orchestrator.(*FindDeploymentStep).Run

/tmp/build/80754af9/bosh-backup-and-restore/orchestrator/find_deployment_step.go:14

github.com/cloudfoundry-incubator/bosh-backup-and-restore/orchestrator.(*Workflow).Run /tmp/build/80754af9/bosh-backup-and-restore/orchestrator/workflow.go:17

github.com/cloudfoundry-incubator/bosh-backup-and-restore/orchestrator.Backuper.Backup /tmp/build/80754af9/bosh-backup-and-restore/orchestrator/backuper.go:55

github.com/cloudfoundry-incubator/bosh-backup-and-restore/cli/command.DirectorBackupCommand.Action

/tmp/build/80754af9/bosh-backup-and-restore/cli/command/director_backup.go:47 github.com/urfave/cli.HandleAction /go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:490 github.com/urfave/cli.Command.Run /go/pkg/mod/github.com/urfave/cli@v1.20.0/command.go:210 github.com/urfave/cli.(App).RunAsSubcommand /go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:379 github.com/urfave/cli.Command.startApp /go/pkg/mod/github.com/urfave/cli@v1.20.0/command.go:298 github.com/urfave/cli.Command.Run /go/pkg/mod/github.com/urfave/cli@v1.20.0/command.go:98 github.com/urfave/cli.(App).Run /go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:255 main.main /tmp/build/80754af9/bosh-backup-and-restore/cmd/bbr/main.go:73 runtime.main /usr/local/go/src/runtime/proc.go:203 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1357 ssh.Run failed

github.com/cloudfoundry-incubator/bosh-backup-and-restore/ssh.Connection.Run /tmp/build/80754af9/bosh-backup-and-restore/ssh/connection.go:79

github.com/cloudfoundry-incubator/bosh-backup-and-restore/ssh.SshRemoteRunner.FindFiles /tmp/build/80754af9/bosh-backup-and-restore/ssh/remote_runner.go:123

github.com/cloudfoundry-incubator/bosh-backup-and-restore/instance.(*JobFinderFromScripts).findBBRScripts /tmp/build/80754af9/bosh-backup-and-restore/instance/job_finder.go:95

github.com/cloudfoundry-incubator/bosh-backup-and-restore/instance.(*JobFinderFromScripts).FindJobs /tmp/build/80754af9/bosh-backup-and-restore/instance/job_finder.go:52

github.com/cloudfoundry-incubator/bosh-backup-and-restore/standalone.DeploymentManager.Find

/tmp/build/80754af9/bosh-backup-and-restore/standalone/deployment_manager.go:53

github.com/cloudfoundry-incubator/bosh-backup-and-restore/orchestrator.(*FindDeploymentStep).Run

/tmp/build/80754af9/bosh-backup-and-restore/orchestrator/find_deployment_step.go:14

github.com/cloudfoundry-incubator/bosh-backup-and-restore/orchestrator.(*Workflow).Run /tmp/build/80754af9/bosh-backup-and-restore/orchestrator/workflow.go:17

github.com/cloudfoundry-incubator/bosh-backup-and-restore/orchestrator.Backuper.Backup /tmp/build/80754af9/bosh-backup-and-restore/orchestrator/backuper.go:55

github.com/cloudfoundry-incubator/bosh-backup-and-restore/cli/command.DirectorBackupCommand.Action

/tmp/build/80754af9/bosh-backup-and-restore/cli/command/director_backup.go:47 github.com/urfave/cli.HandleAction /go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:490 github.com/urfave/cli.Command.Run /go/pkg/mod/github.com/urfave/cli@v1.20.0/command.go:210 github.com/urfave/cli.(App).RunAsSubcommand /go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:379 github.com/urfave/cli.Command.startApp /go/pkg/mod/github.com/urfave/cli@v1.20.0/command.go:298 github.com/urfave/cli.Command.Run /go/pkg/mod/github.com/urfave/cli@v1.20.0/command.go:98 github.com/urfave/cli.(App).Run /go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:255 main.main /tmp/build/80754af9/bosh-backup-and-restore/cmd/bbr/main.go:73 runtime.main /usr/local/go/src/runtime/proc.go:203 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1357 finding scripts failed on bosh/0

github.com/cloudfoundry-incubator/bosh-backup-and-restore/instance.(*JobFinderFromScripts).findBBRScripts /tmp/build/80754af9/bosh-backup-and-restore/instance/job_finder.go:97

github.com/cloudfoundry-incubator/bosh-backup-and-restore/instance.(*JobFinderFromScripts).FindJobs /tmp/build/80754af9/bosh-backup-and-restore/instance/job_finder.go:52

github.com/cloudfoundry-incubator/bosh-backup-and-restore/standalone.DeploymentManager.Find

/tmp/build/80754af9/bosh-backup-and-restore/standalone/deployment_manager.go:53

github.com/cloudfoundry-incubator/bosh-backup-and-restore/orchestrator.(*FindDeploymentStep).Run

/tmp/build/80754af9/bosh-backup-and-restore/orchestrator/find_deployment_step.go:14

github.com/cloudfoundry-incubator/bosh-backup-and-restore/orchestrator.(*Workflow).Run /tmp/build/80754af9/bosh-backup-and-restore/orchestrator/workflow.go:17

github.com/cloudfoundry-incubator/bosh-backup-and-restore/orchestrator.Backuper.Backup /tmp/build/80754af9/bosh-backup-and-restore/orchestrator/backuper.go:55

github.com/cloudfoundry-incubator/bosh-backup-and-restore/cli/command.DirectorBackupCommand.Action

/tmp/build/80754af9/bosh-backup-and-restore/cli/command/director_backup.go:47 github.com/urfave/cli.HandleAction /go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:490 github.com/urfave/cli.Command.Run /go/pkg/mod/github.com/urfave/cli@v1.20.0/command.go:210 github.com/urfave/cli.(App).RunAsSubcommand /go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:379 github.com/urfave/cli.Command.startApp /go/pkg/mod/github.com/urfave/cli@v1.20.0/command.go:298 github.com/urfave/cli.Command.Run /go/pkg/mod/github.com/urfave/cli@v1.20.0/command.go:98 github.com/urfave/cli.(App).Run /go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:255 main.main /tmp/build/80754af9/bosh-backup-and-restore/cmd/bbr/main.go:73 runtime.main /usr/local/go/src/runtime/proc.go:203 runtime.goexit

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pivotal-cf/bbr-pcf-pipeline-tasks/issues/41?email_source=notifications&email_token=AB4FDSJZ5AFN6CSQ4MJSXT3REX2Q7A5CNFSM4GL6BMU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEM62F2Y#issuecomment-591241963, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4FDSOCVIMY526JMVRQN2LREX2Q7ANCNFSM4GL6BMUQ .