The task run-cats failed with "Not logged in."

norshtein commented 5 years ago

Environment

We are using Concourse v3.12 and latest cf-deployment-concourse-tasks(this repo) to setup our pipeline. The Concourse cluster is on Azure. One task in our pipeline is run-cats, its setting is:

- name: run-cats
  serial_groups: [main-group]
  serial: true
  plan:
  - aggregate:
    - {get: updated-integration-configs, trigger: true, resource: store, passed: [sleep]}
    - {get: cf-deployment-concourse-tasks}
    - {get: cf-acceptance-tests}
    - {get: cf-pipeline}
  - task: run-cats
    file: cf-deployment-concourse-tasks/run-cats/task.yml
    input_mapping:
      integration-config: updated-integration-configs
      cf-acceptance-tests: cf-acceptance-tests
    params:
      CONFIG_FILE_PATH: {{CONFIG_FILE_PATH}}
      NODES: 2
      SKIP_REGEXP: "transparently proxies both reserved characters and unsafe characters"

Issue

The run-cats task fails with many Not logged in. Use 'cf login' to log in. . Complete running log can be found here. In the running log, I found one item:

[2018-08-20 05:20:28.51 (UTC)]> cf auth admin [REDACTED] 
API endpoint: https://api.greentongyao.com
Authenticating...
Post https://login.greentongyao.com/oauth/token: dial tcp: lookup login.greentongyao.com on 168.63.129.16:53: read udp 10.254.0.122:46020->168.63.129.16:53: i/o timeout
FAILED

After this item, many Not logged in. Use 'cf login' to log in. are followed, which cause a single test case fails. "168.63.129.16" is Azure's DNS server, I use fly -t target intercept -j my-pipeline/run-cats to get into the container, running dig login.greentongyao.com manually, and I got:

; <<>> DiG 9.9.5-3ubuntu0.17-Ubuntu <<>> login.greentongyao.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40874
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1280
;; QUESTION SECTION:
;login.greentongyao.com.                IN      A

;; ANSWER SECTION:
login.greentongyao.com. 5       IN      A       104.210.59.119

;; Query time: 1 msec
;; SERVER: 168.63.129.16#53(168.63.129.16)
;; WHEN: Mon Aug 20 06:48:07 UTC 2018
;; MSG SIZE  rcvd: 67

Seems the DNS server can resolve the domain correctly. We have seen this issue for several times, and cannot find the root cause of it. Could you please have a look?

cf-gitbot commented 5 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/159879975

The labels on this github issue will be updated when the story is started.

Changdrew commented 5 years ago

Hi @norshtein, we just got around to reviewing this issue, we apologize for the delay. We are unsure of what this could be because we don't really deploy against Azure. You might have some luck asking in the cf-deployment slack channel here. Best of luck!

cloudfoundry / cf-deployment-concourse-tasks

The task run-cats failed with "Not logged in." #74

Environment

Issue