No apparent retries on ICNR connect failure

tjmilligan commented 10 years ago

We have encountered an ICNR connect failure due to a busy resource but there don't seem to be any retries occurring. Log file for the build is below:

cut at dotted line

Started by user [8mha:AAAAmB+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAzuEgYe/dLi1CL94uzKksQCADe0jyLBAAAA[0mskytap user

Building on master in workspace /var/lib/jenkins/jobs/Skytap_Jenkins_Swarm_Slave_Build_Test_Environment/workspace

Creating Configuration from Template

Expanding environment variable ... $WORKSPACE/JenkinsSlaveBuildConfig.$BUILD_NUMBER=>/var/lib/jenkins/jobs/Skytap_Jenkins_Swarm_Slave_Build_Test_Environment/workspace/JenkinsSlaveBuildConfig.55 Template File: Config File: /var/lib/jenkins/jobs/Skytap_Jenkins_Swarm_Slave_Build_Test_Environment/workspace/JenkinsSlaveBuildConfig.55 Config Name: Template ID: 336941 Checking availability of template with id: 336941 Building request url ... Request URL: https://cloud.skytap.com/templates/336941 HTTP GET Request: GET https://cloud.skytap.com/templates/336941 HTTP/1.1 Executing Request: GET https://cloud.skytap.com/templates/336941 HTTP/1.1 HTTP/1.1 200 OK Template is available. Building request url ... Request URL: https://cloud.skytap.com/configurations/?template_id=336941 HTTP POST Request: POST https://cloud.skytap.com/configurations/?template_id=336941 HTTP/1.1 Executing Request: POST https://cloud.skytap.com/configurations/?template_id=336941 HTTP/1.1 HTTP/1.1 200 OK

Configuration successfully created and saved to file: /var/lib/jenkins/jobs/Skytap_Jenkins_Swarm_Slave_Build_Test_Environment/workspace/JenkinsSlaveBuildConfig.55

[workspace] $ /bin/sh -xe /tmp/hudson8545727524380264947.sh

+ /var/lib/jenkins/skytap_wait_for_update.py

Connecting to Network in another Configuration

Expanding environment variable ... $WORKSPACE/JenkinsSlaveBuildConfig.$BUILD_NUMBER=>/var/lib/jenkins/jobs/Skytap_Jenkins_Swarm_Slave_Build_Test_Environment/workspace/JenkinsSlaveBuildConfig.55 User provided file: /var/lib/jenkins/jobs/Skytap_Jenkins_Swarm_Slave_Build_Test_Environment/workspace/JenkinsSlaveBuildConfig.55 HTTP GET Request: GET https://cloud.skytap.com/configurations/1456852 HTTP/1.1 Executing Request: GET https://cloud.skytap.com/configurations/1456852 HTTP/1.1 HTTP/1.1 200 OK Searching configuration's networks for network: SwarmSlave Network Network Name: SwarmSlave Network Network Name Matched. Network ID: 1006628 HTTP GET Request: GET https://cloud.skytap.com/configurations/1456508 HTTP/1.1 Executing Request: GET https://cloud.skytap.com/configurations/1456508 HTTP/1.1 HTTP/1.1 200 OK Searching configuration's networks for network: EngBuild Network Network Name: EngBuild Network Network Name Matched. Network ID: 1006408 Sending network connection request for source: 1006628 to target: 1006408 HTTP POST Request: POST https://cloud.skytap.com/tunnels?source_network_id=1006628&target_network_id=1006408 HTTP/1.1 Executing Request: POST https://cloud.skytap.com/tunnels?source_network_id=1006628&target_network_id=1006408 HTTP/1.1 HTTP/1.1 423 Locked Skytap Error: The resource was busy. Try again later. Build step 'Execute Skytap Action' marked build as failure Finished: FAILURE

ptoma98 commented 10 years ago

Odd.. let me take a look tonight.

On Tue, Jan 21, 2014 at 12:32 PM, tmilligan-skytap <notifications@github.com

wrote:

We have encountered an ICNR connect failure due to a busy resource but there don't seem to be any retries occurring. Log file for the build is below: cut at dotted line

Started by user [8mha:AAAAmB+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAzuEgYe/dLi1CL94uzKksQCADe0jyLBAAAA[0mskytap user Building on master in workspace /var/lib/jenkins/jobs/Skytap_Jenkins_Swarm_Slave_Build_Test_Environment/workspace Creating Configuration from Template

Expanding environment variable ...

$WORKSPACE/JenkinsSlaveBuildConfig.$BUILD_NUMBER=>/var/lib/jenkins/jobs/Skytap_Jenkins_Swarm_Slave_Build_Test_Environment/workspace/JenkinsSlaveBuildConfig.55 Template File: Config File: /var/lib/jenkins/jobs/Skytap_Jenkins_Swarm_Slave_Build_Test_Environment/workspace/JenkinsSlaveBuildConfig.55 Config Name: Template ID: 336941 Checking availability of template with id: 336941 Building request url ... Request URL: https://cloud.skytap.com/templates/336941 HTTP GET Request: GET https://cloud.skytap.com/templates/336941 HTTP/1.1 Executing Request: GET https://cloud.skytap.com/templates/336941 HTTP/1.1 HTTP/1.1 200 OK Template is available. Building request url ... Request URL: https://cloud.skytap.com/configurations/?template_id=336941 HTTP POST Request: POST https://cloud.skytap.com/configurations/?template_id=336941 HTTP/1.1 Executing Request: POST https://cloud.skytap.com/configurations/?template_id=336941 HTTP/1.1 HTTP/1.1 200 OK Configuration successfully created and saved to file: /var/lib/jenkins/jobs/Skytap_Jenkins_Swarm_Slave_Build_Test_Environment/workspace/JenkinsSlaveBuildConfig.55

[workspace] $ /bin/sh -xe /tmp/hudson8545727524380264947.sh

/var/lib/jenkins/skytap_wait_for_update.py Connecting to Network in another Configuration

Expanding environment variable ...

$WORKSPACE/JenkinsSlaveBuildConfig.$BUILD_NUMBER=>/var/lib/jenkins/jobs/Skytap_Jenkins_Swarm_Slave_Build_Test_Environment/workspace/JenkinsSlaveBuildConfig.55 User provided file: /var/lib/jenkins/jobs/Skytap_Jenkins_Swarm_Slave_Build_Test_Environment/workspace/JenkinsSlaveBuildConfig.55 HTTP GET Request: GET https://cloud.skytap.com/configurations/1456852HTTP/1.1 Executing Request: GET https://cloud.skytap.com/configurations/1456852HTTP/1.1 HTTP/1.1 200 OK Searching configuration's networks for network: SwarmSlave Network Network Name: SwarmSlave Network Network Name Matched. Network ID: 1006628 HTTP GET Request: GET https://cloud.skytap.com/configurations/1456508HTTP/1.1 Executing Request: GET https://cloud.skytap.com/configurations/1456508HTTP/1.1 HTTP/1.1 200 OK Searching configuration's networks for network: EngBuild Network Network Name: EngBuild Network Network Name Matched. Network ID: 1006408 Sending network connection request for source: 1006628 to target: 1006408 HTTP POST Request: POST https://cloud.skytap.com/tunnels?source_network_id=1006628&target_network_id=1006408HTTP/1.1 Executing Request: POST https://cloud.skytap.com/tunnels?source_network_id=1006628&target_network_id=1006408HTTP/1.1 HTTP/1.1 423 Locked Skytap Error: The resource was busy. Try again later. Build step 'Execute Skytap Action' marked build as failure Finished: FAILURE

— Reply to this email directly or view it on GitHubhttps://github.com/ptoma98/skytap-cloud-plugin/issues/3 .

ptoma98 commented 10 years ago

Ok, I remember now. I'd written a method to check if a target network was available but commented it out. I think the problem is the network info returned from the API doesn't include any kind of busy status:

[ { "domain": "skytap.example", "gateway": "10.0.1.254", "id": "796112", "name": "ICNR Target Network", "network_type": "automatic", "primary_nameserver": null, "region": "US-West", "secondary_nameserver": null, "subnet": "10.0.1.0/24", "subnet_addr": "10.0.1.0", "subnet_size": 24, "tunnelable": true, "tunnels": [

],
"url": "https://cloud.skytap.com/configurations/1155788/networks/796112",
"vpn_attachments": [

]

} ]

So do I base it off of the busy status of the configuration itself?

Another thing I can do is skip the check altogether and just keep retrying the connect.

ptoma98 commented 10 years ago

Ok.. it looks like the network only obtains a 'status' field once its connected to something else.. either a VPN or another network. I can put in logic such that if there is no status field, to just assume the network is available.

{ "domain": "skytap.example", "gateway": "10.0.0.254", "id": "796112", "name": "ICNR Target Network", "network_type": "automatic", "primary_nameserver": null, "region": "US-West", "secondary_nameserver": null, "subnet": "10.0.0.0/24", "subnet_addr": "10.0.0.0", "subnet_size": 24, "tunnelable": true, "tunnels": [ { "error": null, "id": "tunnel-794010-831746", "source_network": { "domain": "skytap.example", "gateway": "10.0.0.254", "id": "796112", "name": "ICNR Target Network", "network_type": "automatic", "primary_nameserver": null, "region": "US-West", "secondary_nameserver": null, "subnet": "10.0.0.0/24", "subnet_addr": "10.0.0.0", "subnet_size": 24, "url": "https://cloud.skytap.com/configurations/1155788/networks/796112", "vpn_attachments": [ { "connected": true, "id": "796112-vpn-817994", "network": { "configuration_id": "1155788", "id": "796112", "network_name": "ICNR Target Network", "subnet": "10.0.0.0/24" }, "vpn": { "enabled": true, "id": "vpn-817994", "name": "Jenkins Test VPN", "remote_subnets": "10.1.0.0/20" } } ] },

_"status": "notbusy",

  "target_network": {
    "domain": "skytap.example",
    "gateway": "10.2.0.254",
    "id": "832854",
    "name": "Default Network",
    "network_type": "automatic",
    "primary_nameserver": null,
    "region": "US-West",
    "secondary_nameserver": null,
    "subnet": "10.2.0.0/24",
    "subnet_addr": "10.2.0.0",
    "subnet_size": 24,
    "url": "https://cloud.skytap.com/configurations/1204192/networks/832854",
    "vpn_attachments": [

    ]
  }
}

], "url": "https://cloud.skytap.com/configurations/1155788/networks/796112", "vpn_attachments": [ { "connected": true, "id": "796112-vpn-817994", "network": { "configuration_id": "1155788", "id": "796112", "network_name": "ICNR Target Network", "subnet": "10.0.0.0/24" }, "vpn": { "enabled": true, "id": "vpn-817994", "name": "Jenkins Test VPN", "remote_subnets": "10.1.0.0/20" } } ] }

tjmilligan commented 10 years ago

Unless it induces some other kind of error, I'd just base the retry attempt on the return result of the connect API call. If it returns with a busy message or status, do the retry.

Tom Milligan

Sent from my iPhone

On Jan 22, 2014, at 6:48 AM, "Patrick Toma" notifications@github.com<mailto:notifications@github.com> wrote:

Ok, I remember now. I'd written a method to check if a target network was available but commented it out. I think the problem is the network info returned from the API doesn't include any kind of busy status:

[ { "domain": "skytap.example", "gateway": "10.0.1.254", "id": "796112", "name": "ICNR Target Network", "network_type": "automatic", "primary_nameserver": null, "region": "US-West", "secondary_nameserver": null, "subnet": "10.0.1.0/24", "subnet_addr": "10.0.1.0", "subnet_size": 24, "tunnelable": true, "tunnels": [

], "url": "https://cloud.skytap.com/configurations/1155788/networks/796112", "vpn_attachments": [

]

} ]

So do I base it off of the busy status of the configuration itself?

Another thing I can do is skip the check altogether and just keep retrying the connect.

Reply to this email directly or view it on GitHubhttps://github.com/ptoma98/skytap-cloud-plugin/issues/3#issuecomment-33018484.

ptoma98 commented 10 years ago

I think the change you made to the executeHttpRequest method in SkytapUtils should solve this.

Let me know if there's anything you want me to do here.

ptoma98 / skytap-cloud-plugin