HewlettPackard / docker-machine-oneview

HPE OneView plugin for docker machine (Not currently being maintained)
Apache License 2.0
23 stars 17 forks source link

How to specify on which blade docker host should be provisioned. #87

Open anandgagrani opened 8 years ago

anandgagrani commented 8 years ago

In the docker-machine create command (for oneview with docker) how do you specify which blade to use in the oneview infrastructure, for the docker host provision? I would like to provision host on specific blade but in the example command set below..there is no option to specify target blade. Also how to use the docker with oneview plugin...there is no clear document defining all the steps. if you can answer this question ..I am almost done with creating one document for everyone.

docker-machine create \

--engine-env HTTP_PROXY="" \ --engine-env HTTPS_PROXY="" \ --engine-env NO_PROXY="" \ --driver oneview \ --oneview-os-plan "RHEL71_DOCKER_18"\ --oneview-server-template "SP_template" \ --oneview-ov-user=Administrator \ --oneview-ov-password=\ --oneview-ov-domain=\ --oneview-ov-endpoint=https://15.xx.xx.xx/ \ --oneview-icsp-user=administrator \ --oneview-icsp-password=\ --oneview-icsp-endpoint=15.215.xx.xx\ --oneview-ssh-user=administrator \ --oneview-ilo-user=administrator \ --oneview-ilo-password=\ docker-ip-node-01

Also what does this mean: Name: "oneview-ilo-user",
Usage: "ILO User id that is used during ICSP server creation.",

I do not remember using any ilo when creating ICSP vm and then its template.

I am following the steps mentioned in document below(that is rather vague in actual steps) https://h20195.www2.hp.com/V2/getpdf.aspx/4AA6-2595ENW.pdf

sfc-gh-eraigosa commented 8 years ago

Hey, lots of questions, thats fantastic! OK, let me try to answer, let me know if i missed a question.

  1. How can i target a specific blade. Today we don't allow for you to pick a blade by serial or anything like that, but what you can do is assign a profile name to a blade prior to running docker-machine. If docker-machine plugin finds a blade with that name, then it will use it.
  2. How do i use the plugin. Just like any other plugin, place the docker-machine-driver-oneview in the system path, or same folder as the docker-machine executable. Then include the option --driver oneview. Running the command docker-machine --driver oneview create should produce output that comes from the oneview executable and confirms the plugin is talking to docker-machine.

Here is an example command that works for me well:

cat > credentials_andsettings.env << EOF
export ONEVIEW_APIVERSION=200

# you need this so we can create and join a new server to ICSP
# if the server already exist in ICSP we don't provision it
export ONEVIEW_ILO_USER=user
export ONEVIEW_ILO_PASSWORD=password

export ONEVIEW_ICSP_ENDPOINT=https://icsp.yournet.com
export ONEVIEW_ICSP_USER=user
export ONEVIEW_ICSP_PASSWORD=password
export ONEVIEW_ICSP_DOMAIN=domain

export ONEVIEW_OV_ENDPOINT=https://oneview.yournet.com
export ONEVIEW_OV_USER=user
export ONEVIEW_OV_PASSWORD=password
export ONEVIEW_OV_DOMAIN=domain 

export ONEVIEW_SSLVERIFY=true

export ONEVIEW_SERVER_TEMPLATE=oneviewtemplate
export ONEVIEW_OS_PLAN="icsp build plan"

# if you have a proxy for your data center
export proxy_enable=true

export proxy_config='http_proxy=http://proxy.yournet.com:8080/
https_proxy=https://proxy.yournet.com:8080/
no_proxy=/var/run/docker.sock,.proxy.yournet.com,localhost,127.0.0.1'

# proxy settings for --engine-evn options
export http_proxy=http://proxy.yournet.com:8080/
export https_proxy=https://proxy.yournet.com:8080/
export no_proxy=/var/run/docker.sock,.proxy.yournet.com,localhost,127.0.0.1

EOF

source credentials_adnsettings.env

# add the --engine-env options if you have a proxy
docker-machine create \
--oneview-os-plan "${ONEVIEW_OS_PLAN}" \
--oneview-server-template "${ONEVIEW_SERVER_TEMPLATE}" \
--oneview-public-connection-name "Name_of_public_network" \
--driver oneview \
--engine-env HTTP_PROXY="$http_proxy" \
--engine-env HTTPS_PROXY="$https_proxy" \
--engine-env NO_PROXY="$no_proxy" \
--engine-env http_proxy="$https_proxy" \
--engine-env https_proxy="$https_proxy" \
--engine-env no_proxy="$no_proxy" \
nodename
  1. Why do we have to use ilo user/password

This was needed to add a server to icsp. If we see that the server is already in icsp, we don't want to add it to docker-machine or provision it, to avoid destroying someone else server that might have already been provisioned. This way docker-machine always tries to add the new server to icsp (it should not exist in icsp prior to a create command).

Do let us know your progress and issues so we can make it easier.

ddefolo commented 8 years ago

Doesn't this come back to simply needing to support labels? See https://github.com/HewlettPackard/docker-machine-oneview/issues/47

If you supported label filters you can tag whatever server (or servers) you want to deploy with a unique label and then specify that label in your docker-machine call. That seems like a pretty minor change if it isn't supported yet.

Its also easier to use than direct server URI specification (which is the only reasonable way to directly specify I want to deploy server X) as you get to use a conversational label that won't require you to copy paste a long URI. Just saying that is how I would do it.

sfc-gh-eraigosa commented 8 years ago

@ddefolo yep, i agree. In our first pass at the sdk we didn't have support for labels really good.

anandgagrani commented 8 years ago

Thanks guys for quick response. I need more light on answers:

  1. How do you assign a profile name to a blade? i checked options that in oneview but could not find one. let me know where to look for. Or do you mean i should rename the ILO name?
  2. "export ONEVIEW_ILO_USER=user " --is this for the server that i would like to provision?if yes than where do you specify its IP? As i can have multiple blades and i would like to deploy x blade on an x enclosure. Add to the same question does it picks up first server if we do not specify / select blade specifically?
  3. Can i provision a DL server (any server outside c7000 enclosure)?

I am still working on this and your answers shall help me create a practical working paper for people to use with their HP servers and software.

sfc-gh-eraigosa commented 8 years ago

Sure, here you go

  1. You can create a profile from OneView menu: "HP OneView-> Server Profiles, choose Action button , choose Create". You can also do this from "HP OneView-> Server Profile Templates". If you use templates, select the template, and choose "Action -> Create server profile" . Using the profile name that is the same as the docker-machine node name, will cause docker-machine plugin to use that server profile with the assigned blade.
  2. ONEVIEW_ILO_USER, is defined as what is configured as the admin user for ilo administration of the selected blade (we expect that to be consistent from blade to blade). We determine the ip address to use from the OneView server profile data. In the future this will likely not be needed once api's for single sign on to ilo are working better on add server feature for ICsp. You can see more on that in Issue #64 . If you don't specify a blade, we simply pick the first available blade that the template points to, that currently has no profile assigned to it.
  3. We've only tried this on Gen 8 and Gen 9 servers from C7000 and Synergy enclosures. Do let us know about your success or not on DL servers.
anandgagrani commented 8 years ago

Thank Wenlock, I downloaded the plugin docker-machine-driver-oneview.exe and placed it into the docker toolbox directory at C:\Program Files\Docker Toolbox. --hope this is correct.

 I then made the changes as well as applied profile and also created a user on this blade. I then run the command but get an error msg:

Running pre-create checks... Creating machine... (node1) Generating SSH keys... Error creating machine: Error in driver during machine creation: Error in response: User not authorized for this operation. Response Status: 401 Unauthorized

sfc-gh-eraigosa commented 8 years ago

Couple troubleshooting steps.

  1. Verify that the oneview plugin is being found by running docker-machine create -d oneview. This should produce help text with onview specific options and help. If you see it, thats good.
  2. Verify the accounts you are using for ilo, OneView, and ICsp have sufficient permissions to adminster the changes. You can also try adding --debug option to get more output for troubleshooting. IE: docker-machine --debug create -d oneview ...
anandgagrani commented 8 years ago

The plugin works: Administrator@win216 MINGW64 ~ $ docker-machine --debug create -d oneview Docker Machine Version: 0.7.0, build a650a40 Found binary path at C:\Users\Administrator\bin\docker-machine-driver-oneview.exe Launching plugin server for driver oneview Plugin server listening at address 127.0.0.1:52724 () Calling .GetVersion Using API Version 1 () Calling .SetConfigRaw () Calling .GetMachineName (flag-lookup) Calling .GetMachineName (flag-lookup) Calling .DriverName (flag-lookup) Calling .GetCreateFlags (flag-lookup) DBG | DriverName...%s oneview Usage: docker-machine create [OPTIONS] [arg...]

Create a machine

Description: Run 'C:\Program Files\Docker Toolbox\docker-machine.exe create --driver name' to include the create flags for that driver in the help text. . . .

I shall check permissions now for all the user accounts.

anandgagrani commented 8 years ago

Hello Wenlock, I created a new admin user to target machine ILO, oneview and ICSP and used the same in the source file and ran the command again but still getting the same error: $ docker-machine create --oneview-os-plan "${ONEVIEW_OS_PLAN}" --oneview-server-template "${ONEVIEW_SERVER_TEMPLATE}" --oneview-public-connection-nam e "Name_of_public_network" --driver oneview --engine-env HTTP_PROXY="$http_proxy" --engine-env HTTPS_PROXY="$https_proxy" --engine-env NOPROXY="$no proxy" --engine-env http_proxy="$https_proxy" --engine-env https_proxy="$https_proxy" --engine-env no_proxy="$no_proxy" node1 Running pre-create checks... Creating machine... (node1) Generating SSH keys... Error creating machine: Error in driver during machine creation: Error in response: User not authorized for this operation. Response Status: 401 Unauthorized

However the node1 does gets created when i see the docker-machine ls command.

Am i missing any other user? I am running all the docker-machine commands from the docker toolbox interface on windows.

EDIT: earlier i also added the docker_os_build_plan.sh file in the oneview template. should that be modified in anyway?

sfc-gh-eraigosa commented 8 years ago

Hey, doesn't sound like your getting past the first part of simply connecting to OneView... your still only in the docker-machine mechanics so docker_os_build_plan.sh is not in play yet. For the credentials, have you sourced those into the environment settings? IE: I was suggesting you use a small environment script like the following (granted if your using ux shell):

cat > credentials_andsettings.env << EOF
export ONEVIEW_APIVERSION=200

# you need this so we can create and join a new server to ICSP
# if the server already exist in ICSP we don't provision it
export ONEVIEW_ILO_USER=user
export ONEVIEW_ILO_PASSWORD=password

export ONEVIEW_ICSP_ENDPOINT=https://icsp.yournet.com
export ONEVIEW_ICSP_USER=user
export ONEVIEW_ICSP_PASSWORD=password
export ONEVIEW_ICSP_DOMAIN=domain

export ONEVIEW_OV_ENDPOINT=https://oneview.yournet.com
export ONEVIEW_OV_USER=user
export ONEVIEW_OV_PASSWORD=password
export ONEVIEW_OV_DOMAIN=domain 

export ONEVIEW_SSLVERIFY=true

export ONEVIEW_SERVER_TEMPLATE=oneviewtemplate
export ONEVIEW_OS_PLAN="icsp build plan"

# if you have a proxy for your data center
export proxy_enable=true

export proxy_config='http_proxy=http://proxy.yournet.com:8080/
https_proxy=https://proxy.yournet.com:8080/
no_proxy=/var/run/docker.sock,.proxy.yournet.com,localhost,127.0.0.1'

# proxy settings for --engine-evn options
export http_proxy=http://proxy.yournet.com:8080/
export https_proxy=https://proxy.yournet.com:8080/
export no_proxy=/var/run/docker.sock,.proxy.yournet.com,localhost,127.0.0.1

EOF

Update the credentials_andsettings.env file with your environment users, passwords, and urls and make sure you source this script into the shells environment before executing the docker-machine create command. You can source it with the following command:

source credentials_andsettings.env

It also occurs to me, If your using toolbox, your probably using powershell to get to docker-machine executable. I don't currently have the equivalent of these settings in a cmd or ps1 script, but what I do is use mintty which comes with github desktop. You can then launch mintty by choosing a project, and using open in git shell. If your environment is setup correctly docker-machine will be in your path and you can use shell scripts.

However, if you want to avoid adding another tool to get a shell prompt, maybe your other option is to specify all the additional command line arguments that are required, like

(in this case my username is docker and password is pass)

 --oneview-icsp-domain "LOCAL"
   --oneview-icsp-endpoint "https://icspserver.com"
   --oneview-icsp-password "pass"
   --oneview-icsp-user "docker"

   --oneview-ilo-password "pass"
   --oneview-ilo-user "docker"

   --oneview-ov-domain "LOCAL"
   --oneview-ov-endpoint "https://oneview.com"
   --oneview-ov-password "pass"
   --oneview-ov-user "docker"

   --oneview-server-template "DOCKER_1.8_OVTEMP"

Do you think your missing the arguments for the creds and urls maybe?

anandgagrani commented 8 years ago

I used all the credentials in the credentials_andsettings.env and also sourced it. When i install docker toolbox i get the MINGW64 console that i use to run the docker command and i am not using docker shell.

Got a bit more luck after i changed the domain name to local in the env file.

$ docker-machine create --oneview-os-plan "${ONEVIEW_OS_PLAN}" --oneview-server-template "${ONEVIEW_SERVER_TEMPLATE}" --oneview-public-connection-nam e "" --driver oneview --engine-env HTTP_PROXY="$http_proxy" --engine-env HTTPS_PROXY="$https_proxy" --engine-env NO_PROXY="$no_proxy" --engine-env ht tp_proxy="$https_proxy" --engine-env https_proxy="$https_proxy" --engine-env no_proxy="$no_proxy" node1 Running pre-create checks... Creating machine... (node1) Generating SSH keys... (node1) Working on power state. (node1) Desired Power State already set -> Off (node1) Power Task Execution Completed Error creating machine: Error in driver during machine creation: Error in response: User not authorized for this operation. Response Status: 401 Unauthorized

anandgagrani commented 8 years ago

Just powered on the blade from oneview and this is what i observerd: $ docker-machine create --oneview-os-plan "${ONEVIEW_OS_PLAN}" --oneview-server-template "${ONEVIEW_SERVER_TEMPLATE}" --oneview-public-connection-nam e "" --driver oneview --engine-env HTTP_PROXY="$http_proxy" --engine-env HTTPS_PROXY="$https_proxy" --engine-env NO_PROXY="$no_proxy" --engine-env ht tp_proxy="$https_proxy" --engine-env https_proxy="$https_proxy" --engine-env no_proxy="$no_proxy" node1 Running pre-create checks... Creating machine... (node1) Generating SSH keys... (node1) Working on power state. (node1) Powering Off server 2SN54407QB, bay 1 for 2M25450CMY. (node1) Working on power state,2%, Power off server: 2SN54407QB, bay 1. (node1) Working on power state,100%, Successfully powered off server: 2SN54407QB, bay 1.. (node1) Power Task Execution Completed Error creating machine: Error in driver during machine creation: Error in response: User not authorized for this operation. Response Status: 401 Unauthorized

Looks like it is able to connect to oneview but maybe stopping at ICSP.

EDIT:looks like there is some issue between the oneview / ICSP communication. I have created a user with full privileges and i added oneview inside ICSP using that user and it shows as authorized in ICSP. but still same error.

sfc-gh-eraigosa commented 8 years ago

I think your at the point where we are interacting with docker-machine and ICsp for sure, but I don't think your not to far in because you don't have this message:

ICSP creating server for ...

or something like it.

If you can add --debug option we might get some better clues on where you are getting stuck.

IE: something like

$ docker-machine --debug create --oneview-os-plan "${ONEVIEW_OS_PLAN}" ...

One of the first things we try to do with ICsp is to determine if the server is not added to ICsp. We use the blades profile serial number to look this up. Thats a connection to ICsp and a call to the rest API /rest/os-deployment-servers. If you don't have permissions in ICsp to connect or list all servers, then you might get errors. The debug info would be handy on this one.

anandgagrani commented 8 years ago

This is what i get:

(node1) DBG | REQ --> &{Method:POST URL:https://15.215.17.143/rest/login-sessions Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[X-Api-Version :[200] Auth:[none] Content-Type:[application/json; charset=utf-8]] Body:{Reader:} ContentLength:69 TransferEncoding:[] Close:false Host:15.215.17.143 Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr: RequestURI: TLS: Cancel:} (node1) DBG | (node1) DBG | RESP --> &{Status:400 Bad Request StatusCode:400 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Content-Type:[application/json;ch arset=UTF-8] Via:[1.1 example.com] Cache-Control:[no-cache] Date:[Tue, 05 Jul 2016 22:30:39 GMT] Server:[Apache]] Body:0xc0822b8f80 ContentLength:-1 T ransferEncoding:[chunked] Close:true Trailer:map[] Request:0xc0825dd260 TLS:0xc08224d970} (node1) DBG | (node1) DBG | ERROR --> (node1) DBG | (node1) DBG | RestAPICall GET - https://15.215.17.143/rest/os-deployment-servers (node1) DBG | * url => https://15.215.17.143/rest/os-deployment-servers (node1) DBG | * method => ☺ (node1) DBG | *\ proxy => 0x53da20 (node1) DBG | Headers -> X-API-Version -> 200 (node1) DBG | (node1) DBG | Headers -> auth -> none (node1) DBG | (node1) DBG | Headers -> Content-Type -> application/json; charset=utf-8 (node1) DBG | (node1) DBG | REQ --> &{Method:GET URL:https://15.215.17.143/rest/os-deployment-servers Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[X-Api-V ersion:[200] Auth:[none] Content-Type:[application/json; charset=utf-8]] Body: ContentLength:0 TransferEncoding:[] Close:false Host:15.215.17.143 Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr: RequestURI: TLS: Cancel:} (node1) DBG | (node1) DBG | RESP --> &{Status:401 Unauthorized StatusCode:401 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Via:[1.1 example.com] Cache-Cont rol:[no-cache] Date:[Tue, 05 Jul 2016 22:30:39 GMT] Server:[Apache] Content-Type:[application/json;charset=UTF-8]] Body:0xc082380100 ContentLength:-1 TransferEncoding:[chunked] Close:false Trailer:map[] Request:0xc0825dd500 TLS:0xc0822b3080} (node1) DBG | (node1) DBG | ERROR --> (node1) DBG | Error creating machine: Error in driver during machine creation: Error in response: User not authorized for this operation. Response Status: 401 Unauthorized open : The system cannot find the file specified. notifying bugsnag: [Error creating machine: Error in driver during machine creation: Error in response: User not authorized for this operation. Response Status: 401 Unauthorized]

sfc-gh-eraigosa commented 8 years ago

OK, great, it's where I suspected... The login to icsp is failing because we can see this

(node1) DBG | Headers -> auth -> none

after we call /rest/login-sessions

Double check for typos on the ICSP user, password and domain. Also try quoting the values:

export ONEVIEW_ICSP_ENDPOINT=https://icsp.yournet.com
export ONEVIEW_ICSP_USER="user"
export ONEVIEW_ICSP_PASSWORD="password"
export ONEVIEW_ICSP_DOMAIN="domain"

I'm out of time today, but I'll see if i can get you a test curl command to see if you can test login. You can also verify that you can reach your server with this curl command:

curl -Lk --get "$ONEVIEW_ICSP_ENDPOINT/rest/version"

Output should be something like this:

{"currentVersion":200,"minimumVersion":1}

If your not getting that, then it's possible something wrong with the proxy settings.

anandgagrani commented 8 years ago

Thanks wenlock, This is what i get: $ curl -Lk --get "$ONEVIEW_ICSP_ENDPOINT/rest/version" {"currentVersion":200,"minimumVersion":1}

I also quoted all the username and password but same error

One quick question: ICSP has 2 networks -- appliance(15 subnet) and deployment(172 subnet) and target blades and media server are in 172 subnet. oneview also has both the subnets. So in the create / .env file command which subnet of ICSP i should use (ONEVIEW_ICSP_ENDPOINT )- 15.x.x.x or 172.x.x.x. ?

If i use the 172 subnet (in the .env file ) the curl command fails and also the create command.

$ curl -Lk --get "$ONEVIEW_ICSP_ENDPOINT/rest/version" curl: (56) Received HTTP code 503 from proxy after CONNECT

EDIT: I just made the ICSP ip back to 15.x and disabled proxy as you suspected. and it turns out the proxy to be issue. Now the build plan proceed but i get error at 33%:

ID: HPSA-1901 Code: com.opsware.ilo.ILO_CONNECTION_FAILURE Details: Failed to connect with given iLO server. Cause: No route to host Action: Please check your iLO connection credentials. Otherwise, contact your SA administrator.

I am able to ping and also checked the ILO credentials that are ok...but no luck. Just to add, i also get the error when i remove the node:

$ docker-machine rm -f node1 About to remove node1 (node1) Stop ... node1 (node1) Problem shutting down gracefully : Error public_interface custom attribute is not found. (node1) Working on power state. (node1) Desired Power State already set -> Off (node1) Power Task Execution Completed Error removing host "node1": Error in response: Missing mandatory header from the request. Response Status: 400 Bad Request Successfully removed node1

Thanks again for all your time and efforts.

ddefolo commented 8 years ago

3 things:

1) Have you confirmed your clocks are in sync? If the OneView and ICsp appliance times are significantly off from the client where you are calling docker machine then you will get failures. NTP or syncing times with the same vm host are recommended.

2) Are you using the right domain? Keep in mind that the domain is the name of the domain that shows up in the drop-down list of the UI when logging into OneView or ICsp and not what the actual name domain group is in LDAP or your AD server. That confused me at first.

3) Regarding the ICsp 2 networks - the ICsp management IP address (in your case the 15 subnet) address is the only address that you should use for REST interaction. The other subnet is used exclusively when deploying servers to keep the heavier traffic (pushing OS bits) on a different network than your general datacenter management network.

anandgagrani commented 8 years ago

Thanks ddefolo, The ICSP and oneview are diff vms and they are in sync with the VM host. The ILO in managed by 172 DHCP server. There is a time diff in ICSP and ILO.

  1. I have no domains defined. hence i select local in .enc file.
  2. I agree. It goes till ILO but then fails.

I shall sync the time with ntp server and check how it goes. Thanks

ddefolo commented 8 years ago

on 1 I'm not saying OneView and ICsp need their time synced (although that wouldn't be bad to do). I'm suggesting your client and the appliances need their time synced. That means the time of the client where you are calling docker-machine needs to be close to being in sync with the appliance. Make sure to note the time zones of each and factor that in to know if they are truly in sync.

You can simply manually adjust the time of the client system to be in sync with the appliances to see if that resolves the issue. If it does, then you know a more permanent solution like NTP is needed.

anandgagrani commented 8 years ago

Thanks ddfolo. I shall do that and see how it goes. There was a time diff.

Edit: After time sync..still the same issue: Creating machine... (node1) Generating SSH keys... (node1) Working on power state. (node1) Desired Power State already set -> Off (node1) Power Task Execution Completed (node1) Working on power state. (node1) Desired Power State already set -> Off (node1) Power Task Execution Completed (node1) ICSP creating server for : 172.20.239.18 (node1) Initializing creation of server for ICSP, 172.20.239.18. (node1) Waiting on, Add server (iLO 172.20.239.18), 0% (node1) Waiting on, Add server (iLO 172.20.239.18), 33%, Register iLO (node1) Waiting on, Add server (iLO 172.20.239.18), 33%, Register iLO (node1) Waiting on, Add server (iLO 172.20.239.18), 33%, Register iLO (node1) Waiting on, Add server (iLO 172.20.239.18), 33%, Register iLO (node1) Waiting on, Add server (iLO 172.20.239.18), 33%, Register iLO (node1) Waiting on, Add server (iLO 172.20.239.18), 33%, Register iLO (node1) Waiting on, Add server (iLO 172.20.239.18), 33%, Register iLO (node1) Waiting on, Add server (iLO 172.20.239.18), 33%, Register iLO (node1) Waiting on, Add server (iLO 172.20.239.18), 33%, Register iLO (node1) Waiting on, Add server (iLO 172.20.239.18), 33%, Register iLO Error creating machine: Error in driver during machine creation: Register iLO ERROR: ID: HPSA-1901 Code: com.opsware.ilo.ILO_CONNECTION_FAILURE Details: Failed to connect with given iLO server. Cause: No route to host Action: Please check your iLO connection credentials. Otherwise, contact your SA administrator.

anandgagrani commented 8 years ago

Tried with all possible proxys that i can think of but no luck

anandgagrani commented 8 years ago

looks like there was an error in the network interface 172.x of ICSP VM. after correcting it ..it moved further into installation. Now it fails at : (node1) Waiting on, Add server (iLO 172.20.239.18), 67%, Boot server to maintenance (node1) Waiting on, Add server (iLO 172.20.239.18), 67%, Boot server to maintenance Error creating machine: Error in driver during machine creation: Boot server to maintenance ERROR: Server failed to properly boot to maintenance and register with the appliance. Check the Boot to Service OS build plan job log for more details.

ddefolo commented 8 years ago

My recommendation is to make sure you can register your servers in ICsp without failures using the ICsp UI (and triage from a pure ICsp perspective using ICsp troubleshooting docs) before getting docker machine into the picture. The most common things that can go wrong there include:

  1. Your OneView profile doesn't make a connection to the same network that ICsp is using (e.g. compare vlan IDs for your appliance network connections to the profile).
  2. You have more than 1 network connection on that network in the profile as part of plans to have teamed nics (as per ICsp docs you should only have 1 connection until after ICsp is done registering the server)

Beyond that, as per the error above, going to the "Jobs" page in ICsp should provide a more specific error message that will help pinpoint the problem.

anandgagrani commented 8 years ago

Thanks DDefolo, The error in the jobs UI(when i add the server via UI without docker) is same. Also i have only one network in the profile.

Step 4 of 4: Run OGFS Script 'Wait for HP SA Agent' Sleeping for 3 minutes before checking for agent Waiting for registration of agent running on maintenance OS Failed To wait for the HP SA Agent: The agent did not appear on the network (is not listening on its port). Additional information from boot control: hpstartae copyright (c) 2011 hewlett packard - all rights reserved /tmp/ae / AE-test.xml AutomationEngine libcrypto.so tar: libcrypto.so: Cannot utime: No such file or directory libcrypto.so.1.0.0 libcurl.so tar: libcurl.so: Cannot utime: No such file or directory libcurl.so.4 tar: libcurl.so.4: Cannot utime: No such file or directory libcurl.so.4.2.0 libssh2.so tar: libssh2.so: Cannot utime: No such file or directory libssh2.so.1 tar: libssh2.so.1: Cannot utime: No such file or directory libssh2.so.1.0.1 libssl.so tar: libssl.so: Cannot utime: No such file or directory libssl.so.1.0.0 libz.so tar: libz.so: Cannot utime: No such file or directory libz.so.1 tar: libz.so.1: Cannot utime: No such file or directory libz.so.1.2.5 README.TXT tar: Exiting with failure status due to previous errors ping: sendto: Network is unreachable Network Interface is not up, test IP: 172.20.239.50, ping count: 1 ping: sendto: Network is unreachable Network Interface is not up, test IP: 172.20.239.50, ping count: 2 ping: sendto: Network is unreachable Network Interface is not up, test IP: 172.20.239.50, ping count: 3 ping: sendto: Network is unreachable Network Interface is not up, test IP: 172.20.239.50, ping count: 4 ping: sendto: Network is unreachable Network Interface is not up, test IP: 172.20.239.50, ping count: 5 ping: sendto: Network is unreachable Network Interface is not up, test IP: 172.20.239.50, ping count: 6 ping: sendto: Network is unreachable Network Interface is not up, test IP: 172.20.239.50, ping count: 7 ping: sendto: Network is unreachable Network Interface is not up, test IP: 172.20.239.50, ping count: 8 ping: sendto: Network is unreachable Network Interface is not up, test IP: 172.20.239.50, ping count: 9 ping: sendto: Network is unreachable Network Interface is not up, test IP: 172.20.239.50, ping count: 10 Failed to download URL: http://127.0.0.1:8081/osprov_ogfs_agent-LINUX-SLES-11-X86_64 Response code received: 0 Possible reason: Couldn't connect to server Error buffer: Failed to connect to 172.20.239.50: Network is unreachable chmod: *: No such file or directory HPSA OGFS Agent Setup /tmp/post.bat: line 8: ./osprov/ogfs-agent-setup.sh: No such file or directory HPSA OGFS Agent Start /tmp/post.bat: line 10: ./osprov/ogfs-agent-start.sh: No such file or directory Script exec failed: 127, reason:Key has expired URI download failed, exiting Automation Engine - version: 1.0.0

/opt/hp/hp-phoenix/scripts/ae.xml: loaded successfully

Wait for HP SA Agent failed with exit code 120.

EDIT: one observation - when i check this server in ICSP , it is registered successfully but shows Deployment interface 127.1.2.1. Shouldn't it be in 172 range that ICSP deployment interface and dhcp has?

2) : from the docker interface where the create command is run - it fails with this msg after 2 hrs after throuwing above msg in ICSP UI: Error creating machine: Error in driver during machine creation: Boot server to maintenance ERROR: Server failed to properly boot to maintenance and register with the appliance. Check the Boot to Service OS build plan job log for more details.

ddefolo commented 8 years ago

anandgagrani,

You are dealing with a pure issue with your ICsp setup, this has nothing to do with docker so as I a said before I would first assure you can successfully use ICsp without docker in the picture. Interacting with the OneView UI and ICsp UI until you get to the bottom of your issues is likely the best approach. The fact that the SA agent can't phone home to ICsp and that your deployment interface has a weird IP is an indicator to me that you don't have the deployment network nic of ICsp connected to the same network as the server profile in OneView or you don't have a properly configured DHCP server on that network (see item 4 below).

Resources:

  1. Get the appropriate ICsp Administrator guide or look at the online help for your version of ICsp from: https://hpe.com/info/insightcontrol/docs
  2. Look at the Wait for SA agent troubleshooting section. Even though that talks about exit code 6 and you got 120 I have found that many of the same troubleshooting instructions
  3. Look at the Mid and crypto not found troubleshooting instructions since I see that in your errors above. In short, you might need to delete the server from ICsp first before trying to re-add it.
  4. Make double-sure the ICsp deployment network vlan (your 172 subnet) is the same vlan as the network you are connecting to in the OneView profile and that you have a properly configured DHCP server on that network. An easy way to tell that is to enable that connection for PXE boot in the profile and do a network boot (e.g. 1 time network boot or adjust boot order) and confirm you get a PXE boot and the install menu from ICsp. If not there is a troubleshooting section on that in the docs I've been talking about.
  5. If all else fails you need to get back to the basics of troubleshooting (sorry) using Troubleshooting basics