Open anandgagrani opened 8 years ago
Hey, lots of questions, thats fantastic! OK, let me try to answer, let me know if i missed a question.
docker-machine-driver-oneview
in the system path, or same folder as the docker-machine executable. Then include the option --driver oneview
. Running the command docker-machine --driver oneview create
should produce output that comes from the oneview executable and confirms the plugin is talking to docker-machine.Here is an example command that works for me well:
cat > credentials_andsettings.env << EOF
export ONEVIEW_APIVERSION=200
# you need this so we can create and join a new server to ICSP
# if the server already exist in ICSP we don't provision it
export ONEVIEW_ILO_USER=user
export ONEVIEW_ILO_PASSWORD=password
export ONEVIEW_ICSP_ENDPOINT=https://icsp.yournet.com
export ONEVIEW_ICSP_USER=user
export ONEVIEW_ICSP_PASSWORD=password
export ONEVIEW_ICSP_DOMAIN=domain
export ONEVIEW_OV_ENDPOINT=https://oneview.yournet.com
export ONEVIEW_OV_USER=user
export ONEVIEW_OV_PASSWORD=password
export ONEVIEW_OV_DOMAIN=domain
export ONEVIEW_SSLVERIFY=true
export ONEVIEW_SERVER_TEMPLATE=oneviewtemplate
export ONEVIEW_OS_PLAN="icsp build plan"
# if you have a proxy for your data center
export proxy_enable=true
export proxy_config='http_proxy=http://proxy.yournet.com:8080/
https_proxy=https://proxy.yournet.com:8080/
no_proxy=/var/run/docker.sock,.proxy.yournet.com,localhost,127.0.0.1'
# proxy settings for --engine-evn options
export http_proxy=http://proxy.yournet.com:8080/
export https_proxy=https://proxy.yournet.com:8080/
export no_proxy=/var/run/docker.sock,.proxy.yournet.com,localhost,127.0.0.1
EOF
source credentials_adnsettings.env
# add the --engine-env options if you have a proxy
docker-machine create \
--oneview-os-plan "${ONEVIEW_OS_PLAN}" \
--oneview-server-template "${ONEVIEW_SERVER_TEMPLATE}" \
--oneview-public-connection-name "Name_of_public_network" \
--driver oneview \
--engine-env HTTP_PROXY="$http_proxy" \
--engine-env HTTPS_PROXY="$https_proxy" \
--engine-env NO_PROXY="$no_proxy" \
--engine-env http_proxy="$https_proxy" \
--engine-env https_proxy="$https_proxy" \
--engine-env no_proxy="$no_proxy" \
nodename
This was needed to add a server to icsp. If we see that the server is already in icsp, we don't want to add it to docker-machine or provision it, to avoid destroying someone else server that might have already been provisioned. This way docker-machine always tries to add the new server to icsp (it should not exist in icsp prior to a create command).
Do let us know your progress and issues so we can make it easier.
Doesn't this come back to simply needing to support labels? See https://github.com/HewlettPackard/docker-machine-oneview/issues/47
If you supported label filters you can tag whatever server (or servers) you want to deploy with a unique label and then specify that label in your docker-machine call. That seems like a pretty minor change if it isn't supported yet.
Its also easier to use than direct server URI specification (which is the only reasonable way to directly specify I want to deploy server X) as you get to use a conversational label that won't require you to copy paste a long URI. Just saying that is how I would do it.
@ddefolo yep, i agree. In our first pass at the sdk we didn't have support for labels really good.
Thanks guys for quick response. I need more light on answers:
I am still working on this and your answers shall help me create a practical working paper for people to use with their HP servers and software.
Sure, here you go
Thank Wenlock, I downloaded the plugin docker-machine-driver-oneview.exe and placed it into the docker toolbox directory at C:\Program Files\Docker Toolbox. --hope this is correct.
I then made the changes as well as applied profile and also created a user on this blade. I then run the command but get an error msg:
Running pre-create checks... Creating machine... (node1) Generating SSH keys... Error creating machine: Error in driver during machine creation: Error in response: User not authorized for this operation. Response Status: 401 Unauthorized
Couple troubleshooting steps.
docker-machine create -d oneview
. This should produce help text with onview specific options and help. If you see it, thats good.docker-machine --debug create -d oneview ...
The plugin works: Administrator@win216 MINGW64 ~ $ docker-machine --debug create -d oneview Docker Machine Version: 0.7.0, build a650a40 Found binary path at C:\Users\Administrator\bin\docker-machine-driver-oneview.exe Launching plugin server for driver oneview Plugin server listening at address 127.0.0.1:52724 () Calling .GetVersion Using API Version 1 () Calling .SetConfigRaw () Calling .GetMachineName (flag-lookup) Calling .GetMachineName (flag-lookup) Calling .DriverName (flag-lookup) Calling .GetCreateFlags (flag-lookup) DBG | DriverName...%s oneview Usage: docker-machine create [OPTIONS] [arg...]
Create a machine
Description: Run 'C:\Program Files\Docker Toolbox\docker-machine.exe create --driver name' to include the create flags for that driver in the help text. . . .
I shall check permissions now for all the user accounts.
Hello Wenlock, I created a new admin user to target machine ILO, oneview and ICSP and used the same in the source file and ran the command again but still getting the same error: $ docker-machine create --oneview-os-plan "${ONEVIEW_OS_PLAN}" --oneview-server-template "${ONEVIEW_SERVER_TEMPLATE}" --oneview-public-connection-nam e "Name_of_public_network" --driver oneview --engine-env HTTP_PROXY="$http_proxy" --engine-env HTTPS_PROXY="$https_proxy" --engine-env NOPROXY="$no proxy" --engine-env http_proxy="$https_proxy" --engine-env https_proxy="$https_proxy" --engine-env no_proxy="$no_proxy" node1 Running pre-create checks... Creating machine... (node1) Generating SSH keys... Error creating machine: Error in driver during machine creation: Error in response: User not authorized for this operation. Response Status: 401 Unauthorized
However the node1 does gets created when i see the docker-machine ls command.
Am i missing any other user? I am running all the docker-machine commands from the docker toolbox interface on windows.
EDIT: earlier i also added the docker_os_build_plan.sh file in the oneview template. should that be modified in anyway?
Hey, doesn't sound like your getting past the first part of simply connecting to OneView... your still only in the docker-machine mechanics so docker_os_build_plan.sh is not in play yet. For the credentials, have you sourced those into the environment settings? IE: I was suggesting you use a small environment script like the following (granted if your using ux shell):
cat > credentials_andsettings.env << EOF
export ONEVIEW_APIVERSION=200
# you need this so we can create and join a new server to ICSP
# if the server already exist in ICSP we don't provision it
export ONEVIEW_ILO_USER=user
export ONEVIEW_ILO_PASSWORD=password
export ONEVIEW_ICSP_ENDPOINT=https://icsp.yournet.com
export ONEVIEW_ICSP_USER=user
export ONEVIEW_ICSP_PASSWORD=password
export ONEVIEW_ICSP_DOMAIN=domain
export ONEVIEW_OV_ENDPOINT=https://oneview.yournet.com
export ONEVIEW_OV_USER=user
export ONEVIEW_OV_PASSWORD=password
export ONEVIEW_OV_DOMAIN=domain
export ONEVIEW_SSLVERIFY=true
export ONEVIEW_SERVER_TEMPLATE=oneviewtemplate
export ONEVIEW_OS_PLAN="icsp build plan"
# if you have a proxy for your data center
export proxy_enable=true
export proxy_config='http_proxy=http://proxy.yournet.com:8080/
https_proxy=https://proxy.yournet.com:8080/
no_proxy=/var/run/docker.sock,.proxy.yournet.com,localhost,127.0.0.1'
# proxy settings for --engine-evn options
export http_proxy=http://proxy.yournet.com:8080/
export https_proxy=https://proxy.yournet.com:8080/
export no_proxy=/var/run/docker.sock,.proxy.yournet.com,localhost,127.0.0.1
EOF
Update the credentials_andsettings.env
file with your environment users, passwords, and urls and make sure you source this script into the shells environment before executing the docker-machine create
command. You can source it with the following command:
source credentials_andsettings.env
It also occurs to me, If your using toolbox, your probably using powershell to get to docker-machine executable. I don't currently have the equivalent of these settings in a cmd or ps1 script, but what I do is use mintty which comes with github desktop. You can then launch mintty by choosing a project, and using open in git shell. If your environment is setup correctly docker-machine will be in your path and you can use shell scripts.
However, if you want to avoid adding another tool to get a shell prompt, maybe your other option is to specify all the additional command line arguments that are required, like
(in this case my username is docker and password is pass)
--oneview-icsp-domain "LOCAL"
--oneview-icsp-endpoint "https://icspserver.com"
--oneview-icsp-password "pass"
--oneview-icsp-user "docker"
--oneview-ilo-password "pass"
--oneview-ilo-user "docker"
--oneview-ov-domain "LOCAL"
--oneview-ov-endpoint "https://oneview.com"
--oneview-ov-password "pass"
--oneview-ov-user "docker"
--oneview-server-template "DOCKER_1.8_OVTEMP"
Do you think your missing the arguments for the creds and urls maybe?
I used all the credentials in the credentials_andsettings.env and also sourced it. When i install docker toolbox i get the MINGW64 console that i use to run the docker command and i am not using docker shell.
Got a bit more luck after i changed the domain name to local in the env file.
$ docker-machine create --oneview-os-plan "${ONEVIEW_OS_PLAN}" --oneview-server-template "${ONEVIEW_SERVER_TEMPLATE}" --oneview-public-connection-nam e "" --driver oneview --engine-env HTTP_PROXY="$http_proxy" --engine-env HTTPS_PROXY="$https_proxy" --engine-env NO_PROXY="$no_proxy" --engine-env ht tp_proxy="$https_proxy" --engine-env https_proxy="$https_proxy" --engine-env no_proxy="$no_proxy" node1 Running pre-create checks... Creating machine... (node1) Generating SSH keys... (node1) Working on power state. (node1) Desired Power State already set -> Off (node1) Power Task Execution Completed Error creating machine: Error in driver during machine creation: Error in response: User not authorized for this operation. Response Status: 401 Unauthorized
Just powered on the blade from oneview and this is what i observerd: $ docker-machine create --oneview-os-plan "${ONEVIEW_OS_PLAN}" --oneview-server-template "${ONEVIEW_SERVER_TEMPLATE}" --oneview-public-connection-nam e "" --driver oneview --engine-env HTTP_PROXY="$http_proxy" --engine-env HTTPS_PROXY="$https_proxy" --engine-env NO_PROXY="$no_proxy" --engine-env ht tp_proxy="$https_proxy" --engine-env https_proxy="$https_proxy" --engine-env no_proxy="$no_proxy" node1 Running pre-create checks... Creating machine... (node1) Generating SSH keys... (node1) Working on power state. (node1) Powering Off server 2SN54407QB, bay 1 for 2M25450CMY. (node1) Working on power state,2%, Power off server: 2SN54407QB, bay 1. (node1) Working on power state,100%, Successfully powered off server: 2SN54407QB, bay 1.. (node1) Power Task Execution Completed Error creating machine: Error in driver during machine creation: Error in response: User not authorized for this operation. Response Status: 401 Unauthorized
Looks like it is able to connect to oneview but maybe stopping at ICSP.
EDIT:looks like there is some issue between the oneview / ICSP communication. I have created a user with full privileges and i added oneview inside ICSP using that user and it shows as authorized in ICSP. but still same error.
I think your at the point where we are interacting with docker-machine and ICsp for sure, but I don't think your not to far in because you don't have this message:
ICSP creating server for ...
or something like it.
If you can add --debug
option we might get some better clues on where you are getting stuck.
IE: something like
$ docker-machine --debug create --oneview-os-plan "${ONEVIEW_OS_PLAN}" ...
One of the first things we try to do with ICsp is to determine if the server is not added to ICsp. We use the blades profile serial number to look this up. Thats a connection to ICsp and a call to the rest API /rest/os-deployment-servers
. If you don't have permissions in ICsp to connect or list all servers, then you might get errors. The debug info would be handy on this one.
This is what i get:
(node1) DBG | REQ --> &{Method:POST URL:https://15.215.17.143/rest/login-sessions Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[X-Api-Version
:[200] Auth:[none] Content-Type:[application/json; charset=utf-8]] Body:{Reader:} ContentLength:69 TransferEncoding:[] Close:false Host:15.215.17.143
Form:map[] PostForm:map[] MultipartForm:
OK, great, it's where I suspected... The login to icsp is failing because we can see this
(node1) DBG | Headers -> auth -> none
after we call /rest/login-sessions
Double check for typos on the ICSP user, password and domain. Also try quoting the values:
export ONEVIEW_ICSP_ENDPOINT=https://icsp.yournet.com
export ONEVIEW_ICSP_USER="user"
export ONEVIEW_ICSP_PASSWORD="password"
export ONEVIEW_ICSP_DOMAIN="domain"
I'm out of time today, but I'll see if i can get you a test curl command to see if you can test login. You can also verify that you can reach your server with this curl command:
curl -Lk --get "$ONEVIEW_ICSP_ENDPOINT/rest/version"
Output should be something like this:
{"currentVersion":200,"minimumVersion":1}
If your not getting that, then it's possible something wrong with the proxy settings.
Thanks wenlock, This is what i get: $ curl -Lk --get "$ONEVIEW_ICSP_ENDPOINT/rest/version" {"currentVersion":200,"minimumVersion":1}
I also quoted all the username and password but same error
One quick question: ICSP has 2 networks -- appliance(15 subnet) and deployment(172 subnet) and target blades and media server are in 172 subnet. oneview also has both the subnets. So in the create / .env file command which subnet of ICSP i should use (ONEVIEW_ICSP_ENDPOINT )- 15.x.x.x or 172.x.x.x. ?
If i use the 172 subnet (in the .env file ) the curl command fails and also the create command.
$ curl -Lk --get "$ONEVIEW_ICSP_ENDPOINT/rest/version" curl: (56) Received HTTP code 503 from proxy after CONNECT
EDIT: I just made the ICSP ip back to 15.x and disabled proxy as you suspected. and it turns out the proxy to be issue. Now the build plan proceed but i get error at 33%:
ID: HPSA-1901 Code: com.opsware.ilo.ILO_CONNECTION_FAILURE Details: Failed to connect with given iLO server. Cause: No route to host Action: Please check your iLO connection credentials. Otherwise, contact your SA administrator.
I am able to ping and also checked the ILO credentials that are ok...but no luck. Just to add, i also get the error when i remove the node:
$ docker-machine rm -f node1 About to remove node1 (node1) Stop ... node1 (node1) Problem shutting down gracefully : Error public_interface custom attribute is not found. (node1) Working on power state. (node1) Desired Power State already set -> Off (node1) Power Task Execution Completed Error removing host "node1": Error in response: Missing mandatory header from the request. Response Status: 400 Bad Request Successfully removed node1
Thanks again for all your time and efforts.
3 things:
1) Have you confirmed your clocks are in sync? If the OneView and ICsp appliance times are significantly off from the client where you are calling docker machine then you will get failures. NTP or syncing times with the same vm host are recommended.
2) Are you using the right domain? Keep in mind that the domain is the name of the domain that shows up in the drop-down list of the UI when logging into OneView or ICsp and not what the actual name domain group is in LDAP or your AD server. That confused me at first.
3) Regarding the ICsp 2 networks - the ICsp management IP address (in your case the 15 subnet) address is the only address that you should use for REST interaction. The other subnet is used exclusively when deploying servers to keep the heavier traffic (pushing OS bits) on a different network than your general datacenter management network.
Thanks ddefolo, The ICSP and oneview are diff vms and they are in sync with the VM host. The ILO in managed by 172 DHCP server. There is a time diff in ICSP and ILO.
I shall sync the time with ntp server and check how it goes. Thanks
on 1 I'm not saying OneView and ICsp need their time synced (although that wouldn't be bad to do). I'm suggesting your client and the appliances need their time synced. That means the time of the client where you are calling docker-machine needs to be close to being in sync with the appliance. Make sure to note the time zones of each and factor that in to know if they are truly in sync.
You can simply manually adjust the time of the client system to be in sync with the appliances to see if that resolves the issue. If it does, then you know a more permanent solution like NTP is needed.
Thanks ddfolo. I shall do that and see how it goes. There was a time diff.
Edit: After time sync..still the same issue: Creating machine... (node1) Generating SSH keys... (node1) Working on power state. (node1) Desired Power State already set -> Off (node1) Power Task Execution Completed (node1) Working on power state. (node1) Desired Power State already set -> Off (node1) Power Task Execution Completed (node1) ICSP creating server for : 172.20.239.18 (node1) Initializing creation of server for ICSP, 172.20.239.18. (node1) Waiting on, Add server (iLO 172.20.239.18), 0% (node1) Waiting on, Add server (iLO 172.20.239.18), 33%, Register iLO (node1) Waiting on, Add server (iLO 172.20.239.18), 33%, Register iLO (node1) Waiting on, Add server (iLO 172.20.239.18), 33%, Register iLO (node1) Waiting on, Add server (iLO 172.20.239.18), 33%, Register iLO (node1) Waiting on, Add server (iLO 172.20.239.18), 33%, Register iLO (node1) Waiting on, Add server (iLO 172.20.239.18), 33%, Register iLO (node1) Waiting on, Add server (iLO 172.20.239.18), 33%, Register iLO (node1) Waiting on, Add server (iLO 172.20.239.18), 33%, Register iLO (node1) Waiting on, Add server (iLO 172.20.239.18), 33%, Register iLO (node1) Waiting on, Add server (iLO 172.20.239.18), 33%, Register iLO Error creating machine: Error in driver during machine creation: Register iLO ERROR: ID: HPSA-1901 Code: com.opsware.ilo.ILO_CONNECTION_FAILURE Details: Failed to connect with given iLO server. Cause: No route to host Action: Please check your iLO connection credentials. Otherwise, contact your SA administrator.
Tried with all possible proxys that i can think of but no luck
looks like there was an error in the network interface 172.x of ICSP VM. after correcting it ..it moved further into installation. Now it fails at : (node1) Waiting on, Add server (iLO 172.20.239.18), 67%, Boot server to maintenance (node1) Waiting on, Add server (iLO 172.20.239.18), 67%, Boot server to maintenance Error creating machine: Error in driver during machine creation: Boot server to maintenance ERROR: Server failed to properly boot to maintenance and register with the appliance. Check the Boot to Service OS build plan job log for more details.
My recommendation is to make sure you can register your servers in ICsp without failures using the ICsp UI (and triage from a pure ICsp perspective using ICsp troubleshooting docs) before getting docker machine into the picture. The most common things that can go wrong there include:
Beyond that, as per the error above, going to the "Jobs" page in ICsp should provide a more specific error message that will help pinpoint the problem.
Thanks DDefolo, The error in the jobs UI(when i add the server via UI without docker) is same. Also i have only one network in the profile.
Step 4 of 4: Run OGFS Script 'Wait for HP SA Agent' Sleeping for 3 minutes before checking for agent Waiting for registration of agent running on maintenance OS Failed To wait for the HP SA Agent: The agent did not appear on the network (is not listening on its port). Additional information from boot control: hpstartae copyright (c) 2011 hewlett packard - all rights reserved /tmp/ae / AE-test.xml AutomationEngine libcrypto.so tar: libcrypto.so: Cannot utime: No such file or directory libcrypto.so.1.0.0 libcurl.so tar: libcurl.so: Cannot utime: No such file or directory libcurl.so.4 tar: libcurl.so.4: Cannot utime: No such file or directory libcurl.so.4.2.0 libssh2.so tar: libssh2.so: Cannot utime: No such file or directory libssh2.so.1 tar: libssh2.so.1: Cannot utime: No such file or directory libssh2.so.1.0.1 libssl.so tar: libssl.so: Cannot utime: No such file or directory libssl.so.1.0.0 libz.so tar: libz.so: Cannot utime: No such file or directory libz.so.1 tar: libz.so.1: Cannot utime: No such file or directory libz.so.1.2.5 README.TXT tar: Exiting with failure status due to previous errors ping: sendto: Network is unreachable Network Interface is not up, test IP: 172.20.239.50, ping count: 1 ping: sendto: Network is unreachable Network Interface is not up, test IP: 172.20.239.50, ping count: 2 ping: sendto: Network is unreachable Network Interface is not up, test IP: 172.20.239.50, ping count: 3 ping: sendto: Network is unreachable Network Interface is not up, test IP: 172.20.239.50, ping count: 4 ping: sendto: Network is unreachable Network Interface is not up, test IP: 172.20.239.50, ping count: 5 ping: sendto: Network is unreachable Network Interface is not up, test IP: 172.20.239.50, ping count: 6 ping: sendto: Network is unreachable Network Interface is not up, test IP: 172.20.239.50, ping count: 7 ping: sendto: Network is unreachable Network Interface is not up, test IP: 172.20.239.50, ping count: 8 ping: sendto: Network is unreachable Network Interface is not up, test IP: 172.20.239.50, ping count: 9 ping: sendto: Network is unreachable Network Interface is not up, test IP: 172.20.239.50, ping count: 10 Failed to download URL: http://127.0.0.1:8081/osprov_ogfs_agent-LINUX-SLES-11-X86_64 Response code received: 0 Possible reason: Couldn't connect to server Error buffer: Failed to connect to 172.20.239.50: Network is unreachable chmod: *: No such file or directory HPSA OGFS Agent Setup /tmp/post.bat: line 8: ./osprov/ogfs-agent-setup.sh: No such file or directory HPSA OGFS Agent Start /tmp/post.bat: line 10: ./osprov/ogfs-agent-start.sh: No such file or directory Script exec failed: 127, reason:Key has expired URI download failed, exiting Automation Engine - version: 1.0.0
/opt/hp/hp-phoenix/scripts/ae.xml: loaded successfully
Wait for HP SA Agent failed with exit code 120.
EDIT: one observation - when i check this server in ICSP , it is registered successfully but shows Deployment interface 127.1.2.1. Shouldn't it be in 172 range that ICSP deployment interface and dhcp has?
2) : from the docker interface where the create command is run - it fails with this msg after 2 hrs after throuwing above msg in ICSP UI: Error creating machine: Error in driver during machine creation: Boot server to maintenance ERROR: Server failed to properly boot to maintenance and register with the appliance. Check the Boot to Service OS build plan job log for more details.
anandgagrani,
You are dealing with a pure issue with your ICsp setup, this has nothing to do with docker so as I a said before I would first assure you can successfully use ICsp without docker in the picture. Interacting with the OneView UI and ICsp UI until you get to the bottom of your issues is likely the best approach. The fact that the SA agent can't phone home to ICsp and that your deployment interface has a weird IP is an indicator to me that you don't have the deployment network nic of ICsp connected to the same network as the server profile in OneView or you don't have a properly configured DHCP server on that network (see item 4 below).
Resources:
In the docker-machine create command (for oneview with docker) how do you specify which blade to use in the oneview infrastructure, for the docker host provision? I would like to provision host on specific blade but in the example command set below..there is no option to specify target blade. Also how to use the docker with oneview plugin...there is no clear document defining all the steps. if you can answer this question ..I am almost done with creating one document for everyone.
docker-machine create \
--engine-env HTTP_PROXY="" \ --engine-env HTTPS_PROXY="" \ --engine-env NO_PROXY="" \ --driver oneview \ --oneview-os-plan "RHEL71_DOCKER_18"\ --oneview-server-template "SP_template" \ --oneview-ov-user=Administrator \ --oneview-ov-password=\ --oneview-ov-domain=\ --oneview-ov-endpoint=https://15.xx.xx.xx/ \ --oneview-icsp-user=administrator \ --oneview-icsp-password=\ --oneview-icsp-endpoint=15.215.xx.xx\ --oneview-ssh-user=administrator \ --oneview-ilo-user=administrator \ --oneview-ilo-password=\ docker-ip-node-01
Also what does this mean: Name: "oneview-ilo-user",
Usage: "ILO User id that is used during ICSP server creation.",
I do not remember using any ilo when creating ICSP vm and then its template.
I am following the steps mentioned in document below(that is rather vague in actual steps) https://h20195.www2.hp.com/V2/getpdf.aspx/4AA6-2595ENW.pdf