Setup ONAP-Amsterdam CNF vagrant demo environment on Packet.net.

taylor commented 6 years ago

Goal: Setup ONAP environment, using provided scripts/code, which will be used for initial VNF testing

We are starting with the Vagrant code in https://github.com/cncf/onap-demo/tree/amsterdam/vagrant to setup OpenStack (devstack) and ONAP. (This repository is a mirror of https://github.com/onap/demo which is a mirror of https://gerrit.onap.org/r/#/q/project:demo)

This ticket will document items needed to fix the setup to get the ONAP demo environment up and running on OpenStack

We started with the Amsterdam release of onap (branch amsterdam in this repo)

Packet.net server info:

Bare metal server
m1.xlarge.x86
- Memory: 256 GB
- CPU: Intel Xeon E5-2650 (2x) x86 24 Cores @ 2.2 GHz
- Disk: 2.8 TB
OS: Ubuntu 17.10
Network:
- Public 147.75.100.181, 147.75.100.180
- Private 10.80.130.3, 10.80.130.2

Tasks

[x] Clone repo to system
[x] Setup pre-reqs to run vagrant
[x] Run vagrant up
[x] Fix vagrant/virtualbox issues
[x] Fix OpenStack issues
[ ] fix issues with ONAP... see https://github.com/cncf/onap-demo/issues/5

Issues:

SOLVED: Missing/unknown interface used for bridge setup
SOLVED: default route missing after bridge device created
SOLVED: Vagrant randomly hanging on SSH private key
SOLVED OpenStack install error: Currently installed pip version 1 does not meet minimum requirements
SOLVED Openstack component keystone has no stable/newton branch (which is what onap has selected for devstack)
SOLVED Issue: Openstack error message cells not mapped on Ocata release
SOLVED: compute node service failed to start and devstack timesout with hostname error
SOLVED: public endpoint for orchestration service not found
SOLVED: Openstack xxlarge flavor not found

Current

🚧 ONAP error => ERROR [org.jboss.msc.service.fail] (MSC service thread 1-6) MSC000001: Failed to start service jboss.deployment.unit."mso-api-handler-infra-1.2.1.war".

taylor commented 6 years ago

ISsue found: Vagrant hanging on SSH Private key insertion randomly.

Reference tickets

disable inserting key https://github.com/hashicorp/vagrant/issues/5186 zunayed

5186 "Warning: Authentication failure. Retrying... " after packaging box

I am able to vagrant up and have ssh running fine from a base box that I've provisioned.
I then package up the said box and keep getting errors with ssh. I've packaged numerious boxes before with this same process just yesterday. Not sure what the issue is now.

`vagrant package --base "<name from VBoxManage list vms>"` Show more… Assignees

mitchellh

Labels

bug, core

hashicorp/vagrantJan 14th, 2015 at 17:20 Added by GitHub

Taylor Carpenter [40 minutes ago] possible key sometimes overriding the random key https://github.com/hashicorp/vagrant/issues/5186#issuecomment-312349002 skinneejoe Comment on #5186 "Warning: Authentication failure. Retrying... " after packaging box Below is an explanation of why this problem occurs, why solutions in this thread work for some and not others, how Vagrant has failed us, and a side note on how Ubuntu has failed in creating their base boxes.

If you are having trouble understanding the solutions in this thread please read my experiences below, I think it will help you.

How Vagrant authenticates to boxes Show more… hashicorp/vagrantJun 30th, 2017 at 14:03 Added by GitHub

Solution: Add config.ssh.insert_key = false to Vagrantfile

taylor commented 6 years ago

OpenStack (devstack) install error: Currently installed pip version 1 does not meet minimum requirements

==> control: 2018-04-18 14:49:32.325 | [ERROR] /home/vagrant/devstack/inc/python:146 Currently installed pip version 1 does not meet minimum requi
rements (>=6).                                                           
==> control: 2018-04-18 14:49:33.333 | ++ ./stack.sh:main:739                      :   err_trap       
==> control: 2018-04-18 14:49:33.335 | ++ ./stack.sh:err_trap:524                  :   local r=1                                                  
==> control: 2018-04-18 14:49:33.339 | stack.sh failed: full log in stack.sh.log.2018-04-18-144818
==> control: 2018-04-18 14:49:33.341 | Error on exit                                                                                              
==> control: 2018-04-18 14:49:33.342 | ./stack.sh: line 508: generate-subunit: command not found
The SSH command responded with a non-zero exit status. Vagrant                                                                                    
assumes that this means the command failed. The output for this command             
should be in the log above. Please read the output to determine what                                                                              
went wrong.

taylor commented 6 years ago

regarding the pip version

https://bugs.launchpad.net/devstack/+bug/1764760

talks about splitting version to find it. The last post is 4/17/2018

fixed in master http://git.openstack.org/cgit/openstack-dev/devstack/tree/inc/python?h=master#n340

old code used strip (http://git.openstack.org/cgit/openstack-dev/devstack/tree/inc/python?h=stable/newton#n142)

    local pip_version
    pip_version=$(python -c "import pip; \
                        print(pip.__version__.strip('.')[0])")
    if (( pip_version<6 )); then
        die $LINENO "Currently installed pip version ${pip_version} does not" \
            "meet minimum requirements (>=6)."
    fi

when it should have used split

    local pip_version
    pip_version=$(python -c "import pip; \
                        print(pip.__version__.split('.')[0])")
    if (( pip_version<6 )); then
        die $LINENO "Currently installed pip version ${pip_version} does not" \
            "meet minimum requirements (>=6)."
    fi

SOLUTION: create patch and apply from bootstrap.sh

pip_version_check_fix.patch

taylor commented 6 years ago

New Issue: No stable/newton branch on keystone repository

stable/newton is what the onap demo uses for devstack install. It is not available under the same name for keystone so devstack fails to install

2018-04-18 16:57:28.399 | + functions-common:git_clone:540           :   git checkout stable/newton                                               
2018-04-18 16:57:28.403 | error: pathspec 'stable/newton' did not match any file(s) known to git.

Newton has been EOL: https://github.com/openstack/keystone/tree/newton-eol

Devstack does not have a newton-eol tag instead they have a stable/newton branch https://github.com/openstack-dev/devstack/tree/stable/newton

taylor commented 6 years ago

Issue: compute node service failed to start and devstack timesout with hostname error


    compute-1: 2018-04-19 16:40:12.341 | Didn't find service registered by hostname after 120 seconds
    compute-1: 2018-04-19 16:40:12.344 | + functions:wait_for_compute:420           :   openstack --os-cloud devstack-admin --os-region RegionOne compute service list
    compute-1: 2018-04-19 16:40:13.259 | +----+------------------+---------+----------+---------+-------+----------------------------+
    compute-1: 2018-04-19 16:40:13.259 | | ID | Binary           | Host    | Zone     | Status  | State | Updated At                 |
    compute-1: 2018-04-19 16:40:13.259 | +----+------------------+---------+----------+---------+-------+----------------------------+
    compute-1: 2018-04-19 16:40:13.259 | |  4 | nova-conductor   | control | internal | enabled | up    | 2018-04-19T16:40:07.000000 |
    compute-1: 2018-04-19 16:40:13.259 | |  6 | nova-scheduler   | control | internal | enabled | up    | 2018-04-19T16:40:12.000000 |
    compute-1: 2018-04-19 16:40:13.259 | |  7 | nova-consoleauth | control | internal | enabled | up    | 2018-04-19T16:40:09.000000 |
    compute-1: 2018-04-19 16:40:13.259 | |  8 | nova-compute     | control | nova     | enabled | up    | 2018-04-19T16:40:09.000000 |
    compute-1: 2018-04-19 16:40:13.259 | +----+------------------+---------+----------+---------+-------+----------------------------+
    compute-1: 2018-04-19 16:40:13.311 | + functions:wait_for_compute:422           :   return 124
    compute-1: 2018-04-19 16:40:13.314 | + lib/nova:is_nova_ready:1                 :   exit_trap
    compute-1: 2018-04-19 16:40:13.318 | + ./stack.sh:exit_trap:494                 :   local r=124
    compute-1: 2018-04-19 16:40:13.322 | ++ ./stack.sh:exit_trap:495                 :   jobs -p
    compute-1: 2018-04-19 16:40:13.328 | + ./stack.sh:exit_trap:495                 :   jobs=
    compute-1: 2018-04-19 16:40:13.333 | + ./stack.sh:exit_trap:498                 :   [[ -n '' ]]
    compute-1: 2018-04-19 16:40:13.338 | + ./stack.sh:exit_trap:504                 :   kill_spinner
    compute-1: 2018-04-19 16:40:13.343 | + ./stack.sh:kill_spinner:390              :   '[' '!' -z '' ']'
    compute-1: 2018-04-19 16:40:13.347 | + ./stack.sh:exit_trap:506                 :   [[ 124 -ne 0 ]]
    compute-1: 2018-04-19 16:40:13.350 | + ./stack.sh:exit_trap:507                 :   echo 'Error on exit'
    compute-1: 2018-04-19 16:40:13.351 | Error on exit
    compute-1: 2018-04-19 16:40:13.354 | + ./stack.sh:exit_trap:508                 :   generate-subunit 1524155317 696 fail
    compute-1: 2018-04-19 16:40:13.635 | + ./stack.sh:exit_trap:509                 :   [[ -z /opt/stack/logs ]]
    compute-1: 2018-04-19 16:40:13.639 | + ./stack.sh:exit_trap:512                 :   /home/vagrant/devstack/tools/worlddump.py -d /opt/stack/logs
    compute-1: 2018-04-19 16:40:14.220 | + ./stack.sh:exit_trap:518                 :   exit 124
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
root@cnfs-dev:/cnfs/demo-amsterdam/vagrant# vim Vagrantfile                         
Taglist: Exuberant ctags (http://ctags.sf.net) not found in PATH. Plugin is not loaded.
Press ENTER or type command to continue

On compute node in screen session (screen -x under vagrant) the following was found

2018-04-19 16:38:10.764 11207 INFO nova.service [-] Starting compute node (version 15.1.1)
2018-04-19 16:38:10.766 11207 ERROR oslo_service.service [-] Error starting thread.
2018-04-19 16:38:10.766 11207 ERROR oslo_service.service Traceback (most recent call last):
2018-04-19 16:38:10.766 11207 ERROR oslo_service.service   File "/usr/local/lib/python2.7/dist-packages/oslo_service/service.py", line 722, in run_service
2018-04-19 16:38:10.766 11207 ERROR oslo_service.service     service.start()
2018-04-19 16:38:10.766 11207 ERROR oslo_service.service   File "/opt/stack/nova/nova/service.py", line 144, in start
2018-04-19 16:38:10.766 11207 ERROR oslo_service.service     self.manager.init_host()
2018-04-19 16:38:10.766 11207 ERROR oslo_service.service   File "/opt/stack/nova/nova/compute/manager.py", line 1140, in init_host
2018-04-19 16:38:10.766 11207 ERROR oslo_service.service     raise exception.PlacementNotConfigured()
2018-04-19 16:38:10.766 11207 ERROR oslo_service.service PlacementNotConfigured: This compute is not configured to talk to the placement service. Configure the [placement] section of nova.conf and restart the service.
2018-04-19 16:38:10.766 11207 ERROR oslo_service.service
2018-04-19 16:38:10.766 11207 DEBUG oslo_concurrency.lockutils [req-5efe3368-5adc-4f6e-b3b2-fd5381939556 - -] Acquired semaphore "singleton_lock" lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:212
2018-04-19 16:38:10.767 11207 DEBUG oslo_concurrency.lockutils [req-5efe3368-5adc-4f6e-b3b2-fd5381939556 - -] Releasing semaphore "singleton_lock" lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:225

POSSIBLE SOLUTION: add placement-api to enabled services in local.conf

taylor commented 6 years ago

Issue: public endpoint for orchestration service not found

heat service not started.

Enabled in control.conf. destroyed and brought back up the control node

enable_plugin heat https://git.openstack.org/openstack/heat stable/ocata
ENABLED_SERVICES+=heat,h-eng,h-api,h-api-cfn,h-api-cw

taylor commented 6 years ago

Issue: heat template references xxlarge flavor that is not set causing onap setup to exit

Updated setup_contoller.sh to create a xxlarge flavor and added to the create_onap script.

create_onap.sh:

sed -i "s/flavor_xxlarge:.*/flavor_xxlarge: m1.xxlarge/" onap_openstack.env

setup_controller.sh:

openstack flavor delete m1.xxlarge
openstack flavor create --public m1.xxlarge --id auto --ram 8196 --vcpus 4 --disk 100

taylor commented 6 years ago

Issue: ubuntu packages not available

    onap: E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/t/tiff/libtiff5_4.0.6-1ubuntu0.3_amd64.deb  404  Not Found [IP: 91.189.88.149 80]
    onap:                                 
    onap: E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?                                                  
The SSH command responded with a non-zero exit status. Vagrant                       
assumes that this means the command failed. The output for this command              
should be in the log above. Please read the output to determine what                 
went wrong.

Onap-Node needs to run apt update before installing openstack packages.

Add:

apt update -y

to the onap provision script.

taylor commented 6 years ago

Issue: resolution for domain portal.api.simpledemo.openecomp.org

    onap:  192.168.1.218 dcae.api.simpledemo.openecomp.org                           
    onap: + ssh -o StrictHostKeyChecking=no ubuntu@portal.api.simpledemo.openecomp.org -i onap 'curl sina.com.cn'                                
    onap: ssh: Could not resolve hostname portal.api.simpledemo.openecomp.org: Name or service not known                                         
The SSH command responded with a non-zero exit status. Vagrant                       
assumes that this means the command failed. The output for this command              
should be in the log above. Please read the output to determine what                 
went wrong.

taylor commented 6 years ago

Issue: onap vms that are created get assigned a floating ip from the NAT "public subnet" and is not SSH'able from host. Bridging does not work on Packet.net

Fix: @denverwilliams said:

So i had to use an internal virtual box network "layer 2/switch" network for all the vms and when we bring up the openstack compute nodes we also bring up a router vm which can provide internet access to the internal network.

taylor commented 6 years ago

Issue: Openstack hosts not mapped to cells error on Ocata release

Solution: Run nova-manage cell_v2 simple_cell_setup after the compute nodes are up before using any OpenStack resources

taylor commented 6 years ago

Issue: ONAP mso-api-handler-infra failed to start service - class path issue?

17:00:36,019 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0028: Stopped deployment MSOInfrastructureBPMN-1.2
.1.war (runtime-name: MSOInfrastructureBPMN-1.2.1.war) in 80492ms
17:00:55,673 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-6) WFLYSRV0059: Class Path entry scala-reflect.jar in /content/mso-api-handler-infra-1.2.1.war/WEB-INF/lib/scala-compiler-2.12.4.jar  does not point to a valid jar for a Class-Path reference.   17:00:55,679 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-6) WFLYSRV0059: Class Path entry scala-library.jar in /content/mso-api-handler-infra-1.2.1.war/WEB-INF/lib/scala-compiler-2.12.4.jar  does not point to a valid jar for a Class-Path reference.   17:00:56,353 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-6) MSC000001: Failed to start service jboss.deployment.unit."mso-api-handler-infra-1.2.1.war".STRUCTURE: org.jboss.msc.service.StartException in service jboss.deployment.unit."mso-api-handler-infra-
1.2.1.war".STRUCTURE: Failed to start service
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1904)                                         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)                                                    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Container is down
        at org.jboss.msc.service.ServiceContainerImpl.install(ServiceContainerImpl.java:716)
        at org.jboss.msc.service.ServiceTargetImpl.install(ServiceTargetImpl.java:223)
        at org.jboss.msc.service.ServiceControllerImpl$ChildServiceTarget.install(ServiceControllerImpl.java:2401)
        at org.jboss.msc.service.ServiceTargetImpl.install(ServiceTargetImpl.java:223)                                                        at org.jboss.msc.service.ServiceControllerImpl$ChildServiceTarget.install(ServiceControllerImpl.java:2401)
        at org.jboss.msc.service.ServiceBuilderImpl.install(ServiceBuilderImpl.java:317)                                                      at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:208)
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)                                at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)
        ... 3 more

taylor commented 6 years ago

Copying steps from https://github.com/cncf/cnfs/issues/3#issuecomment-389648974

Steps to setup new system and provision openstack:

Provision base dev software and pull down onap-demo code:

git clone git@github.com:cncf/onap-demo.git -b amsterdam-openstack-ocata

cd onap-demo

./vagrant/run_demo.sh

Deploy the router to new packet server:

cd router-node ; vagrant plugin install vagrant-reload ; vagrant up

Deploy openstack to new packet server and run onap creation scripts:

cd ../vagrant ; vagrant up

After this completes OpenStack (ocata release) will be running.

Some ONAP components will be up others fail.

LexCC commented 5 years ago

Copying steps from cncf/cnf-testbed#3 (comment)

Steps to setup new system and provision openstack:

Provision base dev software and pull down onap-demo code:
git clone git@github.com:cncf/onap-demo.git -b amsterdam-openstack-ocata

cd onap-demo

./vagrant/run_demo.sh
Deploy the router to new packet server:
cd router-node ; vagrant plugin install vagrant-reload ; vagrant up
Deploy openstack to new packet server and run onap creation scripts:
cd ../vagrant ; vagrant up
After this completes OpenStack (ocata release) will be running.

Some ONAP components will be up others fail.

I got an error fatal: unable to access 'https://git.openstack.org/openstack/heat/': Failed to connect to git.openstack.org port 443: No route to host

taylor commented 5 years ago

@JieJhih, this repo is not currently being maintained and is only hear for historical purposes. Testing and development of Cloud Native Network Function use cases has moved to the CNCF CNF Testbed.

For current ONAP documentation see the ONAP Wiki (specifically the Developing ONAP page or ONAP docs site (Amsterdam, Casablanca]. The current version of the ONAP demo code is at https://github.com/onap/demo. The ONAP Mailing list are a good place to get help.

That said, it looks like the issue you are seeing is caused by git.openstack.org being replaced by a new host opendev.org. So the URL would be https://opendev.org/openstack/heat. To resolve the error the openstack install code would need to be updated to support the new URL.

The intent of the instructions was for initial analysis of the ONAP amsterdam demo which builds a OpenStack cluster inside of virtual machines. Most of the effort was making sure the host machines and networking was repeatable and updating the OpenStack code to work. This was not meant to be used long term though and it served it's purpose.

I'm not sure what your goals are with trying to get this code up and running, but in general I recommend not using it. For ONAP use the newer releases. For general CNF use case testing there are several choices including the CNCF CNF Testbed.

You can reach the CNF Testbed team on CNCF's Slack in the #cnf-testbed channel in addition to the Github repo https://github.com/cncf/cnf-testbed.

taylor commented 5 years ago

Closing this ticket as complete for original intent.

Open new tickets for any further issues -- noting that the repo is not currently maintained.

cncf / onap-demo