Error Connecting to NFVO via SSL on port 8443

[wittling]

I am running many other components just fine (nvfo, generic-vnfm, zabbix plugin, et al). I compiled the source code - version 1.2.4-SNAPSHOT.jar.

Cannot seem to even get the service to start. I can log into the dashboard at the IP and Port (8443) just fine. But when I start the autoscaling-ase service up, I see that it cannot connect and finally throws an exception.

I know this ip and port work, because I am logged into it on the dashboard. Of course aside of that, a netstat tln does show port 8443 listening. I have tried localhost, and have also tried the actual IP of the instance.

I did put in a external driver.properties file for the openstack4j plugin and set the values on disable-ssl-certificate-check to both false and true, to see if maybe it was an ssl error. I still think it could be an ssl error, but maybe it's not.

At this point, I cannot get the autoscaling engine to work until and unless I can figure out a solution for this. I'm out of clues, so I'll post this.

Configuration File:

Direct log to a log file

logging.file=/var/log/openbaton/autoscaling.log

Logging levels settings

logging.level.org.springframework=INFO logging.level.org.apache.activemq=WARN logging.level.org.openbaton=INFO logging.level.org.openbaton.autoscaling=INFO

AutoScaling settings

autoscaling.server.ip=localhost autoscaling.server.port=9999 autoscaling.management.port=9998

autoscaling.rabbitmq.brokerIp=172.31.0.136 autoscaling.rabbitmq.management.port=15672 autoscaling.rabbitmq.autodelete=true autoscaling.rabbitmq.durable=false autoscaling.rabbitmq.exclusive=false autoscaling.rabbitmq.minConcurrency=5 autoscaling.rabbitmq.maxConcurrency=15

autoscaling.plugin.dir = ./plugins autoscaling.plugin.startup = true

autoscaling.plugin.log.dir = ${autoscaling.plugin.dir}/logs

autoscaling.plugin.log.dir = /var/log/openbaton/plugins/ase

Port settings

server.port=${autoscaling.server.port} management.port=${autoscaling.management.port}

NFVO settings

nfvo.ip=localhost nfvo.port=8443 nfvo.username=[redacted out but it is correct] nfvo.password=[redacted out but it is correct]

######################################### ############## RabbitMQ ################# #########################################

Comma-separated list of addresses to which the client should connect to.

spring.rabbitmq.addresses= 192.168.145.54

Create an AmqpAdmin bean.

spring.rabbitmq.dynamic=true

RabbitMQ host.

spring.rabbitmq.host=${autoscaling.rabbitmq.brokerIp} .... (remainder is original unchanged)

Log File

2017-09-06 17:13:52.089 INFO 25579 --- [main] o.s.w.s.handler.SimpleUrlHandlerMapping : Mapped URL path [/] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler] 2017-09-06 17:13:52.090 INFO 25579 --- [main] o.s.w.s.handler.SimpleUrlHandlerMapping : Mapped URL path [/webjars/] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler] 2017-09-06 17:13:52.110 INFO 25579 --- [main] .m.m.a.ExceptionHandlerExceptionResolver : Detected @ExceptionHandler methods in globalExceptionHandler 2017-09-06 17:13:52.152 INFO 25579 --- [main] o.s.w.s.handler.SimpleUrlHandlerMapping : Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler] 2017-09-06 17:13:52.298 INFO 25579 --- [main] o.s.j.e.a.AnnotationMBeanExporter : Registering beans for JMX exposure on startup 2017-09-06 17:13:52.345 INFO 25579 --- [main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 9999 (http) 2017-09-06 17:14:07.855 ERROR 25579 --- [main] org.openbaton.plugin.mgmt.PluginStartup : After 15 seconds the plugin is not started. 2017-09-06 17:14:23.452 ERROR 25579 --- [main] org.openbaton.plugin.mgmt.PluginStartup : After 15 seconds the plugin is not started. 2017-09-06 17:14:23.453 INFO 25579 --- [main] org.openbaton.autoscaling.utils.Utils : Waiting until NFVO is available... 2017-09-06 17:14:23.454 INFO 25579 --- [main] org.openbaton.autoscaling.utils.Utils : NFVO is listening on port 8443 at 172.31.0.136 2017-09-06 17:14:23.469 ERROR 25579 --- [main] org.openbaton.sdk.api.rest.ProjectAgent : 172.31.0.136:8443 failed to respond

org.apache.http.NoHttpResponseException: 172.31.0.136:8443 failed to respond at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)

[gc4rella]

hi @wittling, it seems that there is an issue with the autoscaling engine when the NFVO has SSL enabled. We are looking at it and will provide a fix as soon as we can.

[wittling]

Thank you.

While I wait for you to do that, I think I need to spin up a new server. I am running KVM (32 Gb RAM) and am running OpenStack Controller on one virtual machine (CentOS 7) in this KVM, and OpenBaton in another (Ubuntu) virtual machine (that runs Zabbix also in addition to OpenBaton). The KVM itself behaves as a Compute Node.

FYI. The Ubuntu 14.04 "auto upgraded" itself to 16.04 - which seems to run OpenBaton okay, although I know you state that you only support 14.0.4.

One issue I am seeing, is that if I spin up several VMs with a NSD JSON, the box seems to get overwhelmed. I get numerous Disk I/O issues reported in Zabbix, as well as IRQ4 errors, and the VMs seem to initialize properly only about 20% of the time. I see these VMs doing a lot of work through CloudInit (I see it installing a bunch of Perl packages, et al before it tries to launch the ems service).

I am going to offload to a 2nd server either the OpenBaton KVM, or the Openstack Controller VM. Do you have a suggestion on which one I should offload to solve this issue?

[gc4rella]

I think you have a pretty complex environment :-)

I think you should separate both control entities (openstack controller and openbaton) from the compute nodes where VMs are executing. I believe that the disk I/O issues you are getting are mainly due to the VMs executions. usually OB does not have very demanding requirements in terms of disk I/O for rather small amount of network service deployments (all it does is to write/read to/from the DB whenever some operations are executed).

Anyway, we had similar issues in the past when using no SSD disks. I suggest you to make some tests only on your openstack environment, so that you can at least identify a potential root cause of your issues. For instance, you could try spawining 10/15 VMs in parallel directly via openstack, and check the performances.

[wittling]

I took your advice. I hooked up a second server. So I now have 2 servers that are very similar (32 Gb RAM, 8 Core - HP Envy).

On Server A (original server):

1. I run LibvirtD
2. I am still running the OpenBaton in the Ubuntu 16.04 VM.
3. I am running Zabbix Agent in this same Ubuntu VM that runs OpenBaton.
4. On the host (no VM), I run OpenStack Nova Compute, and the Neutron LinuxBridge Agent The KVM on this box remains where the VMs will get spawned - although I am also running KVM on the second box and could make that a second compute node if I need to.

On Server B (new Server):

5. I am running KVM / LibvirtD
6. One of the VMs is dedicated to running OpenStack Controlller. I simply migrated the OpenStack Controller from Server A to Server B.
7. I run the Zabbix Server on this box (on the KVM host - not in a VM).

I had a horrible issue trying to spawn new VMs to the Server A LibvirtD from this controller. The VMs would just hang and error out, or they would launch VMs with a XFS Mount Error, or the GuestFS would just spawn up and down (depending on configuration adjustments to nova.conf). Just today, I fixed this problem. I debugged guestfs as the issue, and I removed the guestfs packages, and re-installed the nova-compute by using the rpm and the --nodeps flag. Apparently yum has a hard dependency between guestfs and nova-compute, and if it is installed, it will use guestfs regardless of configuration settings.

So now that I finally have the OpenStack Controller on a VM in Box B, and OpenBaton in a VM on Box A, I am ready to dive back in now and do some more testing.

Is there any progress on the SSL issue for this? Fault Management has same issue. I am keenly interested in trying these services out. Particularly the auto-scaling.

[mpauls]

SSL support is now included in the develop branch. Be aware about the new properties in the configuration file and the changes which need to be done, for instance:

• nfvo.ssl.enabled=false -> true
• nfvo.port=8080 -> 8443

I see that you probably used the master/release version. Keep in mind when using the develop branch that there are some big changes between master and develop for external components (e.g. autoscaling-engine, fault management) due to the fact that we changed the registration approach with the introduction of so called "services".

[wittling]

I downloaded this, and got a Major / Minor error. What version of Java should I be compiling this with?

I am (and have been) using version: java version "1.7.0_131" OpenJDK Runtime Environment (IcedTea 2.6.9) (7u131-2.6.9-3) OpenJDK 64-Bit Server VM (build 24.131-b00, mixed mode)

Earlier on, the openbaton bootstrap scripts were requiring Oracle JDK1.7, but Oracle had removed it from their site (causing the bootstrap script to complain loudly). Because I could not tell if the bootstrap script was running to completion or not because of this, I downloaded the OpenJDK1.7 and compiled all modules one-by-one.

Should I be compiling this new development branch with 1.8 or later? And if so, will it work with all of these other openbaton modules (nfvo, generic-vnfm, zabbix plugin, et al) that were previously compiled with the above 1.7 OpenJDK?

[wittling]

I went ahead and downloaded OpenJDK 1.8 and compiled it - and changed systemd unit file to launch it with the 1.8 java binary. Seems to be running now. But in looking at the log, I see the following error:

2017-10-03 17:14:18.887 INFO 3446 --- [main] org.openbaton.autoscaling.utils.Utils : Waiting until NFVO is available... 2017-10-03 17:14:18.887 INFO 3446 --- [main] org.openbaton.autoscaling.utils.Utils : NFVO is listening on port 8443 at localhost 2017-10-03 17:14:18.895 ERROR 3446 --- [main] org.openbaton.sdk.api.rest.ProjectAgent : missing key file for services

I am stuck until I can get more insight on what this message means and what I need to do to address the error.

[ranzpirat]

This has most likely to do with the new serviceagent registration. In your orchestrator dashboard check the tab "Admin" and "Services" and Press the Enable a new Service button.

Alternatively, get this file by using the openbaton-cli (fill in the placeholder variables with your valid values)

$ cd openbaton-cli
$ sudo python setup.py install # might want to do this in a venv instead
$ # first get the project-id where you want to register your service (ase)
$ openbaton -pid none -u $PASSWORD -p $USER -ip $NFVO_IP --nfvo-port $NFVO_PORT project list
$ # then create the service and receive the keyfile
$ openbaton -pid $PROJECT_ID -u $PASSWORD -p $USER -ip $NFVO_IP --nfvo-port $NFVO_PORT service create '{"name":"autoscaling-engine", "roles":["*"]}' > autoscaling-engine.txt

Then add this file to the envrionment where your ASE is running and change the configuration paramterer here

[flaviomu]

@wittling regarding your question:

Should I be compiling this new development branch with 1.8 or later? And if so, will it work with all of these other openbaton modules (nfvo, generic-vnfm, zabbix plugin, et al) that were previously compiled with the above 1.7 OpenJDK?

yes, to compile the latest development branch you need OpenJDK1.8 (for the NFVO as well as for the other components). On which OS are you installing Open Baton? Only Ubuntu Xenial (or later) (which should already have OpenJDK1.8) is supported for this latest development branch.

[wittling]

I run OpenBaton on a KVM virtual machine that originally ran Ubuntu 14.04, but it was updated to Ubuntu 16.04. CentOS is the standard OS in this shop, so I'm more familiar with CentOS than I am Ubuntu, but I'm familiar "enough" with Ubuntu (for a time we had to support that OS so we all used it to some extent).

I noticed that I had the OpenJDK1.8 - but "just" the jre version - not the jdk version. The jre version is missing tools.jar, which is required to compile. So I installed the jdk version 1.8 and recompiled successfully, and changed systemd unit file to launch it the 1.8 class loader and it looks like it works fine.

The other stuff i'm running is all running on version 1.7. I don't see any errors, so I'll leave it that way because everything else seems to be working well and I'm doing considerable testing with OpenBaton and don't want to lose progress. I'll check the suggestions from ranzpirat above to see if I can get the "missing key file" fixed, and I should - hopefully - be good to go.

[wittling]

@ranzpirat My dashboard does not have an "Admin / Services". Under Admin on my dashboard, which has a version of "NFVO version v4.0.0" in lower left corner, I only have the options Projects, Users and Vim Drivers. This concerns me a bit - am I needing to upgrade my NFVO?

I will try your CLI approach and see what happens with that.

[ranzpirat]

@wittling sorry if that confused you, because of the error message I assumed you were using develop on orchestrator side already. I just responded with that because of the error message which points in that direction. Unfortunately I don't know what/where @mpauls did the fix for SSL and if that requires to update to develop or is also available as hotfix for release-tag.

[wittling]

@ranzpirat no problem! Actually, what you posted there was a big help. I thought that the openbaton client and openbaton cli were the same thing (the openbaton client git repository has both an sdk and cli directory within them that get built). I now realize from your link that the openbaton-cli is a different repository, so I have brought that down, and I will build that and see if I can follow those steps you've listed above.

[ranzpirat]

@wittling Sure thing. Note that the steps I listed above are only usable on a develop (would show up as 4.0.1-Snapshot e.g. in dashboard) environment, as the refactoring around the service-agents was introduced after 4.0.0 release and is also likely to change in the future/as testing continues.