search

CSCfi / pebbles

Pebbles is a frontend to manage cloud resources and lightweight user accounts.
MIT License
7 stars 6 forks source link

Installation of Pouta-Blueprints stopped at "network bridge not found" #529

Open nanjiangshu opened 8 years ago

nanjiangshu commented 8 years ago

Dear Developers of Pouta Blueprints,

I'm a developer at NBIS, Sweden and we are interested in applying Pouta-Blueprints at here at the Swedish SNIC cloud. However, when I tried to install the software following the instruction at https://github.com/CSC-IT-Center-for-Science/pouta-blueprints/blob/master/doc/how_to_install_on_cpouta.md.

My installation procedure aborted with the message

TASK: [single_server_with_docker | check or build image] ****** failed: [localhost] => {"changed": true, "failed": true, "image_id": null} msg: Error: network bridge not found Log:Step 1 : FROM ubuntu:14.04 ---> 43c910d8ff01 Step 2 : MAINTAINER Olli Tourunen "olli.tourunen@csc.fi" ---> Running in 125ef902c21b ---> cadcb033cd68 Removing intermediate container 125ef902c21b Step 3 : RUN apt-get update ---> Running in 404b74e8adf3

FATAL: all hosts have already failed -- aborting

And the full installation message can be found in the attached file.

Any help is highly appreciated.

Best regards,

Nanjiang install_pb.bash.log.txt

tourunen commented 8 years ago

Hi Nanjiang,

are you installing on CentOS-7? That is the only host OS really supported out of the box nowadays (there may be some leftovers from the early days in install_pb.bash for Ubuntu, but that not been tested or maintained at all). The instructions do mention that the installation will be based on CentOS-7.

Is it ok for you to use CentOS?

The containers run Ubuntu, mostly for historical reasons.

nanjiangshu commented 8 years ago

Hi,

Thanks for your quick reply. I tried it also on a VM with CentOS7. But it is now aborted even at earlier stage with the error message:

TASK: [docker_host | Add docker repo key (Ubuntu)] **** fatal: [localhost] => error while evaluating conditional: ansible_lsb.id=="Ubuntu"

FATAL: all hosts have already failed -- aborting

PLAY RECAP **** to retry, use: --limit @/home/centos/playbook.retry

localhost : ok=1 changed=0 unreachable=1 failed=0

It seems it is not working on a server with CentOS7.

What I'm missing here?

Best,

Nanjiang

tourunen commented 8 years ago

Not sure, but could it be that the CentOS image that you are using does not have lsb packages installed? The instructions are based on the image that is publicly available in cPouta, so you may need to add packages and maybe even tweak a thing or two to get it going on another base image. This is how it looks like on a fresh CentOS-7 VM in cPouta:

$ rpm -qa | grep lsb
redhat-lsb-submod-security-4.1-27.el7.centos.1.x86_64
redhat-lsb-core-4.1-27.el7.centos.1.x86_64
nanjiangshu commented 8 years ago

Hi,

I have installed the lsb package and the installation script install_pb.bash progressed a bit further. However, it seems it hangs at the step

TASK: [single_server_with_docker | check or build image] **

For more than one hour. Is that normal or how long time am I supposed to wait?

Best,

Nanjiang

nanjiangshu commented 8 years ago

Hi,

This step finally finished but aborted with error. It seems something was wrong with the apt-get of the Ubuntu within the container. See full error message below.

TASK: [single_server_with_docker | check or build image] ** failed: [localhost] => {"changed": true, "failed": true, "image_id": null} msg: Error: The command '/bin/sh -c apt-get update' returned a non-zero code: 100 Log:Step 1 : FROM ubuntu:14.04 ---> f2d8ce9fa988 Step 2 : MAINTAINER Olli Tourunen "olli.tourunen@csc.fi" ---> Using cache ---> ff2c0c50dca6 Step 3 : RUN apt-get update ---> Running in c31f9fb643d4 Ign http://archive.ubuntu.com trusty InRelease Ign http://archive.ubuntu.com trusty-updates InRelease Ign http://archive.ubuntu.com trusty-security InRelease Get:1 http://archive.ubuntu.com trusty Release.gpg [933 B] Get:2 http://archive.ubuntu.com trusty-updates Release.gpg [933 B] Get:3 http://archive.ubuntu.com trusty-security Release.gpg [933 B] Ign http://archive.ubuntu.com trusty Release Ign http://archive.ubuntu.com trusty-updates Release Ign http://archive.ubuntu.com trusty-security Release Err http://archive.ubuntu.com trusty/main Sources 404 Not Found [IP: 91.189.88.152 80] Err http://archive.ubuntu.com trusty/restricted Sources 404 Not Found [IP: 91.189.88.152 80] Err http://archive.ubuntu.com trusty/universe Sources 404 Not Found [IP: 91.189.88.152 80] Err http://archive.ubuntu.com trusty/main amd64 Packages 404 Not Found [IP: 91.189.88.152 80] Err http://archive.ubuntu.com trusty/restricted amd64 Packages 404 Not Found [IP: 91.189.88.152 80] Err http://archive.ubuntu.com trusty/universe amd64 Packages 404 Not Found [IP: 91.189.88.152 80] Err http://archive.ubuntu.com trusty-updates/main Sources 404 Not Found [IP: 91.189.88.152 80] Err http://archive.ubuntu.com trusty-updates/restricted Sources 404 Not Found [IP: 91.189.88.152 80] Err http://archive.ubuntu.com trusty-updates/universe Sources 404 Not Found [IP: 91.189.88.152 80] Err http://archive.ubuntu.com trusty-updates/main amd64 Packages 404 Not Found [IP: 91.189.88.152 80] Err http://archive.ubuntu.com trusty-updates/restricted amd64 Packages 404 Not Found [IP: 91.189.88.152 80] Err http://archive.ubuntu.com trusty-updates/universe amd64 Packages 404 Not Found [IP: 91.189.88.152 80] Err http://archive.ubuntu.com trusty-security/main Sources 404 Not Found [IP: 91.189.88.152 80] Err http://archive.ubuntu.com trusty-security/restricted Sources 404 Not Found [IP: 91.189.88.152 80] Err http://archive.ubuntu.com trusty-security/universe Sources 404 Not Found [IP: 91.189.88.152 80] Err http://archive.ubuntu.com trusty-security/main amd64 Packages 404 Not Found [IP: 91.189.88.152 80] Err http://archive.ubuntu.com trusty-security/restricted amd64 Packages 404 Not Found [IP: 91.189.88.152 80] Err http://archive.ubuntu.com trusty-security/universe amd64 Packages 404 Not Found [IP: 91.189.88.152 80] Fetched 2799 B in 2h 44min 9s (0 B/s) W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/main/source/Sources 404 Not Found [IP: 91.189.88.152 80]

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/restricted/source/Sources 404 Not Found [IP: 91.189.88.152 80]

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/universe/source/Sources 404 Not Found [IP: 91.189.88.152 80]

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/main/binary-amd64/Packages 404 Not Found [IP: 91.189.88.152 80]

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/restricted/binary-amd64/Packages 404 Not Found [IP: 91.189.88.152 80]

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/universe/binary-amd64/Packages 404 Not Found [IP: 91.189.88.152 80]

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/main/source/Sources 404 Not Found [IP: 91.189.88.152 80]

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/restricted/source/Sources 404 Not Found [IP: 91.189.88.152 80]

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/universe/source/Sources 404 Not Found [IP: 91.189.88.152 80]

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/main/binary-amd64/Packages 404 Not Found [IP: 91.189.88.152 80]

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/restricted/binary-amd64/Packages 404 Not Found [IP: 91.189.88.152 80]

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/universe/binary-amd64/Packages 404 Not Found [IP: 91.189.88.152 80]

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-security/main/source/Sources 404 Not Found [IP: 91.189.88.152 80]

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-security/restricted/source/Sources 404 Not Found [IP: 91.189.88.152 80]

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-security/universe/source/Sources 404 Not Found [IP: 91.189.88.152 80]

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-security/main/binary-amd64/Packages 404 Not Found [IP: 91.189.88.152 80]

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-security/restricted/binary-amd64/Packages 404 Not Found [IP: 91.189.88.152 80]

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-security/universe/binary-amd64/Packages 404 Not Found [IP: 91.189.88.152 80]

E: Some index files failed to download. They have been ignored, or old ones used instead.

FATAL: all hosts have already failed -- aborting

PLAY RECAP **** to retry, use: --limit @/home/centos/playbook.retry

localhost : ok=19 changed=3 unreachable=0 failed=1

tourunen commented 8 years ago

You could just retry the installation, this could be caused just an unresponsive mirror server. Let's see if that helps. I did a clean PB installation just last week and that went just fine, so I'd guess this is something intermittent.

nanjiangshu commented 8 years ago

I have tried it ten times at different times and also on another VM with centos7 but all failed at this step. Would it be OK for your to login to the VM and check what the problem might be? If you can help please send me your ssh key and I will add it to the VM.

Thanks! Best, Nanjiang

nanjiangshu commented 8 years ago

Hi,

Though I am still not able to complete the installation of Ponta-blueprints on VMs at SNIC cloud, I do managed to install it on one of the VMs at an Italian site of EGI cloud. I guess it might be caused by network problem at SNIC cloud but don't know how to solve it.

After installation, I logged in to the admin page and try to do some test according to the document at https://github.com/CSC-IT-Center-for-Science/pouta-blueprints/blob/master/doc/how_to_install_on_cpouta.md.

I changed the PLUGIN_WHITELIST from DummyDriver to OpenStackDriver, but the plugin list on top did not change even after one hour.

I then tried to create a blueprint based on the DummyDriver. It seems fine, but I don't know how to add the security group. There are two items, "Source" and "Port". I added the Port 22 for SSH but leave the "Source" blank. After that, I launched an instance. The instance launched and no error message showed up. But I could not ssh to it.

Would it be possible to give me a little more instructions than on the github?

Thanks very much!

Best,

Nanjiang

tourunen commented 8 years ago

Hi,

great that you are making progress! First: the DummyDriver is just for testing that the system works in general and for providing a mock plugin for development time that does not need any configuration. It does not provision any instances. Anyway, if you are able to make a blueprint and launch an "instance" based on the DummyDriver, you already know that the communication between worker container and api container works.

The fact that OpenStackDriver does not report back is the problem here. Have you launched the status tmux session (instructions are at the bottom of how_to_install_on_cpouta.md)? Check the worker logs in window, anything suspicious there?

tourunen commented 8 years ago

Just to make sure there are no false expectations of the level of support we are able to provide I should add: Our first priority at the moment is making software that we can run and provide a hosted service for our end users. The second is making PB runnable in cPouta for customers that need private installations.

However, we are happy if someone is willing to contribute to PB to make it more widely adaptable. Bug reports and Pull Requests are especially welcome! Also, pointing out the most critical bits in our very thin documentation is valuable.

nanjiangshu commented 8 years ago

Thanks very much for your help! I'm not sure which one is the worker log from the TMUX panel, could you please take a look at this snapshot if your have time (link: https://www.dropbox.com/s/6hjjce20oy7urs7/Screenshot%202016-10-10%2016.12.56.png?dl=0)

Also I have attached a snapshot of my ponta blueprint settings pontablue.snapshot.pdf

Best, Nanjiang

tourunen commented 8 years ago

Tmux window number 2 has the worker logs, you can navigate to that by either ctrl-b + n two times or ctrl-b + 2.

Another alternative is to just check the logs directly in the worker container at /webapps/pouta_blueprints/logs/. You can either run 

ssh cloud-user@worker

or

docker exec -it worker bash

to open a shell in the worker.

nanjiangshu commented 7 years ago

Hi,

I have managed to downloaded and exported the rocker/rstudio and then set up DockerDriver. But when I tried to launch an instance based on the newly created blueprint based on DockerDriver, it said "HTTP 409 User quota exceeded, contact your administrator in order to get more".

I checked the worker log and noticed that _spawn_host failed. However, creating of the instance by "nova" client works on the same server. Do you have any ideas or hints to how to debug this error?

I've pasted the error message from worker log below

[nanjiang@ponta-centos7-1 ~]$ ssh cloud-user@worker tail -F /webapps/pouta_blueprints/logs/celery-system.log File "/shared_folder/source/pouta_blueprints/drivers/provisioning/docker_driver.py", line 419, in _do_housekeep_locked return self._do_housekeep(token, int(time.time())) File "/shared_folder/source/pouta_blueprints/drivers/provisioning/docker_driver.py", line 456, in _do_housekeep elif not shutdown_mode and self._spawn_new_host(hosts=hosts, cur_ts=cur_ts): File "/shared_folder/source/pouta_blueprints/drivers/provisioning/docker_driver.py", line 512, in _spawn_new_host new_host = self._spawn_host(cur_ts, ramp_up) File "/shared_folder/source/pouta_blueprints/drivers/provisioning/docker_driver.py", line 650, in _spawn_host raise RuntimeError('Failed to spawn a new host: %s' % res['error']) RuntimeError: Failed to spawn a new host: flow failed due to: WrappedFailure: [Failure: novaclient.exceptions.NotFound: The resource could not be found. (HTTP 404), Failure: novaclient.exceptions.NotFound: The resource could not be found. (HTTP 404), Failure: novaclient.exceptions.NotFound: The resource could not be found. (HTTP 404)] [2016-10-19 07:46:02,434: INFO/MainProcess] Task pouta_blueprints.tasks.housekeeping[ab65a137-2eae-4f09-94f3-bbcc7df06a16] succeeded in 2.36110342597s: None [2016-10-19 07:47:00,075: INFO/MainProcess] Received task: pouta_blueprints.tasks.publish_plugins[77d18336-5e08-4b8b-9545-f3b0e863a985] expires:[2016-10-19 07:48:00.066418+00:00] [2016-10-19 07:47:00,079: INFO/MainProcess] Received task: pouta_blueprints.tasks.housekeeping[30cb9c44-b226-49e6-b108-1a3f0d8ee63f] expires:[2016-10-19 07:48:00.074174+00:00] [2016-10-19 07:47:00,085: INFO/Worker-1022] pouta_blueprints.tasks.publish_plugins[77d18336-5e08-4b8b-9545-f3b0e863a985]: provisioning plugins queried from worker [2016-10-19 07:47:00,092: INFO/MainProcess] Received task: pouta_blueprints.tasks.periodic_update[0c287e7f-b794-483a-a2bb-a58c7603ec6e] expires:[2016-10-19 07:48:00.076005+00:00] [2016-10-19 07:47:00,551: INFO/Worker-1023] pouta_blueprints.tasks.periodic_update[0c287e7f-b794-483a-a2bb-a58c7603ec6e]: deprovisioning triggered for 0e81fe12dfa3410492c3ed31115fe614 (reason: maximum lifetime exceeded) [2016-10-19 07:47:00,553: INFO/Worker-1021] pouta_blueprints.tasks.housekeeping[30cb9c44-b226-49e6-b108-1a3f0d8ee63f]: provisioning plugins queried from worker [2016-10-19 07:47:00,618: INFO/MainProcess] Task pouta_blueprints.tasks.periodic_update[0c287e7f-b794-483a-a2bb-a58c7603ec6e] succeeded in 0.519167845137s: None [2016-10-19 07:47:01,452: INFO/Worker-1021] pouta_blueprints.tasks.housekeeping[30cb9c44-b226-49e6-b108-1a3f0d8ee63f]: do_housekeep(): active hosts: 0, free slots: 0 now, 0 projected for near future [2016-10-19 07:47:01,841: INFO/Worker-1022] Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt [2016-10-19 07:47:01,882: INFO/Worker-1022] Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt [2016-10-19 07:47:01,943: INFO/MainProcess] Task pouta_blueprints.tasks.publish_plugins[77d18336-5e08-4b8b-9545-f3b0e863a985] succeeded in 1.85959770507s: None [2016-10-19 07:47:01,966: ERROR/Worker-1021] WrappedFailure: [Failure: novaclient.exceptions.NotFound: The resource could not be found. (HTTP 404), Failure: novaclient.exceptions.NotFound: The resource could not be found. (HTTP 404), Failure: novaclient.exceptions.NotFound: The resource could not be found. (HTTP 404)] [2016-10-19 07:47:01,967: ERROR/Worker-1021] error calling 'DockerDriver': Failed to spawn a new host: flow failed due to: WrappedFailure: [Failure: novaclient.exceptions.NotFound: The resource could not be found. (HTTP 404), Failure: novaclient.exceptions.NotFound: The resource could not be found. (HTTP 404), Failure: novaclient.exceptions.NotFound: The resource could not be found. (HTTP 404)]

Best,

Nanjiang

tourunen commented 7 years ago

Nice to see that you are making progress!

Here you get an error when DockerDriver sees that the number of available slots (0 if it was just turned on) is smaller than the target value (whatever you have set in DD_FREE_SLOT_TARGET) and it tries to spawn a new VM in the pool that is running the containers.

Did you get OpenStackDriver to work? DockerDriver uses the same mechanism to spawn a VM to host the notebooks, and OpenStackDriver may be easier to debug.

nanjiangshu commented 7 years ago

Thanks very much for your reply. I still have problems with OpenStackDriver.

The relevant error message I got is

[2016-10-19 10:12:01,880: WARNING/Worker-1031] pouta_blueprints.tasks.publish_plugins[febdc6e3-cc91-4244-90eb-584302ec4574]: plugin returned empty configuration: OpenStackDriver

Best, Nanjiang

tourunen commented 7 years ago

Hmm, you may need to add more debugging verbosity. Could you turn debugging in the worker container on by setting

DEBUG: True

in /etc/pouta_blueprints/config.yaml in the worker container and try again?

nanjiangshu commented 7 years ago

Hi,

I've set

DEBUG: True

in the file within the container and below is the error message from the worker log. It seems communication with the OpenStack failed. However, the "nova" API works before I started the Docker container by install_pb.bash. If I understand correctly, if "nova" is not working properly the script install_pb.bash will be interrupted in the middle, right?

Also, is there a way to display the "nova" environment variables the running instance is using? I'm suspecting the instance does not load the "nova" environment variables from the m2m RC file correctly.

[2016-10-25 10:00:00,072: INFO/MainProcess] Received task: pouta_blueprints.tasks.publish_plugins[a68a5e9d-1d2a-4a5f-9433-7004902337a6] expires:[2016-10-25 10:01:00.065756+00:00] [2016-10-25 10:00:00,079: INFO/MainProcess] Received task: pouta_blueprints.tasks.housekeeping[24bc212a-0aba-4404-9e18-c151bbdc1317] expires:[2016-10-25 10:01:00.072589+00:00] [2016-10-25 10:00:00,078: INFO/Worker-1547] pouta_blueprints.tasks.publish_plugins[a68a5e9d-1d2a-4a5f-9433-7004902337a6]: provisioning plugins queried from worker [2016-10-25 10:00:00,085: INFO/MainProcess] Received task: pouta_blueprints.tasks.periodic_update[43ae9548-0ce6-487c-af91-fcb29aa98581] expires:[2016-10-25 10:01:00.074966+00:00] [2016-10-25 10:00:00,529: INFO/Worker-1546] pouta_blueprints.tasks.housekeeping[24bc212a-0aba-4404-9e18-c151bbdc1317]: provisioning plugins queried from worker [2016-10-25 10:00:00,715: INFO/MainProcess] Task pouta_blueprints.tasks.periodic_update[43ae9548-0ce6-487c-af91-fcb29aa98581] succeeded in 0.621855325066s: None [2016-10-25 10:00:01,532: INFO/MainProcess] Task pouta_blueprints.tasks.housekeeping[24bc212a-0aba-4404-9e18-c151bbdc1317] succeeded in 1.44731416809s: None [2016-10-25 10:00:01,882: ERROR/Worker-1547] error calling 'OpenStackDriver': The resource could not be found. (HTTP 404) [2016-10-25 10:00:01,883: ERROR/Worker-1547] The resource could not be found. (HTTP 404) Traceback (most recent call last): File "/webapps/pouta_blueprints/venv/local/lib/python2.7/site-packages/stevedore/extension.py", line 255, in _invoke_one_plugin response_callback(func(e, _args, _kwds)) File "/webapps/pouta_blueprints/venv/local/lib/python2.7/site-packages/stevedore/extension.py", line 229, in _call_extension_method return getattr(extension.obj, method_name)(_args, _kwds) File "/shared_folder/source/pouta_blueprints/drivers/provisioning/openstack_driver.py", line 21, in get_configuration images = [x.name for x in oss.list_images()] File "/shared_folder/source/pouta_blueprints/services/openstack_service.py", line 518, in list_images return nc.images.list() File "/webapps/pouta_blueprints/venv/local/lib/python2.7/site-packages/novaclient/v2/images.py", line 69, in list return self._list('/images%s%s' % (detail, query), 'images') File "/webapps/pouta_blueprints/venv/local/lib/python2.7/site-packages/novaclient/base.py", line 64, in _list _resp, body = self.api.client.get(url) File "/webapps/pouta_blueprints/venv/local/lib/python2.7/site-packages/novaclient/client.py", line 451, in get return self._cs_request(url, 'GET', _kwargs) File "/webapps/pouta_blueprints/venv/local/lib/python2.7/site-packages/novaclient/client.py", line 407, in _cs_request self.authenticate() File "/webapps/pouta_blueprints/venv/local/lib/python2.7/site-packages/novaclient/client.py", line 586, in authenticate self._v2_auth(auth_url) File "/webapps/pouta_blueprints/venv/local/lib/python2.7/site-packages/novaclient/client.py", line 651, in _v2_auth return self._authenticate(url, body) File "/webapps/pouta_blueprints/venv/local/lib/python2.7/site-packages/novaclient/client.py", line 664, in _authenticate _kwargs) File "/webapps/pouta_blueprints/venv/local/lib/python2.7/site-packages/novaclient/client.py", line 402, in _time_request resp, body = self.request(url, method, **kwargs) File "/webapps/pouta_blueprints/venv/local/lib/python2.7/site-packages/novaclient/client.py", line 396, in request raise exceptions.from_response(resp, body, url, method) NotFound: The resource could not be found. (HTTP 404) [2016-10-25 10:00:01,884: WARNING/Worker-1547] pouta_blueprints.tasks.publish_plugins[a68a5e9d-1d2a-4a5f-9433-7004902337a6]: plugin returned empty configuration: OpenStackDriver [2016-10-25 10:00:01,889: INFO/MainProcess] Task pouta_blueprints.tasks.publish_plugins[a68a5e9d-1d2a-4a5f-9433-7004902337a6] succeeded in 1.81138614682s: None

Best, Nanjiang

tourunen commented 7 years ago

The credentials are stored on ramdisk in /run/shm/pouta_blueprints/creds on the host and mounted ro in the worker container at the same path.

I did a bit of testing with an instance in cPouta, and if your credentials were invalid you would probably get 

Unauthorized: The request you have made requires authentication. (HTTP 401)

instead of 

NotFound: The resource could not be found. (HTTP 404)

when listing images. To debug, I did this:

ssh worker
source /webapps/pouta_blueprints/venv/bin/activate
cd /webapps/pouta_blueprints/source
sudo apt-get install -y libffi-dev libssl-dev

sudo /webapps/pouta_blueprints/venv/bin/pip install bpython ndg-httpsclient pyasn1 cryptography blessings
sudo su - pouta_blueprints
source /webapps/pouta_blueprints/venv/bin/activate
cd /webapps/pouta_blueprints/source
bpython
import pouta_blueprints.services.openstack_service as openstack_service
oss=openstack_service.OpenStackService({'M2M_CREDENTIAL_STORE': '/run/shm/pouta_blueprints/creds'})
oss.list_images()
nanjiangshu commented 7 years ago

Hi,

The OpenStackDriver can now be loaded by the worker after I changed the Identity API back to version 2.0 (from version 3). However, I encountered another error when I tried to launch an instance.

Provisioning OpenStack instance (414c4e0a40b448078fcf24c33b84cab2)
Provisioning failed flow failed due to: Multiple possible networks found, use a Network ID to be more specific. (HTTP 409) (Request-ID: req-67596caa-4873-44a8-92cf-5abca9825f28)

when I boot a VM, I need to specify the private net-id and then assign a public IP to the VM. Then the question is how can I pass this net-id to the worker? Is there any environmental variable that I can specify in the M2M RC file?

Best,

Nanjiang

glormph commented 7 years ago

Nanjiang and I are still working on this, just so we won't forget (and nobody else wastes time) we have figured out one can:

tourunen commented 7 years ago

@nanjiangshu I don't think PB has been tested with a OpenStack tenant that has multiple networks before, so some additional code might be needed. At least I don't see any parameters for network id in OpenStackService.provision_instance()

    def provision_instance(self, display_name, image_name, flavor_name, public_key, extra_sec_groups=None,
                           master_sg_name=None, allocate_public_ip=True, root_volume_size=0,
                           data_volume_size=0, data_volume_type=None, userdata=None):
glormph commented 7 years ago

We're checking that out now (pair programming) :)

tourunen commented 7 years ago

Great! If you can come up with a generic solution I'm sure we can merge it.

jhagberg commented 7 years ago

I just found out the docker problems we have on SNIC Cloud Its about the MTU. The VMs nic on SNIC cloud gets MTU 1450. Docker0 is started by default with mtu1500. And it will bork things up.

What is the default setting on C-pouta? Also the dual network problem is due to that the External network "public network" is visible and available for allocation on the projects. That is not the normal setting I guess. And on C-pouta I can see only my projets network. Do you have any tips for the SNIC cloud admins on what they should change to get the "default" or the same behavior as on c-pouta/regular cloud.

tourunen commented 7 years ago

On cPouta we have seem to have 1500 MTU by default.

On the public network topic: I'm not an expert, but I think OpenStack can be deployed in many different ways. I'd guess having an extra virtual nic that has an external IP instead of a forwarding and natting traffic via a project specific router is better for many cases (there are some extra steps needed to extract the actual IP from the instance metadata for configuring IP sensitive services). I see this as a missing feature from PB.