search

rcbops / chef-cookbooks

RCB OPS - Chef Cookbooks
Other
118 stars 102 forks source link

[glance] apache2/wsgi for glance-api not working #599

Closed odyssey4me closed 10 years ago

odyssey4me commented 11 years ago

When setting up glance-api to work using an SSL endpoint it doesn't work. The service starts and you can execute some functions, but not everything works as it should.

I've changed the endpoint scheme back to http and used the apache2/wsgi configuration to fire up glance-api as an http endpoint and see the following.

When executing glance image-download --file cirros-downloaded.img cirros-image the service comes back with an HTTP 500 error.

The apache debug log shows the following:

[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63] mod_wsgi (pid=17375): Exception occurred processing WSGI script '/etc/apache2/wsgi/glance-api'.
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63] Traceback (most recent call last):
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]     resp = self.call_func(req, *args, **self.kwargs)
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]     return self.func(req, *args, **kwargs)
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 362, in __call__
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]     response = req.get_response(self.application)
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]     application, catch_exc_info=False)
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]     app_iter = application(self.environ, start_response)
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 450, in __call__
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]     return self.app(env, start_response)
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]     resp = self.call_func(req, *args, **self.kwargs)
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]     return self.func(req, *args, **kwargs)
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 362, in __call__
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]     response = req.get_response(self.application)
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]     application, catch_exc_info=False)
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]     app_iter = application(self.environ, start_response)
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 203, in __call__
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]     return app(environ, start_response)
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]     return resp(environ, start_response)
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]     response = self.app(environ, start_response)
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]     return resp(environ, start_response)
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]     resp = self.call_func(req, *args, **self.kwargs)
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]     return self.func(req, *args, **kwargs)
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 569, in __call__
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]     request, **action_args)
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 586, in dispatch
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]     return method(*args, **kwargs)
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/glance/api/v1/images.py", line 323, in show
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]     image_meta['location'])
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/glance/api/v1/images.py", line 299, in _get_from_store
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]     image_data, image_size = get_from_backend(context, where)
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/glance/store/__init__.py", line 224, in get_from_backend
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]     loc = location.get_location_from_uri(uri)
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/glance/store/location.py", line 75, in get_location_from_uri
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63]     raise exception.UnknownScheme(pieces.scheme)
[Tue Oct 08 11:13:57 2013] [error] [client 192.168.4.63] UnknownScheme: http

Adding a few debug log lines into /usr/lib/python2.7/dist-packages/glance/store/location.py reveals the following:

[Tue Oct 08 11:13:57 2013] [error] 2013-10-08 11:13:57.650 INFO glance.store.location [93f95791-559f-4ca3-aea9-0a5e3712ddb4 de66acb66baf4a04bd5cbbd7859f3f2d 1500a7287752474798c5752569de5363] URI: http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img
[Tue Oct 08 11:13:57 2013] [error] 2013-10-08 11:13:57.652 INFO glance.store.location [93f95791-559f-4ca3-aea9-0a5e3712ddb4 de66acb66baf4a04bd5cbbd7859f3f2d 1500a7287752474798c5752569de5363] pieces: ParseResult(scheme=u'http', netloc=u'download.cirros-cloud.net', path=u'/0.3.1/cirros-0.3.1-x86_64-disk.img', params='', query='', fragment='')
[Tue Oct 08 11:13:57 2013] [error] 2013-10-08 11:13:57.652 INFO glance.store.location [93f95791-559f-4ca3-aea9-0a5e3712ddb4 de66acb66baf4a04bd5cbbd7859f3f2d 1500a7287752474798c5752569de5363] SCHEME_TO_CLS_MAP.keys: []

So, somehow the glance-api wsgi service is not picking up the default SCHEME_TO_CLS_MAP which comes from the glance.store.location.register_scheme_map function. This function gets it from the glance.store init which sets the defaults.

Why the defaults aren't being picked up, I haven't yet fathomed.

Of course it works perfectly when the service is not configured to run behind apache2.

odyssey4me commented 11 years ago

Alright, figured it out.

In /etc/glance/glance-api.conf there is a conf entry which may be flavor = keystone+cachemanagement. Whatever the value is, the value of name = "glance-api-keystone+cachemanagement" in the wsgi configuration needs to be 'glance-api' + <value of flavor>

I'll work out the recipe bits and do a PR shortly.

odyssey4me commented 11 years ago

Hmm, not out of the woods yet - the http scheme worked, but now others aren't. It might be specific to my setup as I've been breaking this over and over again. If anyone else can test to verify I'd appreciate it.

odyssey4me commented 11 years ago

Problem: When executing a glance image-upload I received the error (9, 'Bad file descriptor') Solution: https://bugs.launchpad.net/python-glanceclient/+bug/1179392 resolved by https://review.openstack.org/33884

Problem: When executing a glance image-upload I received the error

Request returned failure status.
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>411 Length Required</title>
</head><body>
<h1>Length Required</h1>
<p>A request of the requested method POST requires a valid Content-length.<br />
</p>
</body></html>
 (HTTP N/A)

Solution: As described in http://stackoverflow.com/questions/17255556/http-headers-for-chunked-encoding-post-error-411 the solution is to set WSGIChunkedRequest On in the Apache vhost.

odyssey4me commented 11 years ago

With the above three solutions implemented, I see that I have a similar problem to the original when trying to upload a new image:

[Wed Oct 09 08:43:14 2013] [error] [client 192.168.4.63] mod_wsgi (pid=26069): Exception occurred processing WSGI script '/etc/apache2/wsgi/glance-api'.
[Wed Oct 09 08:43:14 2013] [error] [client 192.168.4.63] Traceback (most recent call last):
[Wed Oct 09 08:43:14 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
[Wed Oct 09 08:43:14 2013] [error] [client 192.168.4.63]     resp = self.call_func(req, *args, **self.kwargs)
[Wed Oct 09 08:43:14 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
[Wed Oct 09 08:43:14 2013] [error] [client 192.168.4.63]     return self.func(req, *args, **kwargs)
[Wed Oct 09 08:43:14 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 362, in __call__
[Wed Oct 09 08:43:14 2013] [error] [client 192.168.4.63]     response = req.get_response(self.application)
[Wed Oct 09 08:43:14 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
[Wed Oct 09 08:43:14 2013] [error] [client 192.168.4.63]     application, catch_exc_info=False)
[Wed Oct 09 08:43:14 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
[Wed Oct 09 08:43:14 2013] [error] [client 192.168.4.63]     app_iter = application(self.environ, start_response)
[Wed Oct 09 08:43:14 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 450, in __call__
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     return self.app(env, start_response)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     resp = self.call_func(req, *args, **self.kwargs)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     return self.func(req, *args, **kwargs)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 362, in __call__
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     response = req.get_response(self.application)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     application, catch_exc_info=False)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     app_iter = application(self.environ, start_response)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     resp = self.call_func(req, *args, **self.kwargs)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     return self.func(req, *args, **kwargs)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 362, in __call__
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     response = req.get_response(self.application)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     application, catch_exc_info=False)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     app_iter = application(self.environ, start_response)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     resp = self.call_func(req, *args, **self.kwargs)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     return self.func(req, *args, **kwargs)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 362, in __call__
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     response = req.get_response(self.application)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     application, catch_exc_info=False)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     app_iter = application(self.environ, start_response)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 203, in __call__
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     return app(environ, start_response)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     return resp(environ, start_response)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     response = self.app(environ, start_response)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     return resp(environ, start_response)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     resp = self.call_func(req, *args, **self.kwargs)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     return self.func(req, *args, **kwargs)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 569, in __call__
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     request, **action_args)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 586, in dispatch
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     return method(*args, **kwargs)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/glance/common/utils.py", line 413, in wrapped
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     return func(self, req, *args, **kwargs)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/glance/api/v1/images.py", line 687, in create
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     image_meta = self._reserve(req, image_meta)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/glance/api/v1/images.py", line 352, in _reserve
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     store = get_store_from_location(location)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/glance/store/__init__.py", line 261, in get_store_from_location
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     loc = location.get_location_from_uri(uri)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]   File "/usr/lib/python2.7/dist-packages/glance/store/location.py", line 75, in get_location_from_uri
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63]     raise exception.UnknownScheme(pieces.scheme)
[Wed Oct 09 08:43:15 2013] [error] [client 192.168.4.63] UnknownScheme: http
odyssey4me commented 11 years ago

Interesting - I don't think that the original solution was actually complete anyway. All it did was allow glance to use the cache which already had the images in it, which is why the download worked.

So it would appear that the original issue still stands - the schemes are not all being read in for some reason.

I suspect the problem may be due to the inclusion of /usr/lib/python2.7/dist-packages/glance/common/wsgi.py in the openstack code.

odyssey4me commented 11 years ago

By the way, I've given up entirely on the apache/wsgi approach. I just don't feel that there's enough testing in place (for the recipes and for the openstack code) to support using this approach in production. There just seems to be one snag after another.

I have opted to rather make use of the reverse proxy for exposing the API's via SSL. This keeps the 'internal' access native and only presents possible problems with access for external clients (workstations, automation tooling, etc). This way we keep our system running and only have to deal with oddities (like non RFC-compliant headers) when it's hit by an end-user trying to do something special.

claco commented 10 years ago

Given that you taken another approach, I'm going to to close this issue and the associated PR that never really gated.

sourcepotatoe commented 10 years ago

I got passed the "SCHEME_TO_CLS_MAP" problem by adding the following lines to the glance-api.wsgi file:

import glance.store glance.store.create_stores() glance.store.verify_default_store()

I found these in the /usr/bin/glance startup script.

Now I am stuck trying to upload and image file through Horizon:

[Mon Dec 16 14:28:28 2013] [error] [client 10.1.184.1] Requested content-length of 9223372036854775807 is larger than the configured limit of 102400 [Mon Dec 16 14:28:28 2013] [error] [client 10.1.184.1] Premature end of script headers: glance-api.py 10.1.184.1 - - [16/Dec/2013:14:28:24 -0600] "POST /v1/images HTTP/1.1" 500 600 "-" "python-glanceclient" I was able to print out the wsgi environment just before the error:

Has anyone come across and conquered this one?

Mark

odyssey4me commented 10 years ago

Nicely done - @andymcc take note the above amendments to the required wsgi config.

@sourcepotatoe perhaps this helps to resolve your new issue? http://httpd.apache.org/docs/2.2/mod/core.html#limitrequestbody

sourcepotatoe commented 10 years ago

I'm not sure if my "fix" works in every situation, but debugging this issue I found that an error was being thrown for the very last chunk read. Instead of setting the maximum size of the chunk to read, I specified the exact size for the last chunk and the error went away. I am open to a better solution, but here is my "fix" below for the "add" method in file /usr/lib/python2.7/dist-packages/glance/store/filesystem.py. Please not that I am using SSL for glance.

    bytes_written = 0
    bytes_to_read = ChunkedFile.CHUNKSIZE
    try:
        with open(filepath, 'wb') as f:

            while bytes_written < image_size:
                if (image_size - bytes_written) < ChunkedFile.CHUNKSIZE:
                    bytes_to_read = image_size - bytes_written
                buf = image_file.read(bytes_to_read)
                bytes_written += len(buf)
                checksum.update(buf)
                f.write(buf)

            """
            for buf in utils.chunkreadable(image_file,
                                           ChunkedFile.CHUNKSIZE):
                bytes_written += len(buf)
                checksum.update(buf)
                f.write(buf)
            """
odyssey4me commented 10 years ago

@sourcepotatoe In my barrage of email catch-up yesterday I recall seeing something about this. I think this was picked up and a patch submitted, so a fix will most likely come down the line eventually. I can't seem to find the email for reference though. :/

sourcepotatoe commented 10 years ago

I would greatly appreciate the email reference for this problem. Someone more intimate with the code probably had a better well-rounded solution.

Mark

From: odyssey4me [mailto:notifications@github.com] Sent: Tuesday, January 07, 2014 12:26 AM To: rcbops/chef-cookbooks Cc: Miller, Mark M (EB SW Cloud - R&D - Corvallis) Subject: Re: [chef-cookbooks] [glance] apache2/wsgi for glance-api not working (#599)

@sourcepotatoehttps://github.com/sourcepotatoe In my barrage of email catch-up yesterday I recall seeing something about this. I think this was picked up and a patch submitted, so a fix will most likely come down the line eventually. I can't seem to find the email for reference though. :/

— Reply to this email directly or view it on GitHubhttps://github.com/rcbops/chef-cookbooks/issues/599#issuecomment-31720350.

sourcepotatoe commented 10 years ago

I would greatly appreciate finding a more official patch for this problem than what I wrote.

sourcepotatoe commented 10 years ago

I have already had to change the code to get the unit tests to pass:

            while bytes_written < image_size:
                if (image_size - bytes_written) < ChunkedFile.CHUNKSIZE:
                    bytes_to_read = image_size - bytes_written
                buf = image_file.read(bytes_to_read)
                if not buf:
                    break
                bytes_written += len(buf)
                checksum.update(buf)
                f.write(buf)
odyssey4me commented 10 years ago

@sourcepotatoe It appears that it wasn't a registered bug as far as I can see, just something notified on the mailing list: http://lists.openstack.org/pipermail/openstack-dev/2013-December/022510.html

sourcepotatoe commented 10 years ago

One of the sharp engineers I work with found a solution to a SSL Glance -> Nova image transfer problem we ran into.

https://github.com/openstack/python-glanceclient/commit/5d90740f33d80db5559becc0cc619d965a4bb292

sourcepotatoe commented 10 years ago

Hello,

I have a few questions concerning enabling SSL with nova-novncproxy. I found the following link/patch submittal: https://github.com/rcbops-cookbooks/nova/pull/355 , but I am not sure if it applies to the Grizzly release of Nova. Does it? If so, where can I find sample configuration files that implement this patch? Can we use Apache2 with WSGI as a frontend to nova-novncproxy?

Regards,

Mark

claco commented 10 years ago

@sourcepotatoe, that changes was merged into the grizzly stable branch and released in v4.1.3. Not sure on the other part of the question.

sourcepotatoe commented 10 years ago

Thank you for your answer. I would assume that we can get SSL enabled with nova-novncproxy if we can just figure out how.

claco commented 10 years ago

Well, sortof. Some form of that PR went in in this commit: https://github.com/rcbops-cookbooks/nova/commit/c6b631d7e5b374b90152704124cabc5d5d8bc269

sourcepotatoe commented 10 years ago

Yes, I saw the commit. I am trying to convert the information in the commit to actual file changes so I can test it and have an example of what the configuration files should look like after crowbar is done with them. I tried the following changes to nova.conf but I am not seeing an error in the log files and it is not working.

Nova.conf:

VNCPROXY

novncproxy_host=10.1.184.2 novncproxy_port=6080 novncproxy_base_url=https://10.1.184.2:6080/vnc_auto.html vncserver_listen=0.0.0.0 vncserver_proxyclient_address=192.168.124.81 vnc_enabled=True

novncproxy_cert=/etc/ssl/certs/d00-50-56-8e-79-e7.cloud.org.crt novncproxy_key =/etc/ssl/openstack/private/server.key

From: Christopher H. Laco [mailto:notifications@github.com] Sent: Tuesday, January 14, 2014 4:45 PM To: rcbops/chef-cookbooks Cc: Miller, Mark M (EB SW Cloud - R&D - Corvallis) Subject: Re: [chef-cookbooks] [glance] apache2/wsgi for glance-api not working (#599)

Well, sortof. Some form of that PR went in in this commit: rcbops-cookbooks/nova@c6b631dhttps://github.com/rcbops-cookbooks/nova/commit/c6b631d7e5b374b90152704124cabc5d5d8bc269

— Reply to this email directly or view it on GitHubhttps://github.com/rcbops/chef-cookbooks/issues/599#issuecomment-32324269.

sourcepotatoe commented 10 years ago

Ok, I figured out what the nova.conf values should be and got it working, although not with an Apache2 WSGI front end. Here is an excerpt from my nova.conf file:

VNCPROXY

novncproxy_host=0.0.0.0 novncproxy_port=6080 novncproxy_base_url=https://10.1.184.2:6080/vnc_auto.html vncserver_listen=0.0.0.0 vncserver_proxyclient_address=192.168.124.81 vnc_enabled=True

cert=/etc/ssl/certs/d00-50-56-8e-79-e7.cloud.org.crt key=/etc/ssl/openstack/private/server.key

odyssey4me commented 10 years ago

To enable SSL for NoVNC you'll need to:

  1. Set the ["nova"]["services"]["novnc-proxy"]["scheme"] attribute in your environment for novncproxy to https
  2. Override the certificate file as novnc.pem and the key file as novnc.key in your nova cookbook files with your desired certificate and key files
  3. Do a chef-client run on your controllers and compute nodes to apply the changed settings

NoVNC's web server cannot be front-ended by Apache - I've tried a few methods without success. NoVNC's web server will not listen on ports 80 and 443 - I've tried a few methods without success. NoVNC's web service uses web sockets, which is rather troublesome when you're trying to work through proxies of any sort as they often don't support the method by which web sockets switches protocols.

shilpa12345 commented 7 years ago

Getting a error while executing command

openstack image create "cirros" --file cirros-0.3.5-x86_64-disk.img --disk-format qcow2 --container-format bare --public

500 Internal Server Error The server has either erred or is incapable of performing the requested operation. (HTTP 500)

shilpa12345 commented 7 years ago

tail /var/log/glance/api.log OUTPUT: 2017-05-16 20:12:15.036 6551 ERROR glance.common.wsgi first_packet = self.connection._read_packet() 2017-05-16 20:12:15.036 6551 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 991, in _read_packet 2017-05-16 20:12:15.036 6551 ERROR glance.common.wsgi packet.check_error() 2017-05-16 20:12:15.036 6551 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error 2017-05-16 20:12:15.036 6551 ERROR glance.common.wsgi err.raise_mysql_exception(self._data) 2017-05-16 20:12:15.036 6551 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception 2017-05-16 20:12:15.036 6551 ERROR glance.common.wsgi raise errorclass(errno, errval) 2017-05-16 20:12:15.036 6551 ERROR glance.common.wsgi ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'glance.images' doesn't exist") [SQL: u'INSERT INTO images (created_at, updated_at, deleted_at, deleted, id, name, disk_format, container_format, size, virtual_size, status, visibility, checksum, min_disk, min_ram, owner, protected) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(id)s, %(name)s, %(disk_format)s, %(container_format)s, %(size)s, %(virtual_size)s, %(status)s, %(visibility)s, %(checksum)s, %(min_disk)s, %(min_ram)s, %(owner)s, %(protected)s)'] [parameters: {'status': 'queued', 'deleted_at': None, 'name': u'cirros', 'deleted': 0, 'checksum': None, 'created_at': datetime.datetime(2017, 5, 16, 14, 42, 15, 24748), 'disk_format': u'qcow2', 'updated_at': datetime.datetime(2017, 5, 16, 14, 42, 15, 24748), 'visibility': u'public', 'owner': u'4c077f02359448b9a5b70950c05e0135', 'protected': 0, 'min_ram': 0, 'container_format': u'bare', 'min_disk': 0, 'virtual_size': None, 'id': '6559abfd-c469-4ce4-bd95-c55926a12f1f', 'size': None}] 2017-05-16 20:12:15.036 6551 ERROR glance.common.wsgi 2017-05-16 20:12:15.044 6551 INFO eventlet.wsgi.server [req-812779f8-d5fa-45a4-97f8-f10f8b903c2b - - - - -] 172.18.117.17 - - [16/May/2017 20:12:15] "POST /v2/images HTTP/1.1" 500 341 0.057280

JCallicoat commented 7 years ago

@shilpa12345 This project is no longer maintained. The currently maintained deployment system lives at https://github.com/openstack/openstack-ansible