search

mistio / mist-ce

Mist is an open source, multicloud management platform
https://mist.io
Apache License 2.0
1.84k stars 310 forks source link

Vsphere machine console does not open in mist io #971

Closed hassanbsee2071 closed 4 years ago

hassanbsee2071 commented 4 years ago

I have installed mist io 4.2.1 on centos 7. Vspere 6 successfully added as cloud. But when I open console of windows machine in mist io, I got internal server error. Mentioned below are logs.

| Traceback (most recent call last): api_1 | File "/usr/local/lib/python3.7/site-packages/pyramid/tweens.py", line 21, in excview_tween api_1 | response = handler(request) api_1 | File "/usr/local/lib/python3.7/site-packages/pyramid/router.py", line 163, in handle_request api_1 | response = view_callable(context, request) api_1 | File "/usr/local/lib/python3.7/site-packages/pyramid/config/views.py", line 329, in attr_view api_1 | return view(context, request) api_1 | File "/usr/local/lib/python3.7/site-packages/pyramid/config/views.py", line 305, in predicate_wrapper api_1 | return view(context, request) api_1 | File "/mist.api/src/mist/api/helpers.py", line 858, in logging_view api_1 | response = func(context, request) api_1 | File "/usr/local/lib/python3.7/site-packages/pyramid/config/views.py", line 355, in rendered_view api_1 | result = view(context, request) api_1 | File "/usr/local/lib/python3.7/site-packages/pyramid/config/views.py", line 501, in _requestonly_view api_1 | response = view(request) api_1 | File "/mist.api/src/mist/api/machines/views.py", line 973, in machine_console api_1 | machine.machine_id api_1 | File "/mist.api/libcloud/libcloud/compute/drivers/vsphere.py", line 970, in ex_open_console api_1 | vc_cert = ssl.get_server_certificate((vcenter_fqdn, 443)) api_1 | File "/usr/local/lib/python3.7/ssl.py", line 1313, in get_server_certificate api_1 | with create_connection(addr) as sock: api_1 | File "/usr/local/lib/python3.7/socket.py", line 727, in create_connection api_1 | raise err api_1 | File "/usr/local/lib/python3.7/socket.py", line 716, in create_connection api_1 | sock.connect(sa) api_1 | TimeoutError: [Errno 110] Operation timed out

d-mo commented 4 years ago

Hi @hassanbsee2071.

It looks like the ssl library is timing out when trying to fetch your vcenter server certificate. Can you confirm that vcenter is listening for https connections on port 443? Does it have a valid TLS certificate?

You should be able to replicate the issue (or see the certificate) with the following line of Python code outside of Mist. e.g. python -c 'import ssl;print(ssl.get_server_certificate(("mist.io", 443)))'

Change "mist.io" with the dns name of your vcenter host.

hassanbsee2071 commented 4 years ago

Hy Dimitris,

Yes Vsphere is listening on 443 but I am using self signed certificate. Is there any option to disable ssl check. I set SSL_VERIFY=False in settings.py but still not resolved.

hassanbsee2071 commented 4 years ago

Moreover mentioned below is output of command that you have mentioned.

/mist.api # python -c 'import ssl;print(ssl.get_server_certificate(("vcenter03.systemsltd.local", 443)))' -----BEGIN CERTIFICATE----- MIIDozCCAougAwIBAgIJAOt8uP/FpCnVMA0GCSqGSIb3DQEBCwUAMIGEMQswCQYD VQQDDAJDQTEXMBUGCgmSJomT8ixkARkWB3ZzcGhlcmUxFTATBgoJkiaJk/IsZAEZ FgVsb2NhbDELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExEjAQBgNV BAoMCXZDZW50ZXIwMzEPMA0GA1UECwwGVk13YXJlMB4XDTE3MTEyMDE4MDE0NloX DTI3MTExNTE4MDE0NFowMjEjMCEGA1UEAwwadkNlbnRlcjAzLlN5c3RlbXNsdGQu bG9jYWwxCzAJBgNVBAYTAlVTMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC AQEAtHE4z8ESsUvEueSECkUWcKuYlNkB1Xc5c2uQHRHqXXaDEPy09L2FIx4B0yp0 EQ/lOk2z0GxljJtQMbc9VsUe518c/gn3Inmowq4uuFFhmlSIX3j1xoJbIU5d+z4S 0vde0Aa2OoaaohhF8auEI2VcaiodUt8EWJ/3r9ZQYgALxDFSg7dmEHuppYYZOPL+ Gqe5SI4GYPLZIfQuxKZiR8/Oufsn3DOnF+XdLb73GGvHifjc1yhIWeio5hAHkSg5 ows4Lt/ULQ110Xwd29YX14tdJGuQisBSxOj8nz2y0Cb/5sL/YOtDaalBruGSHyo7 AucGYa9c9/ufTfDWMaAazYPcSQIDAQABo2kwZzAlBgNVHREEHjAcghp2Q2VudGVy MDMuU3lzdGVtc2x0ZC5sb2NhbDAdBgNVHQ4EFgQUqHOhmTn/n1Ispt9NHBRYemNs a3YwHwYDVR0jBBgwFoAUbPnjGtMLElh2oJvCtyX2rUabr5MwDQYJKoZIhvcNAQEL BQADggEBAHM8l7aoSmdJ0iRqSBc+tFr4ngd/2Jd2E/zCcARbT/o9XHtI9GAY+BaO E0blVue+p1Hcb1Pcb/J/bRZlBRStqi50UitDBePz9AJoHbaiXSEVPgcwgq/d/Wqu oV8JfVRZLRfsZPKPPomKMAUhMcFQyz+sjUZWRPMzEEWsVcbFQ+fgLmr2LeefWcEx JpQ1OyNV4D3eSuqNq2JGrzEB0DCG8dWn2BtJvnKEpyRKLf/T3ilZ8CyOtJccQ3hp hjvpehPa4Mzt0QSCB//1EUOA0/TjAA2QKzL8NZkfat4rntHPXvrbvbMRk9Pdur4g COPy9UvqzTqmWP7Gh3rZn5jxLxWAEpo= -----END CERTIFICATE-----

hassanbsee2071 commented 4 years ago

Dear Dimitris,

When I changed below code of file /mist.api/libcloud/libcloud/compute/drivers/vsphere.py", line 970 from: def ex_open_console(self, vm_uuid, console_port = '9443'): import OpenSSL import ssl content = self.connection.RetrieveContent() server_guid = content.about.instanceUuid search_index = content.searchIndex vm = search_index.FindByUuid(None, vm_uuid, True, True) vcenter_data = content.setting vm_moid = vm._moId vcenter_settings = vcenter_data.setting vcenter_fqdn = "self.host"

    for item in vcenter_settings:
        key = getattr(item, 'key')
        if key == 'VirtualCenter.FQDN':
            vcenter_fqdn = "getattr(item, 'value')"

to:

def ex_open_console(self, vm_uuid, console_port = '9443'):
    import OpenSSL
    import ssl
    content = self.connection.RetrieveContent()
    server_guid = content.about.instanceUuid
    search_index = content.searchIndex
    vm = search_index.FindByUuid(None, vm_uuid, True, True)
    vcenter_data = content.setting
    vm_moid = vm._moId
    vcenter_settings = vcenter_data.setting
   ###  _**vcenter_fqdn = "vcenter03.systemsltd.local"**_

    for item in vcenter_settings:
        key = getattr(item, 'key')
        if key == 'VirtualCenter.FQDN':
        ###     **_vcenter_fqdn = "vcenter03.systemsltd.local"_**

console of machine gets working. Now I have two question:

1) If other vspehere is added then i do not think above code will work because I have hard coded the hostname. Now how should I cater this problem.

2) When I open mist in private window and then open console of machine, this will redirect to vcenter URL which asks for username and password. Is this normal behavior?

d-mo commented 4 years ago

Hi @hassanbsee2071

  1. I'm not sure I understand the issue. The original code should get the correct vcenter fqdn without any changes. Did you use that hostname when adding the cloud or did you put an esxi hostname instead? That could explain the issue. I haven't been able to replicate it in any other way.

  2. This is a known issue that we're working on. There will probably be a fix in the next release.

If you'd like to debug this further maybe we can do it over a call. I'd love to know more about your use case and see if we can help you out in any way.

hassanbsee2071 commented 4 years ago

Hi @d-mo Yes I have used the same name while adding in the cloud. I am also staggered with this kind of behavior. May be I am using self-signed certificate. I will give it a try with another host.

It would be great if we can sort this out over a call. Yes sure I would also love to discuss my use case with you. syedhassan39 is my skype ID and +92-333-5705109 is my whatsapp number.

I am also facing another strange issue. When I implement SSL on mist io URL then monitoring feature does not work. I have debugged via mozilla network element I came to know the error that is internal server error. Image is also attached. I tried this with both self signed and trusted certificate but result was same. When SSL configuration was removed and then monitoring with telegraf works perfectly. It does not work with SSL. May be there is issue with influxdb. I also tried to capture the logs but the logs are same in both cases.

api_1 | api_1 | 2020-02-22 06:24:22,548 INFO [mist.api.helpers][uWSGIWorker8Core0] Bad exception occured, logging to rabbitmq api_1 | 2020-02-22 06:24:22,551 INFO [mist.api.helpers][uWSGIWorker8Core0] Bad exception occured, logging to file api_1 | [pid: 90|app: 0|req: 145/632] 172.18.0.9 () {36 vars in 840 bytes} [Sat Feb 22 06:24:22 2020] GET /api/v1/jobs/1cc2f69a2a714d97adf1becaf6046c6f => generated 21 bytes in 37 msecs (HTTP/1.0 500) 2 headers in 98 bytes (1 switches on core 0) api_1 | 2020-02-22 06:24:23,551 CRITI [mist.api.views][uWSGIWorker1Core0] Uncaught non-mist exception? WTF! api_1 | Traceback (most recent call last): api_1 | File "/usr/local/lib/python3.7/site-packages/pyramid/tweens.py", line 21, in excview_tween api_1 | response = handler(request) api_1 | File "/usr/local/lib/python3.7/site-packages/pyramid/router.py", line 163, in handle_request api_1 | response = view_callable(context, request) api_1 | File "/usr/local/lib/python3.7/site-packages/pyramid/config/views.py", line 596, in call api_1 | return view(context, request) api_1 | File "/usr/local/lib/python3.7/site-packages/pyramid/config/views.py", line 329, in attr_view api_1 | return view(context, request) api_1 | File "/usr/local/lib/python3.7/site-packages/pyramid/config/views.py", line 305, in predicate_wrapper api_1 | return view(context, request) api_1 | File "/mist.api/src/mist/api/helpers.py", line 858, in logging_view api_1 | response = func(context, request) api_1 | File "/usr/local/lib/python3.7/site-packages/pyramid/config/views.py", line 355, in rendered_view api_1 | result = view(context, request) api_1 | File "/usr/local/lib/python3.7/site-packages/pyramid/config/views.py", line 501, in _requestonly_view api_1 | response = view(request) api_1 | File "/mist.api/src/mist/api/logs/views.py", line 158, in show_job api_1 | return get_story(auth_context.owner.id, job_id) api_1 | File "/mist.api/src/mist/api/logs/methods.py", line 733, in get_story api_1 | story_type=story_type, expand=expand) api_1 | File "/mist.api/src/mist/api/logs/methods.py", line 521, in get_stories api_1 | return _on_request_callback(query) api_1 | File "/mist.api/src/mist/api/logs/methods.py", line 496, in _on_request_callback api_1 | body=query) api_1 | File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 76, in _wrapped api_1 | return func(*args, params=params, **kwargs) api_1 | File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/init.py", line 660, in search api_1 | doc_type, '_search'), params=params, body=body) api_1 | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 318, in perform_request api_1 | status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout) api_1 | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 186, in perform_request api_1 | self._raise_error(response.status, raw_data) api_1 | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/base.py", line 125, in _raise_error api_1 | raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info) api_1 | elasticsearch.exceptions.RequestError: RequestError(400, 'search_phase_execution_exception', 'Fielddata is disabled on text fields by default. Set fielddata=true on [stories] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.') api_1 | api_1 | 2020-02-22 06:24:23,559 INFO [mist.api.helpers][uWSGIWorker1Core0] Bad exception occured, logging to rabbitmq api_1 | 2020-02-22 06:24:23,561 INFO [mist.api.helpers][uWSGIWorker1Core0] Bad exception occured, logging to file api_1 | [pid: 75|app: 0|req: 248/633] 172.18.0.9 () {36 vars in 840 bytes} [Sat Feb 22 06:24:23 2020] GET /api/v1/jobs/1cc2f69a2a714d97adf1becaf6046c6f => generated 21 bytes in 33 msecs (HTTP/1.0 500) 2 headers in 98 bytes (1 switches on core 0) api_1 | 2020-02-22 06:24:24,547 CRITI [mist.api.views][uWSGIWorker8Core0] Uncaught non-mist exception? WTF! api_1 | Traceback (most recent call last): api_1 | File "/usr/local/lib/python3.7/site-packages/pyramid/tweens.py", line 21, in excview_tween api_1 | response = handler(request)

When ssl is removed and when ssl is not removed in both cases logs are same. Only difference is that after removal of SSL monitoring works perfectly.

internal server error

hassanbsee2071 commented 4 years ago

On the agent I found below mentioned logs:

Feb 23 05:25:03 WIN-VRVF376500E telegraf: 2020-02-23T10:25:03Z E! [outputs.influxdb]: when writing to [https://cmp.nayatel.com/ingress/a7ea91736edcdaf59311e0a80944ae624bf7dd13d8bcd0a6b7bd1b432f7881f6]: 404 Not Found Feb 23 05:25:03 WIN-VRVF376500E telegraf: 2020-02-23T10:25:03Z E! Error writing to output [influxdb]: could not write any address Feb 23 05:25:08 WIN-VRVF376500E telegraf: 2020-02-23T10:25:08Z E! [outputs.influxdb]: when writing to [https://cmp.nayatel.com/ingress/a7ea91736edcdaf59311e0a80944ae624bf7dd13d8bcd0a6b7bd1b432f7881f6]: 404 Not Found Feb 23 05:25:08 WIN-VRVF376500E telegraf: 2020-02-23T10:25:08Z E! Error writing to output [influxdb]: could not write any address Feb 23 05:25:13 WIN-VRVF376500E telegraf: 2020-02-23T10:25:13Z E! [outputs.influxdb]: when writing to [https://cmp.nayatel.com/ingress/a7ea91736edcdaf59311e0a80944ae624bf7dd13d8bcd0a6b7bd1b432f7881f6]: 404 Not Found

hassanbsee2071 commented 4 years ago

Hi @d-mo ;

Issue has been resolved, I have done following three things:

1) Changed docker-compose.yml to add entry in host file of each docker. Complete file is given below.

2) Change vcenter_fqdn = "self.host" to vcenter_fqdn = "vcenter03.systemsltd.local" and vcenter_fqdn = "getattr(item, 'value')" to vcenter_fqdn = "vcenter03.systemsltd.local" as mentioned above.

3) Change (insecure_skip_verify = false) to (insecure_skip_verify = true) of configuration file present in agent at vi /opt/mistio/mist-telegraf/telegraf.conf

Then disable and enable again monitoring from GUI. After that monitoring works perfectly. I am using self signed certificate but I have also tested on trusted certificate. Trusted certificates are also working.

Docker compose definition to run mist.io in production mode, so no mounted

code, no dev containers etc. Only this single file is required.

version: '2.0'

services:

mongodb: image: mongo:3.2 restart: on-failure:5 volumes:

volumes: elasticsearch: {} influxdb: {} mongodb: {}

cpsaltis commented 4 years ago

I'm glad that you got to the bottom of this @hassanbsee2071.

I'd like to learn a bit more about your use case. How about we sync on Skype? I just pinged you there.