Open discostur opened 8 years ago
bahamat
commented
8 years ago It looks like your Moray zone(s) may be unhealthy. Use the following commands to start diagnosing why.
moray=$(vmadm lookup tags.smartdc_role=moray)
vmadm get $moray
svcs -z $moray -xv
tail -50 $(svcs -L moray)
discostur
commented
8 years ago The moray zone looks ok:
vmadm get $moary:
{
"zonename": "7d7c4fd3-c3c8-4ad1-80e3-29fb06f48bef",
"autoboot": true,
"brand": "joyent-minimal",
"limit_priv": "default",
"v": 1,
"create_timestamp": "2016-04-01T13:26:22.559Z",
"cpu_shares": 8192,
"max_lwps": 1000,
"max_msg_ids": 4096,
"max_sem_ids": 4096,
"max_shm_ids": 4096,
"max_shm_memory": 8192,
"zfs_io_priority": 20,
"max_physical_memory": 8192,
"max_locked_memory": 8192,
"max_swap": 16384,
"cpu_cap": 400,
"billing_id": "16425b42-b818-11e2-90d4-87477f18a688",
"owner_uuid": "930896af-bf8c-48d4-885c-6573a94b1853",
"package_name": "sdc_8192",
"package_version": "1.0.0",
"tmpfs": 8192,
"dns_domain": "local",
"archive_on_delete": true,
"maintain_resolvers": true,
"resolvers": [
"10.99.99.11"
],
"alias": "moray0",
"nics": [
{
"interface": "net0",
"mac": "82:cb:5b:1c:bd:4b",
"vlan_id": 0,
"nic_tag": "admin",
"netmask": "255.255.255.0",
"ip": "10.99.99.17",
"ips": [
"10.99.99.17/24"
],
"primary": true
}
],
"datasets": [
"zones/7d7c4fd3-c3c8-4ad1-80e3-29fb06f48bef/data"
],
"image_uuid": "3c3d2d86-f7cb-11e5-b32d-d768ace12ffa",
"uuid": "7d7c4fd3-c3c8-4ad1-80e3-29fb06f48bef",
"zone_state": "running",
"zonepath": "/zones/7d7c4fd3-c3c8-4ad1-80e3-29fb06f48bef",
"zoneid": 2,
"last_modified": "2016-04-02T15:23:28.000Z",
"firewall_enabled": false,
"server_uuid": "564d9ee7-4a17-cd23-73b6-cb15390be0f2",
"datacenter_name": "coal-1",
"platform_buildstamp": "20160330T234300Z",
"state": "running",
"boot_timestamp": "2016-04-02T15:24:27.000Z",
"pid": 4184,
"customer_metadata": {
"sapi-url": "http://10.99.99.32",
"user-script": "#!/usr/bin/bash\n#\n# This Source Code Form is subject to the terms of the Mozilla Public\n# License, v. 2.0. If a copy of the MPL was not distributed with this\n# file, You can obtain one at http://mozilla.org/MPL/2.0/.\n#\n\n#\n# Copyright (c) 2014, Joyent, Inc.\n#\n\nexport PS4='[\\D{%FT%TZ}] ${BASH_SOURCE}:${LINENO}: ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'\n\nset -o xtrace\nset -o errexit\nset -o pipefail\n\n#\n# The presence of the /var/svc/.ran-user-script file indicates that the\n# instance has already been setup (i.e. the instance has booted previously).\n#\n# Upon first boot, run the setup.sh script if present. On all boots including\n# the first one, run the configure.sh script if present.\n#\n\nSENTINEL=/var/svc/.ran-user-script\n\nDIR=/opt/smartdc/boot\n\nif [[ ! -e ${SENTINEL} ]]; then\n if [[ -f ${DIR}/setup.sh ]]; then\n ${DIR}/setup.sh 2>&1 | tee /var/svc/setup.log\n fi\n\n touch ${SENTINEL}\nfi\n\nif [[ ! -f ${DIR}/configure.sh ]]; then\n echo \"Missing ${DIR}/configure.sh cannot configure.\"\n exit 1\nfi\n\nexec ${DIR}/configure.sh\n",
"assets-ip": "10.99.99.8"
},
"internal_metadata": {},
"routes": {},
"tags": {
"smartdc_role": "moray",
"smartdc_type": "core"
},
"quota": 25,
"zfs_root_recsize": 131072,
"zfs_filesystem": "zones/7d7c4fd3-c3c8-4ad1-80e3-29fb06f48bef",
"zpool": "zones",
"zfs_data_recsize": 131072,
"snapshots": []
}svcs -z $moray -xv
-> nothing, but svcs -z $moray says everything is online
tail -50 $(svcs -L moray)
-> empty; think it is beacause i have 4x moray logfiles:
(coal-1:moray0) ~]# svcs -L moray
/var/svc/log/smartdc-application-moray:moray-2024.log
/var/svc/log/smartdc-application-moray:moray-2023.log
/var/svc/log/smartdc-application-moray:moray-2022.log
/var/svc/log/smartdc-application-moray:moray-2021.logtail -50 /var/svc/log/smartdc-application-moray:moray-2024.log
[ Apr 2 15:12:28 Enabled. ]
[ Apr 2 15:12:28 Rereading configuration. ]
[ Apr 2 15:12:36 Executing start method ("node main.js -f etc/config.json -p 2024 &"). ]
[ Apr 2 15:12:36 Method "start" exited with status 0. ]
[ Apr 2 15:23:20 Stopping because service disabled. ]
[ Apr 2 15:23:20 Executing stop method (:kill). ]
[ Apr 2 15:24:38 Enabled. ]
[ Apr 2 15:25:18 Executing start method ("node main.js -f etc/config.json -p 2024 &"). ]
[ Apr 2 15:25:18 Method "start" exited with status 0. ]
askfongjojo
commented
8 years ago Hi @discostur, I believe what @bahamat meant was that moray was likely unavailable when AdminUI wasn't working for you. To confirm the issue, you can bring up the developer console (or Firebug if you are on FF) to view the ping API response. You will see that the UI is polling for availability of the dependent services, specifically moray and ufds. When moray is unavailable, you'll see something like this in the http response: {"services":{"moray":false,"ufds":true},"time":"2016-04-01T00:01:14.986Z"}
This should be helpful for your troubleshooting.
bahamat
commented
8 years ago @discostur Based on your feedback, I think you just need to restart the adminui service.
Log into your headnode and run the following commands:
sdc-login adminui
svcadm restart adminui
exit; exit@askfongjojo you were right, just after the reboot i can see the following response:
{"services":{"moray":false,"ufds":true},"time":"2016-04-06T19:49:51.693Z"}
@bahamat after doing an
sdc-login adminui svcadm restart adminui
i am able to login to the web gui
bahamat
commented
8 years ago Filed internally as ADMINUI-2293.
Hi,
after rebooting CoaL i'm not able to login via web-GUI. I'm getting the following error:
"Services required for authentication are currently unavailable. Signin will be re-enabled when service is restored"
It seems to be kind of a timing issue: after reboot adminui checks if authentication service is available - but not all zones are finished booting up. So adminui needs to check again after some time if authentication is available now.
I can fix the problem via "svcadm restart adminui" when i'm logged in to the adminui zone.
Logfile gives me the following:
In Addition, i'm seeing an unhandled expection right after the services has started (not quite shure if it is related to my problem):
Thanks, Kilian