ESGF 3.0 installer failed in setup_cog

[muryanto1]

3) Executing CoGInstall... Initialized CoG settings from file: /usr/local/cog/cog_config/cog_settings.cfg Operations to perform: Apply all migrations: admin, auth, captcha, cog, contenttypes, django_comments, django_openid_auth, sessions, sites Running migrations: Applying contenttypes.0001_initial... OK Applying auth.0001_initial... OK Applying admin.0001_initial... OK Applying admin.0002_logentry_remove_auto_add... OK Applying contenttypes.0002_remove_content_type_name... OK Applying auth.0002_alter_permission_name_max_length... OK Applying auth.0003_alter_user_email_max_length... OK Applying auth.0004_alter_user_username_opts... OK Applying auth.0005_alter_user_last_login_null... OK Applying auth.0006_require_contenttypes_0002... OK Applying auth.0007_alter_validators_add_error_messages... OK Applying auth.0008_alter_user_username_max_length... OK Applying captcha.0001_initial... OK Applying sites.0001_initial... OK Applying cog.0001_initial... OK Applying cog.0002_auto_20150706_1045... OK Applying cog.0003_project_nodeswidgetenabled... OK Applying cog.0004_auto_20160106_0812... OK Applying cog.0005_project_shared... OK Applying cog.0006_auto_20160303_1043... OK Applying cog.0007_auto_20160303_1524... OK Applying cog.0008_auto_20160609_1504... OK Applying cog.0009_auto_20160826_0339... OK Applying django_comments.0001_initial... OK Applying django_comments.0002_update_user_email_field_length... OK Applying django_openid_auth.0001_initial... OK Applying django_openid_auth.0002_auto_20160106_0812... OK Applying sessions.0001_initial... OK INFO:root:Updating site: esgf-dev1.llnl.gov INFO:root:Creating project: TestProject INFO:root:Creating admin user INFO:root:Read ESGF administrator password from file: /esg/config/.esgf_pass INFO:root:Created openid:https://esgf-dev1.llnl.gov/esgf-idp/openid/rootAdmin for CoG administrator: rootAdmin INFO:root:Inserting CoG administrator: rootAdmin in ESGF database Traceback (most recent call last): File "setup.py", line 29, in "distutils.commands": [ "setup_cog = cog.installation.setup:CogSetupCommand"] File "/usr/local/conda/envs/esgf-pub/lib/python2.7/distutils/core.py", line 151, in setup dist.run_commands() File "/usr/local/conda/envs/esgf-pub/lib/python2.7/distutils/dist.py", line 953, in run_commands Applying sites.0002_alter_domain_unique... OK Updated current site: name=ESGF-DEV1.LLNL.GOV domain=esgf-dev1.llnl.gov WARNING: File /esg/config/esgf_cogs.xml does not exist, skipping update of ESGF peer nodes Initializing project: TestProject Created group: testproject_users Created group: testproject_contributors Created group: testproject_admins self.run_command(cmd) Configuring the project search profile File "/usr/local/conda/envs/esgf-pub/lib/python2.7/distutils/dist.py", line 972, in run_command Initializing project index Project=TestProject: created Top-Level folder=Bookmarks Project=TestProject: created top-level folder=Presentations Project=TestProject: created top-level folder=Publications Project=TestProject: created top-level folder=Minutes cmd_obj.run() File "/usr/local/cog/cog_install/cog/installation/setup.py", line 45, in run Project=TestProject: created top-level folder=Newsletters cogInstall.install() Project=TestProject: created top-level folder=Proposals File "/usr/local/cog/cog_install/cog/installation/install.py", line 47, in install Project=TestProject: created top-level folder=Figures self._createObjects() File "/usr/local/cog/cog_install/cog/installation/install.py", line 154, in _createObjects Project=TestProject: created top-level folder=Test Cases Project=TestProject: created top-level folder=Evaluations esgfDatabaseManager.insertEsgfUser(user.profile) File "/usr/local/cog/cog_install/cog/plugins/esgf/security.py", line 96, in insertEsgfUser Project Upload directory created: /usr/local/cog/cog_config/site_media/projects/testproject esgfUser = self.getUserByOpenid(openid) File "/usr/local/cog/cog_install/cog/plugins/esgf/security.py", line 142, in getUserByOpenid esgfUser = session.query(ESGFUser).filter(ESGFUser.openid==openid).one()
File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2693, in one ret = list(self) File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2736, in iter return self._execute_and_instances(context) File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances result = conn.execute(querycontext.statement, self._params) File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute return meth(self, multiparams, params) File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement compiled_sql, distilled_params File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context context) File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception exc_info File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause reraise(type(exception), exception, tb=exc_tb) File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context context) File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) schema "esgf_security" does not exist LINE 2: FROM esgf_security."user" ^ [SQL: 'SELECT esgf_security."user".id AS esgf_security_user_id, esgf_security."user".firstname AS esgf_security_user_firstname, esgf_security."user".middlename AS esgf_security_user_middlename, esgf_security."user".lastname AS esgf_security_user_lastname, esgf_security."user".email AS esgf_security_user_email, esgf_security."user".username AS esgf_security_user_username, esgf_security."user".password AS esgf_security_user_password, esgf_security."user".dn AS esgf_security_user_dn, esgf_security."user".openid AS esgf_security_user_openid, esgf_security."user".organization AS esgf_security_user_organization, esgf_security."user".organization_type AS esgf_security_user_organization_type, esgf_security."user".city AS esgf_security_user_city, esgf_security."user".state AS esgf_security_user_state, esgf_security."user".country AS esgf_security_user_country, esgf_security."user".status_code AS esgf_security_user_status_code, esgf_security."user".verification_token AS esgf_security_user_verification_token, esgf_security."user".notification_code AS esgf_security_user_notification_code \nFROM esgf_security."user" \nWHERE esgf_security."user".openid = %(openid_1)s'] [parameters: {'openid_1': u'https://esgf-dev1.llnl.gov/esgf-idp/openid/rootAdmin'}]

python setup.py setup_cog --esgf=true process returncode: 1 DEBUG - esg_functions.py - 443 - stream_subprocess_output - 03/28/2018 04:01:12 PM - stderr None Traceback (most recent call last): File "esg_node.py", line 509, in main(node_type_list) File "esg_node.py", line 453, in main system_component_installation(esg_dist_url, node_type_list) File "esg_node.py", line 325, in system_component_installation esg_cog.main() File "/root/esgf-installer/index_node/esg_cog.py", line 156, in main setup_cog() File "/root/esgf-installer/index_node/esg_cog.py", line 120, in setup_cog esg_functions.stream_subprocess_output("python setup.py setup_cog --esgf=true") File "/root/esgf-installer/esgf_utilities/esg_functions.py", line 447, in stream_subprocess_output raise SubprocessError({"stdout": process.stdout, "returncode": process.returncode}) esgf_utilities.esg_exceptions.SubprocessError: {'returncode': 1, 'stdout': <closed file '', mode 'rb' at 0x7f20447f0ed0>}

[William-Hill]

This issue is a symptom of design. This particular error occurs because CoG can't find the esgf_security schema. But the esgf_security schema gets set up during the esg-security installation as part of an IDP node installation. I can write a somewhat hacky solution that checks the node type and sets up the schema appropriately. But as of now, there an Index node appears to be dependent on an IDP setup being present

[William-Hill]

The ordering also matters as the Index node depends on the esgf_security schema (which is setup as part of an IDP installation).

[William-Hill]

Resolved 11b17710cdaf0040ed95fd705f04fdac5611832a