search

zatosource / zato

ESB, SOA, REST, APIs and Cloud Integrations in Python
https://zato.io
GNU Affero General Public License v3.0
1.12k stars 240 forks source link

Python TB while trying to start zato after fresh quickinstall #75

Closed seletz closed 11 years ago

seletz commented 11 years ago

Problem

After doing a fresh zato quickinstall as per the tutorial, the zato-qs-start.sh gives me a python traceback.

Versions

OSX 10.8 zato 1.1 hotfix from #74 installed

Traceback

 zato quickstart create ./qs-1 postgresql localhost 5432 zato1 zato1 \               ] 8:56 am
    localhost 6379 --verbose

ODB database password (will not be echoed):
Enter the odb_password again (will not be echoed):

Key/value database password (will not be echoed):
Enter the kvdb_password again (will not be echoed):
[1/8] Certificate authority created
[2/8] ODB schema already exists
[3/8] ODB initial data created
[4/8] server1 created
[5/8] server2 created
[6/8] Load-balancer created
[7/8] Web admin created
[8/8] Management scripts created
Quickstart cluster quickstart-918610 created
User [admin] already exists in the ODB
Start the cluster by issuing the /Users/seletz/develop/python/zato/qs-1/zato-qs-start.sh command
Visit https://zato.io/support for more information and support options

(zato-1.1)zato/qs-1 [ ./zato-qs-start.sh                                                                    ] 8:56 am
Starting the Zato quickstart environment
Running sanity checks
Traceback (most recent call last):
  File "/Users/seletz/develop/python/zato/zato-1.1/bin/zato", line 89, in <module>
    sys.exit(zato.cli.zato_command.main())
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/zato_command.py", line 227, in main
    return run_command(get_parser().parse_args())
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/__init__.py", line 164, in run_command
    command_class[args.command](args).run(args)
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/__init__.py", line 358, in run
    sys.exit(self.execute(args))
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/__init__.py", line 612, in execute
    self._get_dispatch()[json_data['component']](args)
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/check_config.py", line 53, in _on_server
    kvdb.init()
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-common/src/zato/common/kvdb.py", line 59, in init
    config['password'] = self.decrypt_func(self.config.password)
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-common/src/zato/common/crypto.py", line 67, in encrypt
    encrypted = self.pub_key.public_encrypt(data, padding)
AttributeError: 'NoneType' object has no attribute 'public_encrypt'
seletz commented 11 years ago

So I'm good at python, trying to dig in myself a bit. Might be that I removed a vital package from the buildout to make it run. I installed m2crypto, though.

seletz commented 11 years ago

Ok, it seems that in zato.common/zato/common/crypto.py the CryptoManager gets initialized w/o public and private keys:

> /Users/seletz/develop/python/zato/zato-1.1/zato-common/src/zato/common/crypto.py(32)__init__()->None
-> import pdb; pdb.set_trace()
(Pdb) args
self = <zato.common.crypto.CryptoManager object at 0x10b9b03d0>
priv_key_location = /Users/seletz/develop/python/zato/qs-2/server1/config/repo/zato-server-priv-key.pem
pub_key_location = None
priv_key = None
pub_key = None
seletz commented 11 years ago

Ah, in the server_conf the paths are built wrongly for getting hold of the .pem files.

seletz commented 11 years ago

No, the argument list is wrong to the crypto manager.

seletz commented 11 years ago

Ok, a quick hack for this traceback -- however, the config object should really know about paths ...

--- check_config.py.orig    2013-06-19 09:17:14.000000000 +0200
+++ check_config.py 2013-06-19 09:21:16.000000000 +0200
@@ -32,7 +32,17 @@
         repo_dir = join(self.config_dir, 'repo')
         server_conf = ConfigObj(join(repo_dir, 'server.conf'))

-        cm = CryptoManager(priv_key_location=abspath(join(repo_dir, server_conf['crypto']['priv_key_location'])))
+        # fix for #75
+        # this should be fixed in the ConfigObj, though.  Paths should be abspath'ed by
+        # that object.
+        crypto_manager_args = []
+        for key, value in server_conf["crypto"].items():
+            crypto_manager_args.append((key, abspath(join(repo_dir, value))))
+        crypto_manager_args = dict(crypto_manager_args)
+        del crypto_manager_args["ca_certs_location"]
+        del crypto_manager_args["cert_location"]
+
+        cm = CryptoManager(**crypto_manager_args)
         cm.load_keys()

         engine_params = dict(server_conf['odb'].items())
dsuch commented 11 years ago

Hi Stefan,

I've just tried it on a fresh install w/ hotfixes on Ubuntu and there are no issues at all. Everything was created, started and works normally.

There should be no need to remove or add any packages to buildout, the idea is that install.sh (w/ hotfixman.sh) does everything and sets everything up without a need for any manual tweaking.

So if you did remove a package to make it run, can you please tell me why you had to do it? Why wouldn't it run without it?

In that time, I only have a remote access to an OS X and I'll be testing it out afresh over there.

Thanks!

seletz commented 11 years ago

Well, originally it barfed -- read: did not compile -- on everything related to inotify. I figured that package is needed due to hot-code loading, and wanted to start w/o that. So that's the only packages removed -- nothing crypto related.

However, I needed to install m2crypt because some package tried to import it and it was not installed. I pip installed that one, and -- for good measure -- put it in the buildout.cfg in the zato part as egg.

dsuch commented 11 years ago

OK, I just did a fresh 1.1 install on OS X 10.8 and had absolutely no problems. I didn't have to manually modify or update anything, it simply works as intended.

When you say you had issues with inotify it makes me think you were trying 1.0 not really 1.1 - 1.0 could not be installed on OS X because of inotify-related packages and changing the installation process so that OS X has its own buildout version (without inotify) was one of new things in 1.1.

So can you please try it out again, with 1.1, and without modifying any packages, dependencies or anything. Just please follow the steps described here https://zato.io/docs/admin/guide/install-zato.html

If it still doesn't work -> please paste here the whole shell session output, starting from downloading the installation package and up to the moment where the traceback appears.

Thanks again!

seletz commented 11 years ago

Ok, did reinstall 1.1, did indeed complete correctly. Did zato quickstart create ..., completed successfully.

HOWEVER, I still get the traceback reported in this issue.

Complete log, including typos et al -- w/o the download -- the installation did complete OK.

The first few errors come from the typo in the quickstart command ('zato' instead of 'zato1').

(zato-1.1)python/zato [ zato quickstart create $(pwd)/qs-1 postgresql localhost 5432 zato1 zato localhost 6379 --verbose                                                                                                                                              ] 1:13 pm

ODB database password (will not be echoed):
Enter the odb_password again (will not be echoed):

Key/value database password (will not be echoed):
Enter the kvdb_password again (will not be echoed):
[1/8] Certificate authority created
Traceback (most recent call last):
  File "/Users/seletz/develop/python/zato/zato-1.1/bin/zato", line 89, in <module>
    sys.exit(zato.cli.zato_command.main())
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/zato_command.py", line 227, in main
    return run_command(get_parser().parse_args())
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/__init__.py", line 164, in run_command
    command_class[args.command](args).run(args)
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/__init__.py", line 358, in run
    sys.exit(self.execute(args))
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/quickstart.py", line 240, in execute
    if create_odb.Create(args).execute(args, False) == self.SYS_ERROR.ODB_EXISTS:
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/create_odb.py", line 33, in execute
    if engine.dialect.has_table(engine.connect(), 'install_state'):
  File "/Users/seletz/.buildout/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.8-x86_64.egg/sqlalchemy/engine/base.py", line 2471, in connect
    return self._connection_cls(self, **kwargs)
  File "/Users/seletz/.buildout/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.8-x86_64.egg/sqlalchemy/engine/base.py", line 878, in __init__
    self.__connection = connection or engine.raw_connection()
  File "/Users/seletz/.buildout/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.8-x86_64.egg/sqlalchemy/engine/base.py", line 2557, in raw_connection
    return self.pool.unique_connection()
  File "/Users/seletz/.buildout/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.8-x86_64.egg/sqlalchemy/pool.py", line 184, in unique_connection
    return _ConnectionFairy(self).checkout()
  File "/Users/seletz/.buildout/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.8-x86_64.egg/sqlalchemy/pool.py", line 401, in __init__
    rec = self._connection_record = pool._do_get()
  File "/Users/seletz/.buildout/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.8-x86_64.egg/sqlalchemy/pool.py", line 746, in _do_get
    con = self._create_connection()
  File "/Users/seletz/.buildout/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.8-x86_64.egg/sqlalchemy/pool.py", line 189, in _create_connection
    return _ConnectionRecord(self)
  File "/Users/seletz/.buildout/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.8-x86_64.egg/sqlalchemy/pool.py", line 282, in __init__
    self.connection = self.__connect()
  File "/Users/seletz/.buildout/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.8-x86_64.egg/sqlalchemy/pool.py", line 344, in __connect
    connection = self.__pool._creator()
  File "/Users/seletz/.buildout/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.8-x86_64.egg/sqlalchemy/engine/strategies.py", line 80, in connect
    return dialect.connect(*cargs, **cparams)
  File "/Users/seletz/.buildout/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.8-x86_64.egg/sqlalchemy/engine/default.py", line 281, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/Users/seletz/.buildout/eggs/psycopg2-2.4.5-py2.7-macosx-10.8-x86_64.egg/psycopg2/__init__.py", line 179, in connect
    connection_factory=connection_factory, async=async)
sqlalchemy.exc.OperationalError: (OperationalError) FATAL:  database "zato" does not exist
 None None
(zato-1.1)python/zato [ export PATH=~/Applications/Postgres.app/Contents/MacOS/bin:$PATH                                                                                                                                                                              ] 1:14 pm
(zato-1.1)python/zato [ zato quickstart create $(pwd)/qs-1 postgresql localhost 5432 zato1 zato localhost 6379 --verbose                                                                                                                                              ] 1:14 pm
Directory /Users/seletz/develop/python/zato/qs-1 is not empty, please re-run the command in an empty directory
(zato-1.1)python/zato [ rm -rf qs-1                                                                                                                                                                                                                                   ] 1:14 pm
(zato-1.1)python/zato [                                                                                                                                                                                                                                               ] 1:14 pm
(zato-1.1)python/zato [                                                                                                                                                                                                                                               ] 1:14 pm
(zato-1.1)python/zato [ mkdir qs-1                                                                                                                                                                                                                                    ] 1:14 pm
(zato-1.1)python/zato [ zato quickstart create $(pwd)/qs-1 postgresql localhost 5432 zato1 zato localhost 6379 --verbose                                                                                                                                              ] 1:14 pm

ODB database password (will not be echoed):
Enter the odb_password again (will not be echoed):

Key/value database password (will not be echoed):
Enter the kvdb_password again (will not be echoed):
[1/8] Certificate authority created
Traceback (most recent call last):
  File "/Users/seletz/develop/python/zato/zato-1.1/bin/zato", line 89, in <module>
    sys.exit(zato.cli.zato_command.main())
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/zato_command.py", line 227, in main
    return run_command(get_parser().parse_args())
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/__init__.py", line 164, in run_command
    command_class[args.command](args).run(args)
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/__init__.py", line 358, in run
    sys.exit(self.execute(args))
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/quickstart.py", line 240, in execute
    if create_odb.Create(args).execute(args, False) == self.SYS_ERROR.ODB_EXISTS:
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/create_odb.py", line 33, in execute
    if engine.dialect.has_table(engine.connect(), 'install_state'):
  File "/Users/seletz/.buildout/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.8-x86_64.egg/sqlalchemy/engine/base.py", line 2471, in connect
    return self._connection_cls(self, **kwargs)
  File "/Users/seletz/.buildout/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.8-x86_64.egg/sqlalchemy/engine/base.py", line 878, in __init__
    self.__connection = connection or engine.raw_connection()
  File "/Users/seletz/.buildout/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.8-x86_64.egg/sqlalchemy/engine/base.py", line 2557, in raw_connection
    return self.pool.unique_connection()
  File "/Users/seletz/.buildout/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.8-x86_64.egg/sqlalchemy/pool.py", line 184, in unique_connection
    return _ConnectionFairy(self).checkout()
  File "/Users/seletz/.buildout/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.8-x86_64.egg/sqlalchemy/pool.py", line 401, in __init__
    rec = self._connection_record = pool._do_get()
  File "/Users/seletz/.buildout/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.8-x86_64.egg/sqlalchemy/pool.py", line 746, in _do_get
    con = self._create_connection()
  File "/Users/seletz/.buildout/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.8-x86_64.egg/sqlalchemy/pool.py", line 189, in _create_connection
    return _ConnectionRecord(self)
  File "/Users/seletz/.buildout/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.8-x86_64.egg/sqlalchemy/pool.py", line 282, in __init__
    self.connection = self.__connect()
  File "/Users/seletz/.buildout/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.8-x86_64.egg/sqlalchemy/pool.py", line 344, in __connect
    connection = self.__pool._creator()
  File "/Users/seletz/.buildout/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.8-x86_64.egg/sqlalchemy/engine/strategies.py", line 80, in connect
    return dialect.connect(*cargs, **cparams)
  File "/Users/seletz/.buildout/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.8-x86_64.egg/sqlalchemy/engine/default.py", line 281, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/Users/seletz/.buildout/eggs/psycopg2-2.4.5-py2.7-macosx-10.8-x86_64.egg/psycopg2/__init__.py", line 179, in connect
    connection_factory=connection_factory, async=async)
sqlalchemy.exc.OperationalError: (OperationalError) FATAL:  database "zato" does not exist
 None None
(zato-1.1)python/zato [ psql --dbname zato1                                                                                                                                                                                                                           ] 1:14 pm
psql (9.2.4)
Type "help" for help.

zato1=# help
You are using psql, the command-line interface to PostgreSQL.
Type:  \copyright for distribution terms
       \h for help with SQL commands
       \? for help with psql commands
       \g or terminate with semicolon to execute query
       \q to quit
zato1=# show tables;
ERROR:  unrecognized configuration parameter "tables"
                         List of relations
 Schema |                  Name                  |   Type   | Owner
--------+----------------------------------------+----------+-------
 public | auth_group                             | table    | zato1
 public | auth_group_id_seq                      | sequence | zato1
 public | auth_group_permissions                 | table    | zato1
 public | auth_group_permissions_id_seq          | sequence | zato1
 public | auth_message                           | table    | zato1
 public | auth_message_id_seq                    | sequence | zato1
 public | auth_permission                        | table    | zato1
 public | auth_permission_id_seq                 | sequence | zato1
 public | auth_user                              | table    | zato1
 public | auth_user_groups                       | table    | zato1
 public | auth_user_groups_id_seq                | sequence | zato1
 public | auth_user_id_seq                       | sequence | zato1
 public | auth_user_user_permissions             | table    | zato1
 public | auth_user_user_permissions_id_seq      | sequence | zato1
 public | channel_amqp                           | table    | zato1
 public | channel_amqp_seq                       | sequence | zato1
 public | channel_wmq                            | table    | zato1
 public | channel_wmq_seq                        | sequence | zato1
 public | channel_zmq                            | table    | zato1
 public | channel_zmq_seq                        | sequence | zato1
 public | cluster                                | table    | zato1
 public | cluster_color_marker                   | table    | zato1
 public | cluster_color_marker_id_seq            | sequence | zato1
 public | cluster_id_seq                         | sequence | zato1
 public | conn_def_amqp                          | table    | zato1
 public | conn_def_amqp_seq                      | sequence | zato1
 public | conn_def_wmq                           | table    | zato1
 public | conn_def_wmq_seq                       | sequence | zato1
 public | depl_package_seq                       | sequence | zato1
 public | depl_status_seq                        | sequence | zato1
 public | deployed_service                       | table    | zato1
 public | deployment_package                     | table    | zato1
 public | deployment_status                      | table    | zato1
 public | django_content_type                    | table    | zato1
 public | django_content_type_id_seq             | sequence | zato1
 public | django_session                         | table    | zato1
 public | django_settings_email                  | table    | zato1
 public | django_settings_email_id_seq           | sequence | zato1
 public | django_settings_integer                | table    | zato1
 public | django_settings_integer_id_seq         | sequence | zato1
 public | django_settings_positiveinteger        | table    | zato1
 public | django_settings_positiveinteger_id_seq | sequence | zato1
 public | django_settings_setting                | table    | zato1
 public | django_settings_setting_id_seq         | sequence | zato1
 public | django_settings_string                 | table    | zato1
 public | django_settings_string_id_seq          | sequence | zato1
 public | django_site                            | table    | zato1
 public | django_site_id_seq                     | sequence | zato1
 public | http_soap                              | table    | zato1
 public | http_soap_seq                          | sequence | zato1
 public | install_state                          | table    | zato1
 public | install_state_seq                      | sequence | zato1
 public | job                                    | table    | zato1
 public | job_cron_seq                           | sequence | zato1
zato1=# show;
ERROR:  syntax error at or near ";"
LINE 1: show;
            ^
zato1=# \h
Available help:
  ABORT                            ALTER SCHEMA                     COMMENT                          CREATE OPERATOR FAMILY           DECLARE                          DROP OPERATOR FAMILY             EXECUTE                          ROLLBACK TO SAVEPOINT
  ALTER AGGREGATE                  ALTER SEQUENCE                   COMMIT                           CREATE ROLE                      DELETE                           DROP OWNED                       EXPLAIN                          SAVEPOINT
  ALTER COLLATION                  ALTER SERVER                     COMMIT PREPARED                  CREATE RULE                      DISCARD                          DROP ROLE                        FETCH                            SECURITY LABEL
  ALTER CONVERSION                 ALTER TABLE                      COPY                             CREATE SCHEMA                    DO                               DROP RULE                        GRANT                            SELECT
  ALTER DATABASE                   ALTER TABLESPACE                 CREATE AGGREGATE                 CREATE SEQUENCE                  DROP AGGREGATE                   DROP SCHEMA                      INSERT                           SELECT INTO
  ALTER DEFAULT PRIVILEGES         ALTER TEXT SEARCH CONFIGURATION  CREATE CAST                      CREATE SERVER                    DROP CAST                        DROP SEQUENCE                    LISTEN                           SET
  ALTER DOMAIN                     ALTER TEXT SEARCH DICTIONARY     CREATE COLLATION                 CREATE TABLE                     DROP COLLATION                   DROP SERVER                      LOAD                             SET CONSTRAINTS
  ALTER EXTENSION                  ALTER TEXT SEARCH PARSER         CREATE CONVERSION                CREATE TABLE AS                  DROP CONVERSION                  DROP TABLE                       LOCK                             SET ROLE
  ALTER FOREIGN DATA WRAPPER       ALTER TEXT SEARCH TEMPLATE       CREATE DATABASE                  CREATE TABLESPACE                DROP DATABASE                    DROP TABLESPACE                  MOVE                             SET SESSION AUTHORIZATION
  ALTER FOREIGN TABLE              ALTER TRIGGER                    CREATE DOMAIN                    CREATE TEXT SEARCH CONFIGURATION DROP DOMAIN                      DROP TEXT SEARCH CONFIGURATION   NOTIFY                           SET TRANSACTION
  ALTER FUNCTION                   ALTER TYPE                       CREATE EXTENSION                 CREATE TEXT SEARCH DICTIONARY    DROP EXTENSION                   DROP TEXT SEARCH DICTIONARY      PREPARE                          SHOW
  ALTER GROUP                      ALTER USER                       CREATE FOREIGN DATA WRAPPER      CREATE TEXT SEARCH PARSER        DROP FOREIGN DATA WRAPPER        DROP TEXT SEARCH PARSER          PREPARE TRANSACTION              START TRANSACTION
  ALTER INDEX                      ALTER USER MAPPING               CREATE FOREIGN TABLE             CREATE TEXT SEARCH TEMPLATE      DROP FOREIGN TABLE               DROP TEXT SEARCH TEMPLATE        REASSIGN OWNED                   TABLE
  ALTER LANGUAGE                   ALTER VIEW                       CREATE FUNCTION                  CREATE TRIGGER                   DROP FUNCTION                    DROP TRIGGER                     REINDEX                          TRUNCATE
  ALTER LARGE OBJECT               ANALYZE                          CREATE GROUP                     CREATE TYPE                      DROP GROUP                       DROP TYPE                        RELEASE SAVEPOINT                UNLISTEN
  ALTER OPERATOR                   BEGIN                            CREATE INDEX                     CREATE USER                      DROP INDEX                       DROP USER                        RESET                            UPDATE
  ALTER OPERATOR CLASS             CHECKPOINT                       CREATE LANGUAGE                  CREATE USER MAPPING              DROP LANGUAGE                    DROP USER MAPPING                REVOKE                           VACUUM
  ALTER OPERATOR FAMILY            CLOSE                            CREATE OPERATOR                  CREATE VIEW                      DROP OPERATOR                    DROP VIEW                        ROLLBACK                         VALUES
  ALTER ROLE                       CLUSTER                          CREATE OPERATOR CLASS            DEALLOCATE                       DROP OPERATOR CLASS              END                              ROLLBACK PREPARED                WITH
zato1=# \?
zato1=# \d
zato1=# ^D\q
(zato-1.1)python/zato [                                                                                                                                                                                                                                               ] 1:16 pm
(zato-1.1)python/zato [                                                                                                                                                                                                                                               ] 1:16 pm
(zato-1.1)python/zato [                                                                                                                                                                                                                                               ] 1:16 pm
(zato-1.1)python/zato [                                                                                                                                                                                                                                               ] 1:16 pm
(zato-1.1)python/zato [ hash -r                                                                                                                                                                                                                                       ] 1:16 pm
(zato-1.1)python/zato [ ll                                                                                                                                                                                                                                            ] 1:16 pm
insgesamt 536
drwxr-xr-x  3 seletz staff    102 19. Jun 13:14 qs-1
drwxr-xr-x 30 seletz staff   1020 19. Jun 13:12 zato-1.1
-rw-r--r--  1 seletz staff 548603 19. Jun 13:07 zato-1.1.tar.bz2
(zato-1.1)python/zato [ rm -rf qs-1                                                                                                                                                                                                                                   ] 1:16 pm
(zato-1.1)python/zato [                                                                                                                                                                                                                                               ] 1:16 pm
(zato-1.1)python/zato [                                                                                                                                                                                                                                               ] 1:16 pm
(zato-1.1)python/zato [                                                                                                                                                                                                                                               ] 1:16 pm
(zato-1.1)python/zato [ ll                                                                                                                                                                                                                                            ] 1:16 pm
insgesamt 536
drwxr-xr-x 30 seletz staff   1020 19. Jun 13:12 zato-1.1
-rw-r--r--  1 seletz staff 548603 19. Jun 13:07 zato-1.1.tar.bz2
(zato-1.1)python/zato [ zato quickstart create $(pwd)/qs-1 postgresql localhost 5432 zato1 zato1 localhost 6379 --verbose                                                                                                                                             ] 1:16 pm
Traceback (most recent call last):
  File "/Users/seletz/develop/python/zato/zato-1.1/bin/zato", line 89, in <module>
    sys.exit(zato.cli.zato_command.main())
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/zato_command.py", line 227, in main
    return run_command(get_parser().parse_args())
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/__init__.py", line 164, in run_command
    command_class[args.command](args).run(args)
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/__init__.py", line 330, in run
    for elem in os.listdir(work_dir):
OSError: [Errno 2] No such file or directory: '/Users/seletz/develop/python/zato/qs-1'
(zato-1.1)python/zato [ mkdir qs-1                                                                                                                                                                                                                                    ] 1:17 pm
(zato-1.1)python/zato [ zato quickstart create $(pwd)/qs-1 postgresql localhost 5432 zato1 zato1 localhost 6379 --verbose                                                                                                                                             ] 1:17 pm

ODB database password (will not be echoed):
Enter the odb_password again (will not be echoed):

Key/value database password (will not be echoed):
Enter the kvdb_password again (will not be echoed):
[1/8] Certificate authority created
[2/8] ODB schema already exists
[3/8] ODB initial data created
[4/8] server1 created
[5/8] server2 created
[6/8] Load-balancer created
[7/8] Web admin created
[8/8] Management scripts created
Quickstart cluster quickstart-844596 created
User [admin] already exists in the ODB
Start the cluster by issuing the /Users/seletz/develop/python/zato/qs-1/zato-qs-start.sh command
Visit https://zato.io/support for more information and support options
(zato-1.1)python/zato [ cd qs-1                                                                                                                                                                                                                                       ] 1:17 pm
(zato-1.1)zato/qs-1 [                                                                                                                                                                                                                                                 ] 1:17 pm
(zato-1.1)zato/qs-1 [                                                                                                                                                                                                                                                 ] 1:17 pm
(zato-1.1)zato/qs-1 [                                                                                                                                                                                                                                                 ] 1:17 pm
(zato-1.1)zato/qs-1 [ ll                                                                                                                                                                                                                                              ] 1:17 pm
insgesamt 12
drwxr-xr-x 9 seletz staff  306 19. Jun 13:17 ca
drwxr-xr-x 5 seletz staff  170 19. Jun 13:17 load-balancer
drwxr-xr-x 7 seletz staff  238 19. Jun 13:17 server1
drwxr-xr-x 7 seletz staff  238 19. Jun 13:17 server2
drwxr-xr-x 5 seletz staff  170 19. Jun 13:17 web-admin
-rwxr----- 1 seletz staff  370 19. Jun 13:17 zato-qs-restart.sh
-rwxr----- 1 seletz staff 1137 19. Jun 13:17 zato-qs-start.sh
-rwxr----- 1 seletz staff  786 19. Jun 13:17 zato-qs-stop.sh
(zato-1.1)zato/qs-1 [                                                                                                                                                                                                                                                 ] 1:17 pm
(zato-1.1)zato/qs-1 [                                                                                                                                                                                                                                                 ] 1:17 pm
(zato-1.1)zato/qs-1 [ ./zato-qs-start.sh                                                                                                                                                                                                                              ] 1:17 pm
Starting the Zato quickstart environment
Running sanity checks
Traceback (most recent call last):
  File "/Users/seletz/develop/python/zato/zato-1.1/bin/zato", line 89, in <module>
    sys.exit(zato.cli.zato_command.main())
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/zato_command.py", line 227, in main
    return run_command(get_parser().parse_args())
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/__init__.py", line 164, in run_command
    command_class[args.command](args).run(args)
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/__init__.py", line 358, in run
    sys.exit(self.execute(args))
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/__init__.py", line 612, in execute
    self._get_dispatch()[json_data['component']](args)
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/check_config.py", line 53, in _on_server
    kvdb.init()
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-common/src/zato/common/kvdb.py", line 59, in init
    config['password'] = self.decrypt_func(self.config.password)
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-common/src/zato/common/crypto.py", line 67, in encrypt
    encrypted = self.pub_key.public_encrypt(data, padding)
AttributeError: 'NoneType' object has no attribute 'public_encrypt'
dsuch commented 11 years ago

OK, the only difference I can see is that I /think/ you're doing everything under virtualenv, is that right?

See, the way the installer works (_install-darwin.sh for you) is:

So there is no need to run virtualenv yourself because a Zato installation is already isolated.

That said, I'm not really a pip/virtualenv user. I know this was something you weren't expecting :-) But you see, I always liked buildout, well, that's just the way it is. In any case, do you think the fact of your using virtualenv may influence things?

I'm just not sure myself and I'd like to ask for your opinion, you're doing something the installation steps don't mention, you're having issues so I'm just trying to figure out how best to deal with it.

seletz commented 11 years ago

Indeed I use virtualenv. The darwin install.sh tries to pip install python packages. I disabled this on purpose completely on my system. Apple ships python and uses python in some of the OOTB software installed on the Mac -- I therefore use a homebrewed python.

And even there, I do not want to pollute my site-packages because I try some software. Therefore I use virtualenv for everything. Thus I have m2crypto nose and zdaemon installed in the virtualenv the install-darwin.sh creates -- that is, I didn't create a new virtualenv.

Note -- this is not a uncommon setup at all, IMNSHO this is the only sane way to do python development. I've been bitten far too often by some package being updated silently and having my software break in very subtle and hard to debug ways.

I'm a old-school Zope and Plone developer, I know my way around when it comes to buildout.

FWIW -- here w/o having the virtualenv activated:

python/zato [ zato-1.1/bin/zato quickstart create $(pwd)/qs-1 postgresql localhost 5432 zato1 zato1 localhost 6379 --verbose                                                                                                                                          ] 2:00 pm

ODB database password (will not be echoed):
Enter the odb_password again (will not be echoed):

Key/value database password (will not be echoed):
Enter the kvdb_password again (will not be echoed):
[1/8] Certificate authority created
[2/8] ODB schema already exists
[3/8] ODB initial data created
[4/8] server1 created
[5/8] server2 created
[6/8] Load-balancer created
[7/8] Web admin created
[8/8] Management scripts created
Quickstart cluster quickstart-380146 created
User [admin] already exists in the ODB
Start the cluster by issuing the /Users/seletz/develop/python/zato/qs-1/zato-qs-start.sh command
Visit https://zato.io/support for more information and support options
python/zato [ export PATH=$PATH:$(pwd)/zato-1.1/bin                                                                                                                                                                                                                   ] 2:01 pm
python/zato [ cd qs-1                                                                                                                                                                                                                                                 ] 2:01 pm
zato/qs-1 [ ./zato-qs-start.sh                                                                                                                                                                                                                                        ] 2:01 pm
Starting the Zato quickstart environment
Running sanity checks
Traceback (most recent call last):
  File "/Users/seletz/develop/python/zato/zato-1.1/bin/zato", line 89, in <module>
    sys.exit(zato.cli.zato_command.main())
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/zato_command.py", line 227, in main
    return run_command(get_parser().parse_args())
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/__init__.py", line 164, in run_command
    command_class[args.command](args).run(args)
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/__init__.py", line 358, in run
    sys.exit(self.execute(args))
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/__init__.py", line 612, in execute
    self._get_dispatch()[json_data['component']](args)
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-cli/src/zato/cli/check_config.py", line 53, in _on_server
    kvdb.init()
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-common/src/zato/common/kvdb.py", line 59, in init
    config['password'] = self.decrypt_func(self.config.password)
  File "/Users/seletz/develop/python/zato/zato-1.1/zato-common/src/zato/common/crypto.py", line 67, in encrypt
    encrypted = self.pub_key.public_encrypt(data, padding)
AttributeError: 'NoneType' object has no attribute 'public_encrypt'
seletz commented 11 years ago

BTW, the installation process would be much less involved if you just added a requirements.txt to your repository -- e.g. the output of "pip freeze".

I know buildout, and I know the advantages of recipes -- indeed, I wrote quite some recipes back then -- and pinned versions. But I have come to learn that most other pythonistas use virtualenv and pip for dependencies nowadays. This is one of the failures of Zope and and Plone IMHO.

dsuch commented 11 years ago

Sure, that's also my idea - to isolate Zato from the outside world as far as possible.

But please understand my situation -> I can install and run Zato fine, others also have no problems but you've done some updates to the installer and it doesn't work for you. Please help me understand what I can do about it.

I would just love to get it going for you but I don't know how -> if you feel nose/m2crypto/zdaemon should not be updated globally at all, can you come up with a solution that doesn't involve it?

As for freezing dependencies with pip -> my idea for 1.2 is to stop connecting to the Internet almost at all. Zato has ~70 dependencies and during this first month I've already seen install.sh failing to do its job because PyPI was down, because someone's sites was or because people were pulling dependencies on their sites down or perhaps because they were hiding their releases on PyPI. And 70 dependencies is only a beginning, I have absolutely no problems with adding 700 more if it means Zato can be used to save weeks, months or years of work down the road.

A thing I'm working right now as we speak is to catch all the dependencies with buildout's download-cache and download them in one go in ./install.sh without having to do all the index scans, PyPI lookups etc. unnecessarily each time anyone installs Zato.

As for making the installation process less involved - it really is a matter of running install.sh and hotfixman.sh - I understand you're not OK with a couple of packages being updated behind the scenes so you chose to change things but at that point I simply don't really know how to help you with it. I would really like to but just don't know how to begin.

Btw., I don't have much experience with Plone/Zope - buildout is just something I've been doing for my other Python projects.

And like I say - Zato is software to save time when someone in in the integrations. However much I value the hard work of other Python programmers from Distutuls-SIG, whether Zato uses pip, buildout or anything else is completely irrelevant to me as long as it's easy to create working environments without spending too much time on it :-)

In fact, I will warmly welcome the day when Linux packagers make it possible to do 'apt-get install zato'.

As for OS X - there's ticket #41 - the idea is to be able to do 'brew install zato'. Given that it seems like an itch you'd like to scratch, would you like to have a look at it? Think this would answer your needs very nicely.

seletz commented 11 years ago

Ok, sure. BTW, I didn't want to blame or flame -- I just would love to try out zato as it might fit into some project I'm working on. My machine is used almost exclusively for python and jython development -- so I think my setup is quite OK -- OTOH I probably need to dig deeper into zato to understand the needs of the installation process.

Re PyPi availability -- I know the pain :p Back in the days when I used buildout I shipped all the eggs and just pointed buildout to a pre-filled download cache and disabled network access. That was a very good solution for shipping releases from staging to production ...

I'll not give up -- I'll probably try using a clean ubuntu virtual machine first.

And I'll try to look at #41.

Thanks so far -- I'll report back when I have more ;)

dsuch commented 11 years ago

Sure Stefan, no worries! :-)

I'm really thankful for all your feedback, it's just that I'm not always able to deal with everything myself. Like say with #68 (CentOS install) - a fellow Pythonista provided hints what was needed and it will be incorporated in 1.2. So the way install.sh works now is quite OK, it tries not to mess around with system packages as long as it's not necessary.

As for the installation needs - it all started with a single buildout.cfg. Then I added versions.cfg. Then I added more dependencies and it turned out it was pain to install some under buildout. So I added install.sh. Then install.sh turned out to be a cool thing for deciding what OS a user is on. Etc .etc.

But basically, from an installation point of view, Zato is just several Python packages, each with its setup.py and buildout is the core thing to run in order to set it up. The rest mostly prepares prerequisites.

Two other things I have in mind for the future are

dsuch commented 11 years ago

OK, Stefan, is it cool with you if I close both #75 and #76? Is it fine if we move the effort over to #41 so there's a proper howebrew-based OS X installation process?

seletz commented 11 years ago

Yeah, sure.

dsuch commented 11 years ago

Closing in favour of #41.

dsuch commented 11 years ago

One thing more, there was a recent fix that seems to have been to do with the same thing you were experiencing.

It will be dealt with if you pull the latest hotfixes

https://zato.io/docs/admin/guide/hotfixes.html

That said, let's still focus on #41, I'm just saying this is a related thing.