Updating a server fails on Whatsapp bridge setup

mdelem commented 5 years ago

Hi there,

I get the following error when running setup-all on a already installed instance:

TASK [matrix-bridge-mautrix-whatsapp : Ensure mautrix-whatsapp config.yaml installed] **************************************************************************************************************
fatal: [matrix.delemotte.com]: FAILED! => {"msg": "An unhandled exception occurred while templating '{{ matrix_mautrix_whatsapp_configuration_yaml|from_yaml|combine(matrix_mautrix_whatsapp_configuration_extension, recursive=True) }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while templating '# Homeserver details.\nhomeserver:\n    # The address that this appservice can use to connect to the homeserver.\n    address: {{ matrix_mautrix_whatsapp_homeserver_address }}\n    # The domain of the homeserver (for MXIDs, etc).\n    domain: {{ matrix_mautrix_whatsapp_homeserver_domain }}\n# Application service host/registration related details.\n# Changing these values requires regeneration of the registration.\n\nappservice:\n    # The address that the homeserver can use to connect to this appservice.\n    address: {{ matrix_mautrix_whatsapp_appservice_address }}\n\n    # The hostname and port where this appservice should listen.\n    hostname: 0.0.0.0\n    port: 8080\n\n    # Database config.\n    database:\n        # The database type. \"sqlite3\" and \"postgres\" are supported.\n        type: sqlite3\n        # The database URI.\n        #   SQLite: File name is enough. https://github.com/mattn/go-sqlite3#connection-string\n        #   Postgres: Connection string. For example, postgres://user:password@host/database\n        uri: mautrix-whatsapp.db\n        # Maximum number of connections. Mostly relevant for Postgres.\n        max_open_conns: 20\n        max_idle_conns: 2\n\n    # Path to the Matrix room state store.\n    state_store_path: ./mx-state.json\n\n    # The unique ID of this appservice.\n    id: whatsapp\n    # Appservice bot details.\n    bot:\n        # Username of the appservice bot.\n        username: whatsappbot\n        # Display name and avatar for bot. Set to \"remove\" to remove display name/avatar, leave empty\n        # to leave display name/avatar as-is.\n        displayname: WhatsApp bridge bot\n        avatar: mxc://maunium.net/NeXNQarUbrlYBiPCpprYsRqr\n\n    # Authentication tokens for AS <-> HS communication. Autogenerated; do not modify.\n    as_token: \"{{ matrix_mautrix_whatsapp_appservice_token }}\"\n    hs_token: \"{{ matrix_mautrix_whatsapp_homeserver_token }}\"\n\n# Bridge config\nbridge:\n    # Localpart template of MXIDs for WhatsApp users.\n    # {{ '{{.}}' }} is replaced with the phone number of the WhatsApp user.\n    username_template: \"{{ 'whatsapp_{{.}}' }}\"\n    # Displayname template for WhatsApp users.\n    # {{ '{{.Notify'}}' }} - nickname set by the WhatsApp user\n    # {{ '{{.Jid}}' }}    - phone number (international format)\n    # The following variables are also available, but will cause problems on multi-user instances:\n    # {{ '{{.Name}}' }}   - display name from contact list\n    # {{ '{{.Short}}' }}  - short display name from contact list\n    displayname_template: \"{{ '{{if .Notify}}{{.Notify}}{{else}}{{.Jid}}{{end}} (WA)' }}\"\n    # WhatsApp connection timeout in seconds.\n    connection_timeout: 20\n    # Maximum number of times to retry connecting on connection error.\n    max_connection_attempts: 3\n    # Number of seconds to wait between connection attempts.\n    # Negative numbers are exponential backoff: -connection_retry_delay + 1 + 2^attempts\n    connection_retry_delay: -1\n    # Whether or not the bridge should send a notice to the user's management room when it retries connecting.\n    # If false, it will only report when it stops retrying.\n    report_connection_retry: true\n    # Number of seconds to wait for contacts and chats to be sent at startup before syncing.\n    # If you have lots of chats, it might take more than a second.\n    contact_wait_delay: 1\n    # Number of chats to sync for new users.\n    initial_chat_sync_count: 10\n    # Number of old messages to fill when creating new portal rooms.\n    initial_history_fill_count: 20\n    # Maximum number of chats to sync when recovering from downtime.\n    # Set to -1 to sync all new chats during downtime.\n    recovery_chat_sync_limit: -1\n    # Whether or not to sync history when recovering from downtime.\n    recovery_history_backfill: true\n    # Maximum number of seconds since last message in chat to skip\n    # syncing the chat in any case. This setting will take priority\n    # over both recovery_chat_sync_limit and initial_chat_sync_count.\n    # Default is 3 days = 259200 seconds\n    sync_max_chat_age: 259200\n\n    # Whether or not to sync with custom puppets to receive EDUs that\n    # are not normally sent to appservices.\n    sync_with_custom_puppets: true\n    # Whether or not to invite own WhatsApp user's Matrix puppet into private\n    # chat portals when backfilling if needed.\n    # This always uses the default puppet instead of custom puppets due to\n    # rate limits and timestamp massaging.\n    invite_own_puppet_for_backfilling: true\n    # Whether or not to explicitly set the avatar and room name for private\n    # chat portal rooms. This can be useful if the previous field works fine,\n    # but causes room avatar/name bugs.\n    private_chat_portal_meta: false\n\n    # The prefix for commands. Only required in non-management rooms.\n    command_prefix: \"!wa\"\n\n    # Permissions for using the bridge.\n    # Permitted values:\n    #     user - Access to use the bridge to chat with a WhatsApp account.\n    #    admin - User level and some additional administration tools\n    # Permitted keys:\n    #        * - All Matrix users\n    #   domain - All users on that homeserver\n    #     mxid - Specific user\n    permissions:\n        \"{{ matrix_mautrix_whatsapp_homeserver_domain }}\": user\n# Logging config.\nlogging:\n    # The directory for log files. Will be created if not found.\n    directory: ./logs\n    # Available variables: .Date for the file date and .Index for different log files on the same day.\n    file_name_format: \"{{ '{{.Date}}-{{.Index}}.log' }}\"\n    # Date format for file names in the Go time format: https://golang.org/pkg/time/#pkg-constants\n    file_date_format: \"2006-01-02\"\n    # Log file permissions.\n    file_mode: 0600\n    # Timestamp format for log entries in the Go time format.\n    timestamp_format: \"Jan _2, 2006 15:04:05\"\n    # Minimum severity for log messages.\n    # Options: debug, info, warn, error, fatal\n    print_level: debug\n'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while templating '{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'whatsapp-appservice-token') | to_uuid }}'. Error was a <class 'ValueError'>, original message: invalid characters in sha512_crypt salt"}

Any idea where to start in order to fix this?

In any case, thanks for the hard work!

mdelem commented 5 years ago

As a quick workaround I initialized matrix_mautrix_whatsapp_appservice_token and matrix_mautrix_whatsapp_homeserver_token in my vars.yaml. This does the trick.

spantaleev commented 5 years ago

@Cadair has recently reported the same issue.

Could you tell us more about your setup:

are you running the playbook as-is, or are you integrating its roles into another playbook of yours?
is your matrix_synapse_macaroon_secret_key value encrypted using Ansible Vault or something?
what is your matrix_synapse_macaroon_secret_key value like? Any weird characters? (Mine contains things like this ^*#&, but those don't seem to cause trouble for me)
what version of Ansible are you running and on what distro? (pasting ansible --version would be great)

mdelem commented 5 years ago

Hi @spantaleev,

I'm running the paybook as-is
My matrix_synapse_macaroon_secret_key is stored in my vars.yaml, no Ansible Vault
The value of my matrix_synapse_macaroon_secret_key was generated withpwgen -s 64 1, it has lowercase chars, uppercase chars and numbers. No spaces nor special chars.

$ ansible --version


config file = None
configured module search path = ['/Users/***/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/Cellar/ansible/2.8.1_1/libexec/lib/python3.7/site-packages/ansible
executable location = /usr/local/bin/ansible
python version = 3.7.3 (default, Mar 27 2019, 09:23:15) [Clang 10.0.1 (clang-1001.0.46.3)]```

spantaleev commented 5 years ago

I believe I've found and fixed the problem.

It only happens on Mac, because Ansible relies on passlib for password_hash there. For other operating systems, Ansible falls back to using Python's own crypt module. Those operating systems usually don't install passlib as a dependency for Ansible.

Until now, our salt strings were sometimes too long (longer than the 16 character limit) and they also included characters that are not allowed (like -).

While crypt is very forgiving when it comes to salt length and salt characters, passlib is not.

In 782356d42171142dc9a485a, I've made salt strings obey passlib's strict requirements, so it should not longer complain.

spantaleev / matrix-docker-ansible-deploy

Updating a server fails on Whatsapp bridge setup #209