search

modoboa / modoboa-installer

An installer for Modoboa
MIT License
262 stars 138 forks source link

Docs : upgrade paths #39

Open almereyda opened 8 years ago

almereyda commented 8 years ago

Independently from installing a bare system, it is often advised to upgrade the modular components of modoboa as suggested via the /core/#info/ route of the web interface.

module pip package
Modoboa modoboa
Amavis modoboa-amavis
PDF credentials modoboa-pdfcredentials
Postfix autoreply modoboa-postfix-autoreply
Sieve filters modoboa-sievefilters
Statistics modoboa-stats
Webmail modoboa-webmail

These packages are upgraded within the modoboa virtual environment before reissuing the installer script. If, for example, a new modoboa version has been published on PyPi and a newer, unpublished version exists already in some soure control, one could do the following:

# . /srv/modoboa/env/bin/activate
(env)# pip install --upgrade modoboa
(env)# pip install https://github.com/modoboa/modoboa-pdfcredentials/archive/master.zip --upgrade
(env)# deactivate

Then a redeployment happens with

# ./run.py email.ecobytes.net --debug

One could even use different installer.cfgs to not reconfigure amavis, clamav, dovecot, nginx, razor, postfix, spamassassin and uswgi at all times. These could be loaded by parsing the provided FQDN of an instance and mapping it to distinct FQDN.cfg while an --upgrade switch omits the reconfiguration of the additional services.

almereyda commented 8 years ago

This is also discussed in https://groups.google.com/forum/#!topic/modoboa-users/sm76-n7J5f4 which tells me not to use --upgrade with pip.

spinza commented 8 years ago

The upgrade process is not clear. That to me is the main issue. It is not clear how to upgrade.

almereyda commented 8 years ago

Trying to install the IMAP migration plugin lead me to the following upgrade path:

# cd /srv/modoboa/
# apt-get update && apt-get upgrade
# source /srv/modoboa/env/bin/activate
# pip freeze --local | grep -v '^\-e' | cut -d = -f 1  | xargs -n1 pip install -U
# # ^^ from http://stackoverflow.com/a/3452888/1959568
# pip install modoboa-imap-migration
# # ^^ any not yet installed from https://pypi.python.org/pypi?%3Aaction=search&term=modoboa&submit=search
# deactivate
# cd /root/modoboa-installer/
# vim instance.cfg # to add modoboa-imap-migration to the at least represented there
# reboot
almereyda commented 8 years ago

Version specific upgrade notes live in http://modoboa.readthedocs.io/en/latest/getting_started/upgrade.html

almereyda commented 8 years ago

After the last upgrade, I ran into some internal server error. Rerunning the installer scripts helped getting rid of the flawed instance, reusing the upgraded pip packages from the procedure presented above.

almereyda commented 7 years ago

The installation instructions for Modoboa itself need to be updated with -installer based upgrade procedures.

https://github.com/tonioo/modoboa/blob/master/doc/installation.rst mentions to use it and links here, which could be extended with few example commands, but why https://github.com/tonioo/modoboa/blob/master/doc/upgrade.rst could also show similar commands as those above. The pip upgrade logic from the magic stackoverflow command would maybe find a place in this installer, too.

Having migrations handled by the installer may also be a nice trait. Before or deleting the instance folder?

tonioo commented 7 years ago

The installer does not support upgrades at the moment so the documentation only proposes the manual way.

On 22/11/2016 19:53, jon r wrote:

The installation instructions for Modoboa itself need to be updated with |-installer| based upgrade procedures.

https://github.com/tonioo/modoboa/blob/master/doc/installation.rst mentions to use it and links here, which could be extended with few example commands, but why https://github.com/tonioo/modoboa/blob/master/doc/upgrade.rst could also show similar commands as those above. The pip upgrade logic from the magic stackoverflow command would maybe find a place in this installer, too.

Having migrations handled by the installer may also be a nice trait. Before or deleting the |instance| folder?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/modoboa/modoboa-installer/issues/39#issuecomment-262330537, or mute the thread https://github.com/notifications/unsubscribe-auth/AAgc7t0rnXD4fLe3Yhx7SVLuR5WKubXXks5rAzobgaJpZM4IddKG.

almereyda commented 7 years ago

Still I have to upgrade an instance if it is installed by the installer. For upgrading the virtualenv, it appears advisable to run processes within it as the user modoboa, else the newly installed packages are installed as root. Then stopping nginx before reinstantiating the modoboa instance helps allowing Let's Encrypt to bind on port 443. Probably some inlining as in https://github.com/JrCs/docker-letsencrypt-nginx-proxy-companion/blob/master/app/nginx_location.conf can be useful here.

almereyda commented 7 years ago

During the major upgrade from 1.6.2 to 1.7.2, the following additions appeared: installer.cfg needs to be adapted with your email host. It now assumes and prepends a predefined subdomain mail to your mailhost. After running it before with ./run.py email.ecobytes.net, we have now updated the installer configuration with hostname = email.%(domain)s and invoke with ./run.py ecobytes.net. Output below also includes the --debug flag.

python manage.py migrate --noinput failed, check your configuration Traceback (most recent call last): File "manage.py", line 22, in execute_from_command_line(sys.argv) File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/init.py", line 367, in execute_from_command_line utility.execute() File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/init.py", line 359, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 294, in run_from_argv self.execute(*args, cmd_options) File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 345, in execute output = self.handle(*args, *options) File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa/core/management/commands/load_initial_data.py", line 47, in handle lc = models.LocalConfig.objects.first() File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method return getattr(self.get_queryset(), name)(args, kwargs) File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/db/models/query.py", line 556, in first objects = list((self if self.ordered else self.order_by('pk'))[:1]) File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/db/models/query.py", line 256, in iter self._fetch_all() File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/db/models/query.py", line 1087, in _fetch_all self._result_cache = list(self.iterator()) File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/db/models/query.py", line 54, in iter results = compiler.execute_sql() File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 835, in execute_sql cursor.execute(sql, params) File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute return self.cursor.execute(sql, params) File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in exit six.reraise(dj_exc_type, dj_exc_value, traceback) File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: column core_localconfig._parameters does not exist LINE 1: ...ig"."site_id", "core_localconfig"."api_versions", "core_loca...

They were merged with

email# sudo -u modoboa /bin/bash modoboa@email:~$ cd /srv/modoboa modoboa@email:~/instance$ source env/bin/activate (env)modoboa@email:~$ cd instance (env)modoboa@email:~/instance$ python manage.py makemigrations --merge Merging admin Branch 0010_auto_20170215_1328

Merging will only work if the operations printed above do not conflict with each other (working on different fields or models) Do you want to merge these migration branches? [y/N] y

Created new merge migration /srv/modoboa/env/local/lib/python2.7/site-packages/modoboa/admin/migrations/0011_merge_20170408_1331.py Merging core Branch 0012_auto_20170215_1034

Merging will only work if the operations printed above do not conflict with each other (working on different fields or models) Do you want to merge these migration branches? [y/N] y

Created new merge migration /srv/modoboa/env/local/lib/python2.7/site-packages/modoboa/core/migrations/0013_merge_20170408_1331.py (env)modoboa@email:~/instance$ deactivate


as suggested by the output above. Rerunning the installer then successfully redeployed the Web UI.
almereyda commented 6 years ago

In upgrading from 1.9.1 we had to manually introduce the missing leaf migrations again:

modoboa@email:~$ source env/bin/activate
(env)modoboa@email:~$ cd instance/
(env)modoboa@email:~/instance$ python manage.py makemigrations --merge
/srv/modoboa/env/local/lib/python2.7/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel pac
kage will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-
binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Merging admin
  Branch 0013_auto_20180124_2311
    - Add field transport to domain
    - Add field dkim_key_length to domain
    - Add field dkim_key_selector to domain
    - Add field dkim_private_key_path to domain
    - Add field dkim_public_key to domain
    - Add field enable_dkim to domain
    - Alter field dkim_key_selector on domain
    - Alter field type on domain
    - Alter field type on mailboxoperation
  Branch 0011_merge_20170408_1331
    - Alter field description on alias
    - Alter field expire_at on alias

Merging will only work if the operations printed above do not conflict
with each other (working on different fields or models)
Do you want to merge these migration branches? [y/N] y

Created new merge migration /srv/modoboa/env/local/lib/python2.7/site-packages/modoboa/admin/migrations/0014_mer
ge_20180221_2305.py
Merging core
  Branch 0013_merge_20170408_1331
    - Alter field first_name on user
    - Alter field language on user
    - Alter field last_name on user
  Branch 0015_auto_20180112_1642
    - Alter field date_joined on user
    - Alter field first_name on user
    - Alter field last_name on user
    - Raw Python operation
    - Alter field language on user

Merging will only work if the operations printed above do not conflict
with each other (working on different fields or models)
Do you want to merge these migration branches? [y/N] y

Created new merge migration /srv/modoboa/env/local/lib/python2.7/site-packages/modoboa/core/migrations/0016_merg
e_20180221_2305.py
(env)modoboa@email:~/instance$ deactivate

before running the upgrade.

almereyda commented 6 years ago

In running the update to 1.10.3, this time a non user friendly warning is thrown regarding skipped initialisation of an existing instance:

Target directory for Modoboa deployment (/srv/modoboa/instance) already exists. If you choose to continue, it will be removed.
Do you confirm? (Y/n) 
/srv/modoboa/env/local/lib/python2.7/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from
 release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.or
g/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Traceback (most recent call last):
  File "manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv                   
    self.execute(*args, **cmd_options)                                                                                                   
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa/core/management/commands/load_initial_data.py", line 64, in handle
    extension.load_initial_data()
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_amavis/modo_extension.py", line 42, in load_initial_data
    policy = create_user_and_policy("@{0}".format(dom.name))
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_amavis/lib.py", line 289, in create_user_and_policy
    return Policy.objects.get(policy_name=name[:32])
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/db/models/query.py", line 384, in get
    (self.model._meta.object_name, num)
modoboa_amavis.models.MultipleObjectsReturned: get() returned more than one Policy -- it returned 2!

['/srv/modoboa/env/bin/python2', u'manage.py', u'load_initial_data', u'--admin-username', u'admin'] failed, check your configuration
almereyda commented 6 years ago

Upgrade notes now live in http://modoboa.readthedocs.io/en/latest/upgrade.html

The notes in http://modoboa.readthedocs.io/en/latest/upgrade.html#extensions also apply

> sudo -i <modoboa_user>
> source <virtuenv_path>/bin/activate
> pip install <EXTENSION>==<VERSION>
> cd <modoboa_instance_dir>
> python manage.py migrate
> python manage.py collectstatic
> python manage.py check --deploy