support celery 5

[Findus23]

related to #31

The way celery is started, doesn't seem to be supported any more in celery 5 (shipped with Debian Bullseye)

https://github.com/Salamek/gitlab-tools/blob/2b66194ad5a75c7c887728554d38d9a6e85dab22/gitlab_tools/bin/gitlab_tools.py#L513-L521

https://github.com/celery/celery/blob/552e067b40198429cd7c866a397069366ac8e530/celery/bin/celery.py#L206-L213

[Findus23]

sudo -u gitlab-tools celery -A gitlab_tools.tasks.gitlab_tools beat

and

sudo -u gitlab-tools celery -A gitlab_tools.tasks.gitlab_tools worker

seem to be starting the right processes, but they never do anything and I don't know celery well enough to get any further here

[Findus23]

I probably also need to add -S gitlab_tools.celery_beat.schedulers.DatabaseScheduler to the beat process, but there are more things that needs to be updated:

  File "/usr/lib/python3/dist-packages/gitlab_tools/celery_beat/schedulers.py", line 7, in <module>
    from celery.utils.encoding import safe_str
ModuleNotFoundError: No module named 'celery.utils.encoding'

[Findus23]

Last update (even though it is off-topic here): I have now installed the software in a virtualenv, (had to remove the version limitations for psycopg2 and PyYAML for them to compile with Python 3.9 and force flask==1.1.2 and gitdb2==3.0.1), adapted the paths in the systemd files and uwsgi config and now everything works fine

[Salamek]

I have another project with same base code as this ported to celery 5, this is how celery 5 commands looks like:

@command
def celerydev():
    setup_logging('celerydev')
    app = create_app(parse_options(), no_sql=True)
    Logging._setup = True  # Disable Celery from setting up logging, already done in setup_logging().
    with app.app_context():
        hostname = OPTIONS['--name'] if OPTIONS['--name'] else host_format(default_nodename(None))
        worker = celery.Worker(
            hostname=hostname, pool_cls=None, loglevel='WARNING',
            logfile=None,  # node format handled by celery.app.log.setup
            pidfile=node_format(None, hostname),
            statedb=node_format(None, hostname),  # ctx.obj.app.conf.worker_state_db
            no_color=False,
            concurrency=5,
            schedule='/tmp/celery.db',
            beat=True

        )
        worker.start()
        return worker.exitcode

@command
def celerybeat():
    setup_logging('celerybeat')
    app = create_app(parse_options(), no_sql=True)
    Logging._setup = True
    with app.app_context():
        return celery.Beat(
            logfile=None,
            pidfile=OPTIONS['--pid'],
            schedule=OPTIONS['--schedule']
        ).run()

@command
def celeryworker():
    setup_logging('celeryworker{}'.format(OPTIONS['--name']))
    app = create_app(parse_options(), no_sql=True)
    Logging._setup = True
    with app.app_context():
        hostname = OPTIONS['--name'] if OPTIONS['--name'] else host_format(default_nodename(None))
        worker = celery.Worker(
            hostname=hostname, pool_cls=None, loglevel='WARNING',
            logfile=None,  # node format handled by celery.app.log.setup
            pidfile=node_format(None, hostname),
            statedb=node_format(None, hostname),  # ctx.obj.app.conf.worker_state_db
            no_color=False,
            autoscale='10,1',
            without_gossip=True

        )
        worker.start()
        return worker.exitcode

Debian Bullseye is still in testing, so i will not migrate this project yet, but i will when bullseye is released (i will be forced to do it since i will be upgrading too)

[Salamek]

This is solved in version 1.2.3