search

Salamek / gitlab-tools

Web application providing tools not avaiable in GitLab CE (Gitlab push/pull mirror and more)
GNU General Public License v3.0
99 stars 15 forks source link

Periodic sync not triggered #37

Closed ericxliu1990 closed 1 year ago

ericxliu1990 commented 2 years ago

I can manually trigger the sync using the web UI or webhook. But the period sync never triggered. Did I missed something trivial?

Periodic sync: Every minute (/1 *) 

94d5e2f-9610-4957-9089-6c5be5e8f57e sync_pull_mirror    Web hook    SUCCESS None    Jun 30, 2022, 6:16:40 AM    
b5d3d387-e7c2-492b-a688-57b2e0753ed3    sync_pull_mirror    Manual  SUCCESS None    Jun 30, 2022, 6:04:41 AM    
06af6586-90e3-4173-9ee0-ac276dc2b6b4    create_ssh_config   Manual  SUCCESS None    Jun 30, 2022, 5:52:15 AM    
 29ca70fa-52e5-4edb-ad8b-561e3b8c279b   save_pull_mirror    Manual  SUCCESS None    Jun 30, 2022, 5:52:19 AM    
ac0bb692-042a-4b9a-841e-64b8b9ed8e31    sync_pull_mirror    Manual  SUCCESS None    Jun 30, 2022, 5:43:10 AM    
6fe154ee-956f-4bf2-92ee-8396677a95cf    create_ssh_config   Manual  SUCCESS None    Jun 30, 2022, 5:41:41 AM    
 4bf3c0ec-b0c5-4024-b1b5-6aa8e3dd2448   save_pull_mirror    Manual  SUCCESS None    Jun 30, 2022, 5:41:47 AM    
be48cddf-a95b-4348-866f-c211d8f191e4    sync_pull_mirror    Manual  SUCCESS None    Jun 29, 2022, 6:00:21 AM    
2ffc1628-9753-402a-93bf-0656f750ef5e    create_ssh_config   Manual  SUCCESS None    Jun 21, 2022, 10:38:10 PM   
 4952c8b1-b81a-49df-97bf-02a3f7cbde64   save_pull_mirror    Manual  SUCCESS None    Jun 21, 2022, 10:38:19 PM   
gitlab-tools git:(main) kubectl -n gitlab-tools logs -f gitlab-tools-649f54b688-hvl48
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] 00-user-directory.sh: applying...
[fix-attrs.d] 00-user-directory.sh: exited 0.
[fix-attrs.d] 10-ssh-keys.sh: applying...
[fix-attrs.d] 10-ssh-keys.sh: exited 0.
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 00-generate-configuration.sh: executing...
[START] Checking for required variables...
[START] Testing required environment variables...
[FINISH] All required environment variables are in place.
[START] Setting defaults for optional environment variables...
[FINISH] Set some sane-defaults for environment variables.
[START] Generating configuration file...
[cont-init.d] 00-generate-configuration.sh: exited 0.
[cont-init.d] 10-migrate-database.sh: executing...
[START] Migrating the database...
[START] Creating database tables...
D0630 11:16:36.484 3047 cmd.py:870] Popen(['git', 'version'], cwd=/opt/gitlab-tools, universal_newlines=False, shell=None, istream=None)
D0630 11:16:36.490 3047 cmd.py:870] Popen(['git', 'version'], cwd=/opt/gitlab-tools, universal_newlines=False, shell=None, istream=None)
D0630 11:16:36.541 3047 base.py:272] Configuring Raven for host: None
I0630 11:16:36.542 3047 base.py:219] Raven is not configured (logging is disabled). Please see the documentation for more information.
/opt/gitlab-tools/gitlab_tools/bin/gitlab_tools.py:220: SADeprecationWarning: The Engine.table_names() method is deprecated and will be removed in a future release.  Please refer to Inspector.get_table_names(). (deprecated since: 1.4)
  tables_before = set(db.engine.table_names())
/opt/gitlab-tools/gitlab_tools/bin/gitlab_tools.py:222: SADeprecationWarning: The Engine.table_names() method is deprecated and will be removed in a future release.  Please refer to Inspector.get_table_names(). (deprecated since: 1.4)
  tables_after = set(db.engine.table_names())
[START] Migrating the database...
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> 56189bfb2c5f, empty message
Traceback (most recent call last):
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
    self.dialect.do_execute(
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.DuplicateTable: relation "fingerprint" already exists

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/gitlab-tools/manage.py", line 7, in <module>
    main()
  File "/opt/gitlab-tools/gitlab_tools/bin/gitlab_tools.py", line 558, in main
    getattr(command, 'chosen')()  # Execute the function specified by the user.
  File "/opt/gitlab-tools/gitlab_tools/bin/gitlab_tools.py", line 553, in _db
    cli.main(args=sys.argv[1:])
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/flask/cli.py", line 426, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/flask_migrate/cli.py", line 136, in upgrade
    _upgrade(directory, revision, sql, tag, x_arg)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/flask_migrate/__init__.py", line 96, in wrapped
    f(*args, **kwargs)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/flask_migrate/__init__.py", line 271, in upgrade
    command.upgrade(config, revision, sql=sql, tag=tag)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/command.py", line 320, in upgrade
    script.run_env()
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/script/base.py", line 563, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 92, in load_python_file
    module = load_module_py(module_id, path)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 108, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/opt/gitlab-tools/gitlab_tools/migrations/env.py", line 89, in <module>
    run_migrations_online()
  File "/opt/gitlab-tools/gitlab_tools/migrations/env.py", line 81, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/runtime/environment.py", line 851, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/runtime/migration.py", line 620, in run_migrations
    step.migration_fn(**kw)
  File "/opt/gitlab-tools/gitlab_tools/migrations/versions/56189bfb2c5f_.py", line 21, in upgrade
    op.create_table('fingerprint',
  File "<string>", line 8, in create_table
  File "<string>", line 3, in create_table
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/operations/ops.py", line 1254, in create_table
    return operations.invoke(op)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/operations/base.py", line 392, in invoke
    return fn(self, operation)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/operations/toimpl.py", line 114, in create_table
    operations.impl.create_table(table)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/ddl/impl.py", line 352, in create_table
    self._exec(schema.CreateTable(table))
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/ddl/impl.py", line 193, in _exec
    return conn.execute(construct, multiparams)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1306, in execute
    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/sql/ddl.py", line 80, in _execute_on_connection
    return connection._execute_ddl(
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1398, in _execute_ddl
    ret = self._execute_context(
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context
    self._handle_dbapi_exception(
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2043, in _handle_dbapi_exception
    util.raise_(
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
    self.dialect.do_execute(
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.DuplicateTable) relation "fingerprint" already exists

[SQL:
CREATE TABLE fingerprint (
    updated TIMESTAMP WITHOUT TIME ZONE,
    created TIMESTAMP WITHOUT TIME ZONE,
    id SERIAL NOT NULL,
    user_id INTEGER,
    hostname VARCHAR(255),
    sha256_fingerprint VARCHAR(255),
    hashed_hostname VARCHAR(255),
    PRIMARY KEY (id),
    FOREIGN KEY(user_id) REFERENCES "user" (id),
    CONSTRAINT _user_id_hashed_hostname_uc UNIQUE (user_id, hashed_hostname)
)

]
(Background on this error at: https://sqlalche.me/e/14/f405)
[INFO] Migrations are failed or already applied.
[cont-init.d] 10-migrate-database.sh: exited 0.
[cont-init.d] 20-start-services.sh: executing...
-------------------------
[START] Starting Nginx server...
Starting nginx: nginx.
-------------------------
[START] Starting RabbitMQ server...
Starting RabbitMQ Messaging Server: rabbitmq-server.
-------------------------
[START] Starting Redis server...
Starting redis-server: redis-server.
[cont-init.d] 20-start-services.sh: exited 0.
[cont-init.d] done.
[services.d] starting services
[START] Starting the celerybeat...
[START] Starting uwsgi...
[START] Starting the celeryworker...
[services.d] done.
[app] [uWSGI] getting INI configuration from /etc/uwsgi/gitlab-tools.ini
[app] *** Starting uWSGI 2.0.19.1-debian (64bit) on [Thu Jun 30 11:16:44 2022] ***
[app] compiled with version: 10.2.1 20210110 on 11 June 2021 09:08:33
[app] os: Linux-5.10.0-15-amd64 #1 SMP Debian 5.10.120-1 (2022-06-09)
[app] nodename: gitlab-tools-649f54b688-hvl48
[app] machine: x86_64
[app] clock source: unix
[app] pcre jit disabled
[app] detected number of CPU cores: 2
[app] current working directory: /opt/gitlab-tools
[app] detected binary path: /usr/bin/uwsgi-core
[app] chdir() to /opt/gitlab-tools
[app] your memory page size is 4096 bytes
[app] detected max file descriptor number: 1048576
[app] lock engine: pthread robust mutexes
[app] thunder lock: enabled
[app] uwsgi socket 0 bound to UNIX address /tmp/gitlab-tools.sock fd 3
[app] Python version: 3.9.2 (default, Feb 28 2021, 17:03:44)  [GCC 10.2.1 20210110]
[app] PEP 405 virtualenv detected: /opt/gitlab-tools/venv
[app] Set PythonHome to /opt/gitlab-tools/venv
[app] Python main interpreter initialized at 0x56251c670780
[app] python threads support enabled
[app] your server socket listen backlog is limited to 100 connections
[app] your mercy for graceful operations on workers is 60 seconds
[app] mapped 203216 bytes (198 KB) for 1 cores
[app] *** Operational MODE: single process ***
[app] WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x56251c670780 pid: 3438 (default app)
[app] spawned uWSGI master process (pid: 3438)
[app] spawned uWSGI worker 1 (pid: 3455, cores: 1)
[bgtask] celery beat v5.0.2 (singularity) is starting.
[bgtask] __    -    ... __   -        _
[bgtask] LocalTime -> 2022-06-30 11:16:46
[bgtask] Configuration ->
[bgtask]     . broker -> amqp://guest:**@127.0.0.1:5672//
[bgtask]     . loader -> celery.loaders.app.AppLoader
[bgtask]     . scheduler -> celery.beat.PersistentScheduler
[bgtask]     . db -> /home/service/celery_beat.db
[bgtask]     . logfile -> [stderr]@%WARNING
[bgtask]     . maxinterval -> 5.00 minutes (300s)
[bgtask] W0630 11:16:46.060 3442 beat.py:564] DB Reset: Account for new __version__ field
[worker]
[worker]  -------------- celery@1 v5.0.2 (singularity)
[worker] --- ***** -----
[worker] -- ******* ---- Linux-5.10.0-15-amd64-x86_64-with-glibc2.31 2022-06-30 11:16:46
[worker] - *** --- * ---
[worker] - ** ---------- [config]
[worker] - ** ---------- .> app:         gitlab_tools.application:0x7f3970dbc7f0
[worker] - ** ---------- .> transport:   amqp://guest:**@127.0.0.1:5672//
[worker] - ** ---------- .> results:     postgresql://gitlab-tools:**@postgresql13.postgresql13.svc.cluster.local/gitlab-tools
[worker] - *** --- * --- .> concurrency: {min=1, max=10} (prefork)
[worker] -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
[worker] --- ***** -----
[worker]  -------------- [queues]
[worker]                 .> celery           exchange=celery(direct) key=celery
[worker]
[worker]
[worker] W0630 11:16:46.194 3446 warnings.py:109] /opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/app/utils.py:204: CDeprecationWarning:
[worker]     The 'CELERY_RESULT_BACKEND' setting is deprecated and scheduled for removal in
[worker]     version 6.0.0. Use the result_backend instead
[worker]
[worker]   deprecated.warn(description=f'The {setting!r} setting',
[worker]
[worker] W0630 11:16:46.194 3446 worker.py:156] Please run `celery upgrade settings path/to/settings.py` to avoid these warnings and to allow a smoother upgrade to Celery 6.0.
Salamek commented 2 years ago

When database is created a db stamp command should be called to stamp database version for migrations to know from where to where migrate... that should be handled by gitlab-tools post_install --config_prod --user=gitlab-tools command...

And for periodic sync to work, you need to have celery_beat service running

Salamek commented 2 years ago

@cenk1cenk2 maybe docker issue?^

ericxliu1990 commented 2 years ago

Thanks Slamek for the quick response. Is the issue due to inappropriate DB init in docker? If so, should I move this issue to https://github.com/cenk1cenk2/docker-gitlab-tools?

As the log suggested, celerybeat service was running. I also can confirm celerybeat and celeryworker running in the same docker container. 

(venv) root@gitlab-tools-649f54b688-44llp:/opt/gitlab-tools# ps aux  | grep [c]elery
service     3439  0.0  1.0 115660 85224 ?        S    13:39   0:00 python manage.py celerybeat --config_prod --schedule=/home/service/celery_beat.db --pid=/tmp/celerybeat.pid
service     3443  0.0  1.0 116312 86460 ?        S    13:39   0:01 python manage.py celeryworker --config_prod
service     3459  0.0  0.9 115380 75940 ?        S    13:39   0:00 python manage.py celeryworker --config_prod
Salamek commented 2 years ago

@ericxliu1990 well looks like that migration crash is issue in docker version since i could not find any code running db stamp after database init and that migration crash can be related to scheduled tasks not working (IDK) we need to get those migrations working first and applied correctly....

cenk1cenk2 commented 2 years ago

Hey, I run the migrations on any start-up but I guess it would only be applied correctly if the database is fresh, I did not do any stamping for the migrations version, I thought something like that should be used at the end of the migration.

Where the database initiation and initial migrations happen is in this https://github.com/cenk1cenk2/docker-gitlab-tools/blob/main/hostfs/etc/cont-init.d/10-migrate-database.sh script. What should we change about it to make it proper?

Salamek commented 2 years ago

@cenk1cenk2 after db create_all there should be db stamp (?db stamp head?) command as in https://github.com/Salamek/gitlab-tools/blob/d48892e53309a2d8ddf798f1fb79db1cf9ae0a4e/gitlab_tools/bin/gitlab_tools.py#L288 That marks DB with current last migration UUID so next migration can know from where to start migrating from...

Salamek commented 2 years ago

@ericxliu1990 as for you, you need to insert correct UUIDs intoalembic_version table to match your previous version installation, after that db migration should work ok. You can check existing uuids in git by filtering by your previous version tag and looking into gitlab_tools/migrations dir

cenk1cenk2 commented 2 years ago

Thanks for the heads up @Salamek, sorry for missing that.

Now the migrations are not failing after the stamping procedure. The thing is that I left the bash script as is so it is still too okay for migrations to fail. So existing users can upgrade and it will stamp it for them and after that, it will stop failing.

But yeah, anybody can pull the newer version now and give it a try, if it also solves the problem of cron jobs.

Salamek commented 2 years ago

@ericxliu1990 i have looked into migrations folder and all migrations are ~4y old, so to fix your migration crash, just call python manage.py db stamp head

After this crash is solved, we can move to the periodic tasks not working...

Salamek commented 1 year ago

There were some incorrect imports in scheduler implementation, should be fixed in https://github.com/Salamek/gitlab-tools/commit/ccf0fc8ffc209a5fc32535d085ec9e2d1e3af4d8