search

go-gitea / gitea

Git with a cup of tea! Painless self-hosted all-in-one software development service, including Git hosting, code review, team collaboration, package registry and CI/CD
https://gitea.com
MIT License
44k stars 5.4k forks source link

support HA mode for redis & postgres #16017

Closed visteras closed 3 years ago

visteras commented 3 years ago

Description

I used Kubernetes+Postgres+Redis as managed from Yandex.Cloud, but when i try start helm chart i get few problems: 1) If i used transaction mode for postgres bouncer - i see error because this mode not support ( https://blog.bullgare.com/2019/06/pgbouncer-and-prepared-statements/ ) 2) i use redis sentinel, and if i used cache/redis i see next error:

GetCommitsCount: ERR AUTH <password> called without any password configured for the default user. Are you sure your configuration is correct?

And... why? I think if cache not work this reason send error message to log and already return normal result, but return 500 error?... P.S. My redis password regexp: 0-9a-Z!, i send this because i dont known really reason why redis not work..., but without sentinel when i run gitea with redis(only a-Z password) and this work!

zeripath commented 3 years ago

what is your connection string for redis? (suitably sanitised)

visteras commented 3 years ago

redis host:

redis://:<passwd>@<host>:<port>/<pool_id>?pool_size=100&idle_timeout=180s
visteras commented 3 years ago

P.S. Example connect in documentation managed redis: Load cert:

mkdir -p /usr/local/share/ca-certificates/Yandex && \
wget "https://storage.yandexcloud.net/cloud-certs/CA.pem" -O /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt

example connect:

package main

import (
    "github.com/go-redis/redis/v7"
)

func main() {
    client := redis.NewUniversalClient(
        &redis.UniversalOptions{
            Addrs:      []string{"<host>:<port>"},
            MasterName: "redis",
            Password:   "<password>",
            DB:         0,
            ReadOnly:   false,
        },
    )
    err := client.Set("foo", "bar", 0).Err()
    if err != nil {
        panic(err)
    }
}
zeripath commented 3 years ago

From modules/nosql/redis.go:

// A RedisURI matches the pattern:
//
// redis://[username:password@]host[:port][/database][?[option=value]*]
// rediss://[username:password@]host[:port][/database][?[option=value]*]
// redis+socket://[username:password@]path[/database][?[option=value]*]
// redis+sentinel://[password@]host1 [: port1][, host2 [:port2]][, hostN [:portN]][/ database][?[option=value]*]
// redis+cluster://[password@]host1 [: port1][, host2 [:port2]][, hostN [:portN]][/ database][?[option=value]*]
//
zeripath commented 3 years ago

I think it would be useful to see your app.ini (suitably sanitised) of course. I was hoping that you would have provided this when I asked for the redis url.

visteras commented 3 years ago

I tryed use verified-ca for postgres, but... not work, only if requred work with pg, and transaction mode not work too( And if i use redis-sentinel variant i see:

2021/05/30 10:28:20 ...ules/context/repo.go:845:func1() [E] GetCommitsCount: redis: all sentinels specified in configuration are unreachable
<IP>:0 - visteras [30/May/2021:10:28:20 &#43;0000] "GET /lib/queue HTTP/1.1" 500 20303 "\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"
2021/05/30 10:28:20 Completed GET /lib/queue 500 Internal Server Error in 196.852738ms

Current config:

    APP_NAME: Visteras&Friends
    RUN_MODE: prod
    RUN_USER: git

    repository:
      ROOT: /data/repos

    repository.local:
      LOCAL_COPY_PATH: /data/gitea/tmp/local-repo

    repository.upload:
      TEMP_PATH: /data/gitea/uploads

    server:
      APP_DATA_PATH: /data/gitea
      SSH_DOMAIN: git.example.com
      HTTP_PORT: 3000
      ROOT_URL: https://git.example.com/
      DISABLE_SSH: false
      START_SSH_SERVER: false
      # SSH_LISTEN_PORT: 2200
      SSH_PORT: 22
      LFS_CONTENT_PATH: /data/gitea/lfs
      DOMAIN: git.example.com
      LFS_START_SERVER: true
      LFS_JWT_SECRET: <LFS_JWT_SECRET>
      OFFLINE_MODE: false

    database:
      DB_TYPE: postgres
      HOST: <hostpg>:<portpg>
      NAME: gitea
      USER: gitea
      PASSWD: <PASSWD>
      LOG_SQL: false
      SSL_MODE: require

    cache:
      ENABLED: true
      ADAPTER: memory
#      HOST: redis://:<passwd>@<host>:<port>/<pool_id>?pool_size=100&idle_timeout=180s
#      HOST: redis+sentinel://<passwd>@<host>:<port>/<pool_id>?pool_size=100&idle_timeout=180s
      ITEM_TTL: 16h

    admin:
      DEFAULT_EMAIL_NOTIFICATIONS: enabled
    indexer:
      ISSUE_INDEXER_PATH: /data/gitea/indexers/issues.bleve

    session:
      PROVIDER_CONFIG: /data/gitea/sessions
      PROVIDER: file

    picture:
      AVATAR_UPLOAD_PATH: /data/gitea/avatars
      DISABLE_GRAVATAR: false
      ENABLE_FEDERATED_AVATAR: true

    attachment:
      PATH: /data/gitea/attachments

    log:
      MODE: console
      LEVEL: Info
      STACKTRACE_LEVEL: None
      ROUTER_LOG_LEVEL: Info
      ROUTER: console
      ENABLE_ACCESS_LOG: true
      ACCESS: console
      ENABLE_XORM_LOG: false

    security:
      PASSWORD_COMPLEXITY: spec
      INSTALL_LOCK: true
      SECRET_KEY: <SECRET_KEY>
      INTERNAL_TOKEN: <INTERNAL_TOKEN>
      LOGIN_REMEMBER_DAYS: 7
      REVERSE_PROXY_LIMIT: 1
      REVERSE_PROXY_TRUSTED_PROXIES: 10.0.0.0/8
      DISABLE_GIT_HOOKS: true

    service:
      DISABLE_REGISTRATION: true
      REQUIRE_SIGNIN_VIEW: true
      REGISTER_EMAIL_CONFIRM: true
      ENABLE_NOTIFY_MAIL: true
      ALLOW_ONLY_EXTERNAL_REGISTRATION: false
      ENABLE_CAPTCHA: true
      DEFAULT_KEEP_EMAIL_PRIVATE: false
      DEFAULT_ALLOW_CREATE_ORGANIZATION: true
      DEFAULT_ENABLE_TIMETRACKING: true
      NO_REPLY_ADDRESS: noreply.example.org
      ENABLE_USER_HEATMAP: true
      ENABLE_TIMETRACKING: true

    mailer:
      ENABLED: true
      FROM: <FROM>
      USER: <USER>
      PASSWD: <PASSWD>
      HOST: <HOST>

    openid:
      ENABLE_OPENID_SIGNIN: false
      ENABLE_OPENID_SIGNUP: false

    oauth2:
      JWT_SECRET: <oauth2>
visteras commented 3 years ago

P.S. If i use redis+cluster i see:

2021/05/30 10:39:56 ...ules/context/repo.go:845:func1() [E] GetCommitsCount: ERR AUTH <password> called without any password configured for the default user. Are you sure your configuration is correct?
zeripath commented 3 years ago

OK, the redis urls are interpreted here:

https://github.com/go-gitea/gitea/blob/effad26c0e7348d27f7fdd1c0cbf120d7558cf67/modules/nosql/manager_redis.go#L42-L205

I see you have:

redis://:<passwd>@<host>:<port>/<pool_id>?pool_size=100&idle_timeout=180s

But you state you want MasterName set:

client := redis.NewUniversalClient(
        &redis.UniversalOptions{
            Addrs:      []string{"<host>:<port>"},
            MasterName: "redis",
            Password:   "<password>",
            DB:         0,
            ReadOnly:   false,
        },
    )

That would require you to set the master_name parameter e.g.:

redis://:<passwd>@<host>:<port>/<pool_id>?pool_size=100&idle_timeout=180s&master_name=redis

Otherwise if you're sure that the redis connection strings are right and you've got the configuration of the server right, the place to look at is in that function and I would guess adding some opportune log.Trace(...) to 144-204 would be helpful.

visteras commented 3 years ago

HOST: redis+sentinel://<passwd>@<host>:<port>/<pool_id>?pool_size=100&idle_timeout=180s&master_name=redis work for me, i think this can be adding to documentation for other users, thanks! Any ideas with pg bouncer?

zeripath commented 3 years ago

We would need to add binary_parameters=yes to the pq connection string.

zeripath commented 3 years ago

https://github.com/go-gitea/gitea/blob/7a484c0788f14412bd0531fdd021b4b292145bc5/modules/setting/database.go#L160-L170 would need to be adjusted to add it.

zeripath commented 3 years ago

Actually I think you can just change the [database] NAME to include ?binary_parameters=yes at the end.

zeripath commented 3 years ago

In which case feel free to put up a PR for updating the docs.

visteras commented 3 years ago

I'll check this for a few days, but for now I'm trying to figure out why my push actions are not showing up in the event list ...

zeripath commented 3 years ago

Resynchronize hooks

zeripath commented 3 years ago

Ensure your repositories are not mounted noexec

silverwind commented 3 years ago

FWIW it is possible and recommended to have multiple redis sentinels in a HA setup, I'm not sure if that is supported as well via a single URL.

zeripath commented 3 years ago

@silverwind when I wrote the common nosql support I made it so we support redis+sentinel://[:password@]host:port[,host2:port2,...][/service_name[/db]][?param1=value1[&param2=value=2&...]] scheme which maps to a: redis.NewFailoverClient(opts.Failover()). We also support redis+cluster:// which maps to: redis.NewClusterClient(opts.Cluster()) or redis:// which maps to: redis.NewClient(opts.Simple())

You can use:

See https://github.com/go-gitea/gitea/blob/main/modules/nosql/manager_redis.go

visteras commented 3 years ago

Actually I think you can just change the [database] NAME to include ?binary_parameters=yes at the end.

Thx for help, this worked for me