Gitea persistent storage config

[songmeo]

Gitea version: 1.13.1

Description

https://github.com/go-gitea/gitea/issues/14176#issuecomment-752671335

It's not documented that only data partition is persistent and everything under /app/gitea/data is not. This is not only about avatars but also other things like attachments, mail queues, etc. What if people restart their Gitea instances and find out that all data not under /data is lost?

[t-rood]

here, too. easy to showcase by removing the container and adding back in. is it "supported" to add a volume link to /app/gitea/data AFTER the initial setup has completed or can we move the respective locations back to the default mapped volume path using app.ini?

[wxiaoguang]

It sounds like this problem need to be looked into

[lafriks]

It's not supposed to have something in /app/gitea/data. How did it end up there? Everything should be under /data https://github.com/go-gitea/gitea/blob/main/Dockerfile#L59

or under /var/lib/gitea and /etc/gitea for rootless docker: https://github.com/go-gitea/gitea/blob/main/Dockerfile.rootless#L68

[t-rood]

when I upload a custom avatar picture for a user or an organization, they are displayed fine even after container restart. If I then throw away and recreate the docker container, the icons are broken. I use the default folder/volume mappings as shown below, so where are those uploads stored if not on /data mapping?

this is my compose.yml extract:

    volumes:
      - '/volume1/docker/gitea1/data/:/data/'
      - '/volume1/docker/gitea1/conf/app.ini:/data/gitea/conf/app.ini'
      - '/etc/localtime:/etc/localtime:ro'

[lafriks]

What values for paths do you have in app.ini?

[t-rood]

copied from runtime instance right after first initialization, as below:

APP_NAME = xxx
RUN_USER = git
RUN_MODE = prod

[database]
DB_TYPE  = sqlite3
HOST     = 127.0.0.1:3306
NAME     = gitea
USER     = gitea
PASSWD   = 
SCHEMA   = 
SSL_MODE = disable
CHARSET  = utf8
PATH     = /data/gitea/data/gitea.db
LOG_SQL  = false

[repository]
ROOT = /data/gitea/data/gitea-repositories

[server]
SSH_DOMAIN       = git
DOMAIN           = git
HTTP_PORT        = 3000
ROOT_URL         = http://xxx:3000/
DISABLE_SSH      = false
SSH_PORT         = 22
LFS_START_SERVER = true
LFS_CONTENT_PATH = /data/gitea/data/lfs
LFS_JWT_SECRET   = xxx
OFFLINE_MODE     = false
LANDING_PAGE     = login

[mailer]
ENABLED = false

[service]
REGISTER_EMAIL_CONFIRM            = false
ENABLE_NOTIFY_MAIL                = false
DISABLE_REGISTRATION              = false
ALLOW_ONLY_EXTERNAL_REGISTRATION  = false
ENABLE_CAPTCHA                    = false
REQUIRE_SIGNIN_VIEW               = false
DEFAULT_KEEP_EMAIL_PRIVATE        = false
DEFAULT_ALLOW_CREATE_ORGANIZATION = true
DEFAULT_ENABLE_TIMETRACKING       = true
NO_REPLY_ADDRESS                  = noreply.localhost

[picture]
DISABLE_GRAVATAR        = false
ENABLE_FEDERATED_AVATAR = true

[openid]
ENABLE_OPENID_SIGNIN = true
ENABLE_OPENID_SIGNUP = true

[session]
PROVIDER = file

[log]
MODE      = console
LEVEL     = info
ROOT_PATH = /data/gitea/log
ROUTER    = console

[security]
INSTALL_LOCK       = true
INTERNAL_TOKEN     = xxx
PASSWORD_HASH_ALGO = pbkdf2

[lafriks]

Just noticed in issue description it says you are running 1.13.1, that's right? It's quite old one

[lafriks]

You should upgrade to at least 1.15.x or 1.16.0. I think there have been quite a lot changes related to paths as we have quite same bunch of problems due to how historically paths was handled

[t-rood]

I added back to a historic thread, my bad. I'm deploying via :latest tag which displays "1.16.0 built with GNU Make 4.3, go1.17.6"

[t-rood]

and I started with this version. never used any older build. just sayin'. But isn't it "interesting" I'm facing the same historic issue, so maybe a regression?

[lafriks]

I just spent 10 minutes looking for this issue :D it's 2021 jan not 2022 :D

Anyway I can not reproduce this issue. Just deployed new gitea instance with :latest tag and all paths are correct.

I used such stack yaml:

version: '3.5'
services:
  gitea:
    image: gitea/gitea:latest
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - DB_TYPE=postgres
      - DB_HOST=db:5432
      - DB_NAME=gitea
      - DB_USER=gitea
      - DB_PASSWD=gitea
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: '2G'
      restart_policy:
        condition: any
        delay: 5s
      placement:
        constraints: 
          - node.platform.os==linux
          - node.role==manager
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.gitea-test.rule=Host(`test-domain`)"
        - "traefik.http.routers.gitea-test.entrypoints=web-secure"
        - "traefik.http.routers.gitea-test.tls.certresolver=letsencryptresolver"
        - "traefik.http.services.gitea-test.loadbalancer.server.port=3000"
    networks:
      - internal
      - public
    volumes:
      - gitea:/data
    depends_on:
      - db

  db:
    image: postgres:14-alpine
    deploy:
      restart_policy:
        condition: any
        delay: 5s
      placement:
        constraints: 
          - node.platform.os==linux
          - node.role==manager
    environment:
      - POSTGRES_USER=gitea
      - POSTGRES_PASSWORD=gitea
      - POSTGRES_DB=gitea
    networks:
      - internal
    volumes:
      - postgres:/var/lib/postgresql/data

networks:
  internal:
  public:
    external: true

volumes:
  gitea:
  postgres:

[t-rood]

read again please. I didn't claim about app.ini malfunction. I use the above defaults, upload a custom avatar png to both the organization and the user profile, redeploy the container using the previous /data mapping and the avatars are broken. So they obviously are NOT saved into /data, but elsewhere....

[lafriks]

yes, by paths are correct I meant that attachments/avatars etc are saved in /data subdirectories.

[lafriks]

and nothing is lost between container restarts (even user session data)

[t-rood]

if I restart the container, the avatars are still there. if I redeploy the container, they are lost. so this prooves they are etiher not in /data or the linkage to custom avatar pngs is not preserved in /data or database. result is the same: persistency is not preserved when upgrading to a never build or containers get moved around. that's not how docker is designed to be.

[lafriks]

just removed and created new stack and all is working:

[t-rood]

when I do the same, I have broken rectangles around custom avatars. only the build-in ones work fine.

[lafriks]

containers get moved around

do you have docker swarm with multiple hosts? could be that /volume1/docker is not the same on all of them?

also I noticed your app.ini is quite different from one gitea generated for me:

APP_NAME = Gitea: Git with a cup of tea
RUN_MODE = prod
RUN_USER = git

[repository]
ROOT = /data/git/repositories

[repository.local]
LOCAL_COPY_PATH = /data/gitea/tmp/local-repo

[repository.upload]
TEMP_PATH = /data/gitea/uploads

[server]
APP_DATA_PATH    = /data/gitea
DOMAIN           = test-gitea.pico.ninja/
SSH_DOMAIN       = test-gitea.pico.ninja/
HTTP_PORT        = 3000
ROOT_URL         = https://test-gitea.pico.ninja/
DISABLE_SSH      = true
SSH_PORT         = 22
SSH_LISTEN_PORT  = 22
LFS_START_SERVER = true
LFS_CONTENT_PATH = /data/git/lfs
LFS_JWT_SECRET   = xxx
OFFLINE_MODE     = false

[database]
PATH     = /data/gitea/gitea.db
DB_TYPE  = postgres
HOST     = db:5432
NAME     = gitea
USER     = gitea
PASSWD   = gitea
LOG_SQL  = false
SCHEMA   = 
SSL_MODE = disable
CHARSET  = utf8

[indexer]
ISSUE_INDEXER_PATH = /data/gitea/indexers/issues.bleve

[session]
PROVIDER_CONFIG = /data/gitea/sessions
PROVIDER        = file

[picture]
AVATAR_UPLOAD_PATH            = /data/gitea/avatars
REPOSITORY_AVATAR_UPLOAD_PATH = /data/gitea/repo-avatars
DISABLE_GRAVATAR              = false
ENABLE_FEDERATED_AVATAR       = true

[attachment]
PATH = /data/gitea/attachments

[log]
MODE      = console
LEVEL     = info
ROUTER    = console
ROOT_PATH = /data/gitea/log

[security]
INSTALL_LOCK                  = true
SECRET_KEY                    = 
REVERSE_PROXY_LIMIT           = 1
REVERSE_PROXY_TRUSTED_PROXIES = *
INTERNAL_TOKEN                = xxx
PASSWORD_HASH_ALGO            = pbkdf2

[service]
DISABLE_REGISTRATION              = false
REQUIRE_SIGNIN_VIEW               = false
REGISTER_EMAIL_CONFIRM            = false
ENABLE_NOTIFY_MAIL                = false
ALLOW_ONLY_EXTERNAL_REGISTRATION  = false
ENABLE_CAPTCHA                    = false
DEFAULT_KEEP_EMAIL_PRIVATE        = false
DEFAULT_ALLOW_CREATE_ORGANIZATION = true
DEFAULT_ENABLE_TIMETRACKING       = true
NO_REPLY_ADDRESS                  = noreply.localhost

[mailer]
ENABLED = false

[openid]
ENABLE_OPENID_SIGNIN = true
ENABLE_OPENID_SIGNUP = true

[t-rood]

I'm using sqlite3(local) and you postgress but that's it all. I'm using a single-node repro, so the volume is perfectly mounted all the time. otherwise my repositoriers wouldn't show up after redeploying...

[lafriks]

You are also missing picture, attachment sections and quite some other settings (ex. PROVIDER_CONFIG under sessions)

[lafriks]

these missing could be your problem:

[repository.local]
LOCAL_COPY_PATH = /data/gitea/tmp/local-repo

[repository.upload]
TEMP_PATH = /data/gitea/uploads

[indexer]
ISSUE_INDEXER_PATH = /data/gitea/indexers/issues.bleve

[session]
PROVIDER_CONFIG = /data/gitea/sessions
PROVIDER        = file

[picture]
AVATAR_UPLOAD_PATH            = /data/gitea/avatars
REPOSITORY_AVATAR_UPLOAD_PATH = /data/gitea/repo-avatars

[attachment]
PATH = /data/gitea/attachments

[t-rood]

that has been the originally created app.ini before I used the mountpoint so it came right out of initialization on containers first time launch when the gitea has be spawn - it was not a download from "elsewhere" on the internet. when I drill the filesystem of my running container I see the avatars landing in /app/gitea/data/avatars and this is the original post in this thread. so the issue directly comes from the "factory" app.ini. for sure I can take your additional path sections but I'm wondering why the initialization sequence is that much broken.

[t-rood]

and is that "everything" now when it comes to path adjustments as the "default" seems not really been fixed to /data at all? When paths are ommitted, they should not end up in /app/gitea/data....

[t-rood]

actually have these all under /app/gitea/data:

attachments
avatars
indexers
jwt
queues
repo-archive
repo-avatars
sessions

and not all of them are part of your ini, too. when you drill down your running container, is /app/gitea/data empty at all?

[lafriks]

I do not have even /app/gitea/data directory created, there is only /app/gitea/gitea binary.

Mine was also generated by gitea, so it's hard to tell how did you end up with different app.ini than me.

GItea (when it was forked from Gogs years ago) inherited behavior that paths by default are relative to the binary so we have been dealing with that for quite some time and it could be that not all places have been tracked down yet. As we try to keep breaking changes to minimum and it can be hard to maintain that balance.

[t-rood]

from a docker perspective I don't see how can handle the relative paths to gitea binary other than what I observe now. and I can recreate the issue over and over.

thinking what's easier now: fixing each misleading relative path by app.ini [and your ini doesn't cover all folders yet, too, even if your dir tree even doesn't exist] , or adding an additional persistency mount point to the /app/gitea/data directory - so back to what @songmeo pointed out a year ago?

[t-rood]

could it happen because of the sqlite3 localdb driver instead of remotesql with postgres? just guessing.

[lafriks]

All new paths added in latest versions to gitea are not relative to binary but to APP_DATA_PATH. It does not matter sqlite or postgres.

Also I have been running multiple gitea instances in docker and none have been problems with paths in between upgrades (without changing app.ini), so I would say it's stabilized now

[t-rood]

then I'll go for the APP_DATA_PATH, which is also not in my app.ini yet, for whatever reason. but I can change this for sure. ;-) thnx.

[lafriks]

Oh, I missed that somehow. APP_DATA_PATH is actually most important one 😆

[t-rood]

ready for closure. thnx again.

[lafriks]

I will close issue as I can't reproduce issue with not correctly generated app.ini (nor for new, nor with upgrade from older versions)