search

go-gitea / gitea

Git with a cup of tea! Painless self-hosted all-in-one software development service, including Git hosting, code review, team collaboration, package registry and CI/CD
https://gitea.com
MIT License
44.16k stars 5.41k forks source link

`migrations.ALLOWED_DOMAINS` config value does not default to allow everything #31733

Open mpeter50 opened 1 month ago

mpeter50 commented 1 month ago

Description

Repositories can not be migrated anymore unless the following lines are present in Gitea's configuration file:

[migrations]
ALLOWED_DOMAINS = *

When this not present, starting a mirror migration from github.com results in this error message being shown:

image

The documentation's config cheat sheet for v1.22.1 tells that if this config value is undefined, it should not block anything:

ALLOWED_DOMAINS: empty: Domains allowlist for migrating repositories, default is blank. It means everything will be allowed.

I have first noticed this issue in v1.22.0, and the issue is also present after updating to v1.22.1. On an other instance of mine, which is still running v1.21.4, migrations can be created fine, without overriding the default value of the migrations.ALLOWED_DOMAINS.

Gitea Version

v1.22.1

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

https://gist.github.com/mpeter50/8a50ed8f4605ee833b3e7dc552e6eb2b

Screenshots

No response

Git Version

No response

Operating System

No response

How are you running Gitea?

Docker Compose, with a locally built image from unmodified source at versioned tag.

Database

MySQL/MariaDB

yp05327 commented 1 month ago

Can you provide more settings in your app.ini? I tested it in local, and there's no problem to migration from Github.

Dev-Wiki commented 1 month ago

我的也是如此,配置如下:

APP_NAME = DevWiki Gitea
RUN_MODE = prod
RUN_USER = git
WORK_PATH = /data/gitea

[repository]
ROOT = /data/git/repositories

[repository.local]
LOCAL_COPY_PATH = /data/gitea/tmp/local-repo

[repository.upload]
TEMP_PATH = /data/gitea/uploads

[server]
APP_DATA_PATH = /data/gitea
DOMAIN = git.devwiki.net
SSH_DOMAIN = git.devwiki.net
HTTP_PORT = 此处打码
ROOT_URL = https://git.devwiki.net/
DISABLE_SSH = false
START_SSH_SERVER = true
SSH_PORT = 22
SSH_LISTEN_PORT = 此处打码
LFS_START_SERVER = true
LFS_JWT_SECRET = 此处打码
OFFLINE_MODE = false
LANDING_PAGE = explore

[lsf]
PATH = /data/git/lfs

[database]
PATH = /data/gitea/gitea.db
DB_TYPE = sqlite3
HOST = localhost:3306
NAME = gitea
USER = root
PASSWD = 
LOG_SQL = false
SCHEMA = 
SSL_MODE = disable
CHARSET = utf8

[indexer]
ISSUE_INDEXER_PATH = /data/gitea/indexers/issues.bleve

[session]
PROVIDER_CONFIG = /data/gitea/sessions
PROVIDER = file

[picture]
AVATAR_UPLOAD_PATH = /data/gitea/avatars
REPOSITORY_AVATAR_UPLOAD_PATH = /data/gitea/repo-avatars
DISABLE_GRAVATAR = false
ENABLE_FEDERATED_AVATAR = true

[attachment]
PATH = /data/gitea/attachments

[log]
MODE = console
LEVEL = info
## ROUTER = console
ROOT_PATH = /data/gitea/log

[security]
INSTALL_LOCK = true
LOGIN_REMEMBER_DAYS = 3
SECRET_KEY = 
REVERSE_PROXY_LIMIT = 1
REVERSE_PROXY_TRUSTED_PROXIES = *
INTERNAL_TOKEN = 此处打码
PASSWORD_HASH_ALGO = 此处打码

[service]
# 禁用注册，启用后只能用管理员添加用户
DISABLE_REGISTRATION = true
# 是否所有页面都必须登录后才可访问
REQUIRE_SIGNIN_VIEW = false
# 是否显示注册按钮
SHOW_REGISTRATION_BUTTON = false
# 启用注册邮件激活，前提是 Mailer 已经启用
REGISTER_EMAIL_CONFIRM = false
# 是否发送工单创建等提醒邮件，需要 Mailer 被激活
ENABLE_NOTIFY_MAIL = false
ALLOW_ONLY_EXTERNAL_REGISTRATION = false
ENABLE_CAPTCHA = true
REQUIRE_CAPTCHA_FOR_LOGIN = true
DEFAULT_KEEP_EMAIL_PRIVATE = true
DEFAULT_ALLOW_CREATE_ORGANIZATION = true
DEFAULT_ENABLE_TIMETRACKING = true
NO_REPLY_ADDRESS = noreply.localhost
RESET_PASSWD_CODE_LIVE_MINUTES = 5
ENABLE_CACHE_AVATAR = true

[migrations]
ALLOWED_DOMAINS = 
ALLOW_LOCALNETWORKS = true
SKIP_TLS_VERIFY = true
BLOCKED_DOMAINS = 

[mailer]
ENABLED = false

[openid]
ENABLE_OPENID_SIGNIN = false
ENABLE_OPENID_SIGNUP = false

[oauth2]
JWT_SECRET = 此处打码
Dev-Wiki commented 1 month ago

经测试,ALLOWED_DOMAINS 设置为空不行, 必须设置具体的域名才可以

[migrations]
ALLOWED_DOMAINS = 
ALLOW_LOCALNETWORKS = true
SKIP_TLS_VERIFY = true
BLOCKED_DOMAINS =
mpeter50 commented 1 month ago

Oh, totally forgot about this, sorry.

This is my app.ini. I have redacted a few things.

https://gist.github.com/mpeter50/ae8b4341f0eac3587e31b5e1c22efdfc

mpeter50 commented 1 month ago

@Dev-Wiki are you having the same problem?

lunny commented 4 weeks ago

I cannot reproduce this on main branch(0d24c9f383255605d68a92cc5f087c3f16a1d735) and release/v1.22 branch(3913ef69d5a9a268bc44ef17a3bb8717ba924627)

mpeter50 commented 4 weeks ago

To be more complete, these are the complete migration settings:

image

Maybe this is helpful. Not sure why did I cut the issue screenshot that way. Please note that this is a different repo that the one in the issue screenshot.

mpeter50 commented 4 weeks ago

Hmm just did an additional test, and maybe this is happening while the Gitea container does not have internet access, e.g. because of container settings.

Of course its fine that it cannot do the migration without access to the source service. The problem may be that the failure is handled incorrectly.

If you have the capacity, it might be nice to allow creating migrations regardless of accessibility of the source service, but delay it until intenet access is available. It could be useful for when internet is genuinely not accessible for some time, or for when the target server is unstable, so that users may have a slightly easier time rescuing their repos. Not sure how realistic are these scenarios, probably not too much.

yp05327 commented 4 weeks ago

I also cannot reproduce it in main and release 1.22

Dev-Wiki commented 4 weeks ago

@Dev-Wiki are you having the same problem?

problem resolved!

ini config for this:

[migrations]
ALLOW_LOCALNETWORKS = true
SKIP_TLS_VERIFY = true
ALLOWED_DOMAINS = *
BLOCKED_DOMAINS =

use not empty! use not empty! use * not empty!

Dev-Wiki commented 4 weeks ago

I also cannot reproduce it in main and release 1.22

problem resolved!

ini config for this:

[migrations]
ALLOW_LOCALNETWORKS = true
SKIP_TLS_VERIFY = true
ALLOWED_DOMAINS = *
BLOCKED_DOMAINS =

use not empty! use not empty! use * not empty!

yp05327 commented 4 weeks ago

@mpeter50 has mentioned this in the description

image

But I can not reproduce it even I left it empty in my app.ini

Dev-Wiki commented 4 weeks ago

@mpeter50 has mentioned this in the description

image

But I can not reproduce it even I left it empty in my app.ini

你使用的什么版本? 我是1.22.1, 这样配置没有提示任何错误, 但是我本地重新搭建了一个gitea 使用1.22.0 什么都没配置这个, 也能迁移任意网站, 现在还不清楚这个设置的逻辑. 我这个配置目前只在我自己的nas上测试可以,

另外一种可用的方式为:

[migrations]
ALLOW_LOCALNETWORKS = true
SKIP_TLS_VERIFY = true
ALLOWED_DOMAINS = *.github.com,github.com
wxiaoguang commented 4 weeks ago

The reason is simple: the sentence Blank means everything will be allowed. is wrong, it is from #13610 (2020-11-29), which is older than the security fix by #17605 & #20016 (2021-11-20, 2022-06-19), since #17605, "empty" means allowing external hosts by default, but some document updating was forgotten at that time.

The document is just outdated and needs to be updated.

lunny commented 4 weeks ago

The reason is simple: the sentence Blank means everything will be allowed. is wrong, it is from #13610 (2020-11-29), which is older than the security fix by #17605 & #20016 (2021-11-20, 2022-06-19), since #17605, "empty" means allowing external hosts by default, but some document updating was forgotten at that time.

The document is just outdated and needs to be updated.

Yes, see https://gitea.com/gitea/docs/pulls/55 . But from the screenshot, it's migrating from github.com which should be an external address.

wxiaoguang commented 4 weeks ago

Oh yes, re-read the code again, it is a bug. Will fix it.

wxiaoguang commented 4 weeks ago

Well, no code problem, it is well covered by a test. Maybe some users are using fake-IP proxies to access GitHub?

image

mpeter50 commented 4 weeks ago

To add a bit of a clarification, it seems that to me the problem was caused by my Gitea container being configured like this:

image

The proxy net is a reverse proxy through which I access Gitea, it does not work in the other direction. Normally the container is not given the network that Docker Compose calls as default, and that means Gitea does not have access to the network to which the Docker Host is connected. This is when the discussed error occurs. When I uncomment the default network, and recreate the Gitea container, the migration will work on the next try.

wxiaoguang commented 4 weeks ago

Well, that's my guess, you are using proxy for external access ......

mpeter50 commented 4 weeks ago

Well, no code problem, it is well covered by a test. Maybe some users are using fake-IP proxies to access GitHub?

I do not know about such happening here. I have an internal DNS server if that counts, but shouldnt modify anything.

wxiaoguang commented 4 weeks ago

More contexts:

mpeter50 commented 4 weeks ago

This is the definition of the proxy network from above:

image

Here it is seen that it is externally defined, outside of the compose file. According to running docker network inspect proxy-internal, the network only hosts a few other services' frontends, and an Nginx, which is only configured as a reverse proxy.

The default network is a network created by docker compose, technically named as gitea_default, according to docker network inspect gitea_default it uses the bridge driver. Only the Gitea container is connected to it.

mpeter50 commented 4 weeks ago

If I can help with diagnosing in any way, let me know.

This is the result of nslookup github.com executed inside the Gitea container when it is not given access to the internet:

sudo docker exec -it gitea bash
fce11f93a4df:/var/lib/gitea$ nslookup github.com
Server:         127.0.0.11
Address:        127.0.0.11:53

** server can't find github.com: SERVFAIL

** server can't find github.com: SERVFAIL

fce11f93a4df:/var/lib/gitea$

This is the result of nslookup github.com executed inside the Gitea container when it is given access to the internet:

sudo docker exec -it gitea bash
cd1b6e0424cd:/var/lib/gitea$ nslookup github.com
Server:         127.0.0.11
Address:        127.0.0.11:53

Non-authoritative answer:

Non-authoritative answer:
Name:   github.com
Address: 140.82.121.4

cd1b6e0424cd:/var/lib/gitea$
mpeter50 commented 4 weeks ago

I was wondering if maybe Gitea would be processing the SERVFAIL response as an empty IP address being resolved or something like that, when trying to check if the migration source is allowed, and detecting that as a not-a-public address?

wxiaoguang commented 4 weeks ago

Sorry but no more idea from my side ..... maybe it needs a reproducible setup and some code level debugging work, I am not able to provide more support at the moment. (and will leave keyboard for a while)

mpeter50 commented 4 weeks ago

I just noticed that there are related entries in the System Notices:

image

However, by doing further tries with revoked internet access, these do not appear to be generated when the discussed disallowed host error apepars. I have tried to migrate a new repo several times, every time producing the error, but no new system notices have appeared.

yp05327 commented 4 weeks ago

By the hit from https://github.com/go-gitea/gitea/issues/31733#issuecomment-2297912917 and https://github.com/go-gitea/gitea/issues/31733#issuecomment-2297872720

I added this in my hosts

127.0.0.1 github.com

Then I got the same error.