`migrations.ALLOWED_DOMAINS` config value does not default to allow everything

[mpeter50]

Description

Repositories can not be migrated anymore unless the following lines are present in Gitea's configuration file:

[migrations]
ALLOWED_DOMAINS = *

When this not present, starting a mirror migration from github.com results in this error message being shown:

The documentation's config cheat sheet for v1.22.1 tells that if this config value is undefined, it should not block anything:

ALLOWED_DOMAINS: empty: Domains allowlist for migrating repositories, default is blank. It means everything will be allowed.

I have first noticed this issue in v1.22.0, and the issue is also present after updating to v1.22.1. On an other instance of mine, which is still running v1.21.4, migrations can be created fine, without overriding the default value of the migrations.ALLOWED_DOMAINS.

Gitea Version

v1.22.1

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

https://gist.github.com/mpeter50/8a50ed8f4605ee833b3e7dc552e6eb2b

Screenshots

No response

Git Version

No response

Operating System

No response

How are you running Gitea?

Docker Compose, with a locally built image from unmodified source at versioned tag.

Database

MySQL/MariaDB

[yp05327]

Can you provide more settings in your app.ini? I tested it in local, and there's no problem to migration from Github.

[Dev-Wiki]

我的也是如此,配置如下:

APP_NAME = DevWiki Gitea
RUN_MODE = prod
RUN_USER = git
WORK_PATH = /data/gitea

[repository]
ROOT = /data/git/repositories

[repository.local]
LOCAL_COPY_PATH = /data/gitea/tmp/local-repo

[repository.upload]
TEMP_PATH = /data/gitea/uploads

[server]
APP_DATA_PATH = /data/gitea
DOMAIN = git.devwiki.net
SSH_DOMAIN = git.devwiki.net
HTTP_PORT = 此处打码
ROOT_URL = https://git.devwiki.net/
DISABLE_SSH = false
START_SSH_SERVER = true
SSH_PORT = 22
SSH_LISTEN_PORT = 此处打码
LFS_START_SERVER = true
LFS_JWT_SECRET = 此处打码
OFFLINE_MODE = false
LANDING_PAGE = explore

[lsf]
PATH = /data/git/lfs

[database]
PATH = /data/gitea/gitea.db
DB_TYPE = sqlite3
HOST = localhost:3306
NAME = gitea
USER = root
PASSWD = 
LOG_SQL = false
SCHEMA = 
SSL_MODE = disable
CHARSET = utf8

[indexer]
ISSUE_INDEXER_PATH = /data/gitea/indexers/issues.bleve

[session]
PROVIDER_CONFIG = /data/gitea/sessions
PROVIDER = file

[picture]
AVATAR_UPLOAD_PATH = /data/gitea/avatars
REPOSITORY_AVATAR_UPLOAD_PATH = /data/gitea/repo-avatars
DISABLE_GRAVATAR = false
ENABLE_FEDERATED_AVATAR = true

[attachment]
PATH = /data/gitea/attachments

[log]
MODE = console
LEVEL = info
## ROUTER = console
ROOT_PATH = /data/gitea/log

[security]
INSTALL_LOCK = true
LOGIN_REMEMBER_DAYS = 3
SECRET_KEY = 
REVERSE_PROXY_LIMIT = 1
REVERSE_PROXY_TRUSTED_PROXIES = *
INTERNAL_TOKEN = 此处打码
PASSWORD_HASH_ALGO = 此处打码

[service]
# 禁用注册，启用后只能用管理员添加用户
DISABLE_REGISTRATION = true
# 是否所有页面都必须登录后才可访问
REQUIRE_SIGNIN_VIEW = false
# 是否显示注册按钮
SHOW_REGISTRATION_BUTTON = false
# 启用注册邮件激活，前提是 Mailer 已经启用
REGISTER_EMAIL_CONFIRM = false
# 是否发送工单创建等提醒邮件，需要 Mailer 被激活
ENABLE_NOTIFY_MAIL = false
ALLOW_ONLY_EXTERNAL_REGISTRATION = false
ENABLE_CAPTCHA = true
REQUIRE_CAPTCHA_FOR_LOGIN = true
DEFAULT_KEEP_EMAIL_PRIVATE = true
DEFAULT_ALLOW_CREATE_ORGANIZATION = true
DEFAULT_ENABLE_TIMETRACKING = true
NO_REPLY_ADDRESS = noreply.localhost
RESET_PASSWD_CODE_LIVE_MINUTES = 5
ENABLE_CACHE_AVATAR = true

[migrations]
ALLOWED_DOMAINS = 
ALLOW_LOCALNETWORKS = true
SKIP_TLS_VERIFY = true
BLOCKED_DOMAINS = 

[mailer]
ENABLED = false

[openid]
ENABLE_OPENID_SIGNIN = false
ENABLE_OPENID_SIGNUP = false

[oauth2]
JWT_SECRET = 此处打码

[Dev-Wiki]

经测试,ALLOWED_DOMAINS 设置为空不行, 必须设置具体的域名才可以

[migrations]
ALLOWED_DOMAINS = 
ALLOW_LOCALNETWORKS = true
SKIP_TLS_VERIFY = true
BLOCKED_DOMAINS = 

[mpeter50]

Oh, totally forgot about this, sorry.

This is my app.ini. I have redacted a few things.

https://gist.github.com/mpeter50/ae8b4341f0eac3587e31b5e1c22efdfc

[mpeter50]

@Dev-Wiki are you having the same problem?

[lunny]

I cannot reproduce this on main branch(0d24c9f383255605d68a92cc5f087c3f16a1d735) and release/v1.22 branch(3913ef69d5a9a268bc44ef17a3bb8717ba924627)

[mpeter50]

To be more complete, these are the complete migration settings:

Maybe this is helpful. Not sure why did I cut the issue screenshot that way. Please note that this is a different repo that the one in the issue screenshot.

[mpeter50]

Hmm just did an additional test, and maybe this is happening while the Gitea container does not have internet access, e.g. because of container settings.

Of course its fine that it cannot do the migration without access to the source service. The problem may be that the failure is handled incorrectly.

If you have the capacity, it might be nice to allow creating migrations regardless of accessibility of the source service, but delay it until intenet access is available. It could be useful for when internet is genuinely not accessible for some time, or for when the target server is unstable, so that users may have a slightly easier time rescuing their repos. Not sure how realistic are these scenarios, probably not too much.

[yp05327]

I also cannot reproduce it in main and release 1.22

[Dev-Wiki]

@Dev-Wiki are you having the same problem?

problem resolved!

ini config for this:

[migrations]
ALLOW_LOCALNETWORKS = true
SKIP_TLS_VERIFY = true
ALLOWED_DOMAINS = *
BLOCKED_DOMAINS = 

use not empty! use not empty! use * not empty!

[Dev-Wiki]

I also cannot reproduce it in main and release 1.22

problem resolved!

ini config for this:

[migrations]
ALLOW_LOCALNETWORKS = true
SKIP_TLS_VERIFY = true
ALLOWED_DOMAINS = *
BLOCKED_DOMAINS = 

use not empty! use not empty! use * not empty!

[yp05327]

@mpeter50 has mentioned this in the description

But I can not reproduce it even I left it empty in my app.ini

[Dev-Wiki]

@mpeter50 has mentioned this in the description

But I can not reproduce it even I left it empty in my app.ini

你使用的什么版本? 我是1.22.1, 这样配置没有提示任何错误, 但是我本地重新搭建了一个gitea 使用1.22.0 什么都没配置这个, 也能迁移任意网站, 现在还不清楚这个设置的逻辑. 我这个配置目前只在我自己的nas上测试可以,

另外一种可用的方式为:

[migrations]
ALLOW_LOCALNETWORKS = true
SKIP_TLS_VERIFY = true
ALLOWED_DOMAINS = *.github.com,github.com

[wxiaoguang]

The reason is simple: the sentence Blank means everything will be allowed. is wrong, it is from #13610 (2020-11-29), which is older than the security fix by #17605 & #20016 (2021-11-20, 2022-06-19), since #17605, "empty" means allowing external hosts by default, but some document updating was forgotten at that time.

The document is just outdated and needs to be updated.

[lunny]

The reason is simple: the sentence Blank means everything will be allowed. is wrong, it is from #13610 (2020-11-29), which is older than the security fix by #17605 & #20016 (2021-11-20, 2022-06-19), since #17605, "empty" means allowing external hosts by default, but some document updating was forgotten at that time.

The document is just outdated and needs to be updated.

Yes, see https://gitea.com/gitea/docs/pulls/55 . But from the screenshot, it's migrating from github.com which should be an external address.

[wxiaoguang]

Oh yes, re-read the code again, it is a bug. Will fix it.

[wxiaoguang]

Well, no code problem, it is well covered by a test. Maybe some users are using fake-IP proxies to access GitHub?

[mpeter50]

To add a bit of a clarification, it seems that to me the problem was caused by my Gitea container being configured like this:

The proxy net is a reverse proxy through which I access Gitea, it does not work in the other direction. Normally the container is not given the network that Docker Compose calls as default, and that means Gitea does not have access to the network to which the Docker Host is connected. This is when the discussed error occurs. When I uncomment the default network, and recreate the Gitea container, the migration will work on the next try.

[wxiaoguang]

Well, that's my guess, you are using proxy for external access ......

[mpeter50]

Well, no code problem, it is well covered by a test. Maybe some users are using fake-IP proxies to access GitHub?

I do not know about such happening here. I have an internal DNS server if that counts, but shouldnt modify anything.

[wxiaoguang]

More contexts:

• 20325

• 20328

[mpeter50]

This is the definition of the proxy network from above:

Here it is seen that it is externally defined, outside of the compose file. According to running docker network inspect proxy-internal, the network only hosts a few other services' frontends, and an Nginx, which is only configured as a reverse proxy.

The default network is a network created by docker compose, technically named as gitea_default, according to docker network inspect gitea_default it uses the bridge driver. Only the Gitea container is connected to it.

[mpeter50]

If I can help with diagnosing in any way, let me know.

This is the result of nslookup github.com executed inside the Gitea container when it is not given access to the internet:

sudo docker exec -it gitea bash
fce11f93a4df:/var/lib/gitea$ nslookup github.com
Server:         127.0.0.11
Address:        127.0.0.11:53

** server can't find github.com: SERVFAIL

** server can't find github.com: SERVFAIL

fce11f93a4df:/var/lib/gitea$

This is the result of nslookup github.com executed inside the Gitea container when it is given access to the internet:

sudo docker exec -it gitea bash
cd1b6e0424cd:/var/lib/gitea$ nslookup github.com
Server:         127.0.0.11
Address:        127.0.0.11:53

Non-authoritative answer:

Non-authoritative answer:
Name:   github.com
Address: 140.82.121.4

cd1b6e0424cd:/var/lib/gitea$

[mpeter50]

I was wondering if maybe Gitea would be processing the SERVFAIL response as an empty IP address being resolved or something like that, when trying to check if the migration source is allowed, and detecting that as a not-a-public address?

[wxiaoguang]

Sorry but no more idea from my side ..... maybe it needs a reproducible setup and some code level debugging work, I am not able to provide more support at the moment. (and will leave keyboard for a while)

[mpeter50]

I just noticed that there are related entries in the System Notices:

However, by doing further tries with revoked internet access, these do not appear to be generated when the discussed disallowed host error apepars. I have tried to migrate a new repo several times, every time producing the error, but no new system notices have appeared.

[yp05327]

By the hit from https://github.com/go-gitea/gitea/issues/31733#issuecomment-2297912917 and https://github.com/go-gitea/gitea/issues/31733#issuecomment-2297872720

I added this in my hosts

127.0.0.1 github.com

Then I got the same error.

[GiteaBot]

We close issues that need feedback from the author if there were no new comments for a month. :tea:

[mpeter50]

Can we reopen this issue? I do not know what additional feedback I can give. Though if you ask for something, of course I may be able to give more information.

[wxiaoguang]

I do not know what additional feedback I can give. Though if you ask for something, of course I may be able to give more information.

https://github.com/go-gitea/gitea/issues/31733#issuecomment-2297915772

it needs a reproducible setup (to make sure others could reproduce in their environments)

[mpeter50]

I will try to make a copy of the instance and trim it down to what is needed for this to happen. This will take some time, though.

If anyone else having this issue is reading this, please consider of you can make a reproducer too.

[GiteaBot]

We close issues that need feedback from the author if there were no new comments for a month. :tea:

[mpeter50]

Can we reopen this issue? I did not yet find the time to make a minimal reproducer installation.