search

Loongson-Cloud-Community / dockerfiles

14 stars 14 forks source link

仓库有点太大了，建议优化一下 #212

Closed wojiushixiaobai closed 3 months ago

wojiushixiaobai commented 3 months ago

目前仓库来到了 900M +，完整的拉取一次需要不少时间，能不能将一些二进制文件给放到其他仓库托管构建时通过 wget 在线获取。

$ du -h -d 2 . | grep M | sort -u
1.6M    ./minio
1.6M    ./minio/minio
1.9M    ./kubesphere/fluent-bit
11M     ./kubernetes-sigs/cluster-proportional-autoscaler
125M    ./kubesphere
14M     ./victoriametrics/vmalert
15M     ./victoriametrics/vmagent
18M     ./victoriametrics/victoria-metrics
2.7M    ./library/pulsar-manager
2.9M    ./library/logstash
21M     ./bitnami/redis-cluster
26M     ./prometheus
26M     ./prometheus/prometheus-operator
27M     ./bitnami
27M     ./confluentinc
27M     ./confluentinc/base
3.2M    ./flant
3.2M    ./flant/jq
33M     ./kubernetes
33M     ./kubernetes/etcd
4.4M    ./deepflowio
4.4M    ./deepflowio/deepflow
43M     ./kubesphere/kube-events-ruler
43M     ./loongson
43M     ./loongson/loongnix-server
44M     ./jaegertracing
44M     ./jaegertracing/jaeger-operator
475M    ./.git/objects
476M    ./.git
5.8M    ./bitnami/redis-export
52M     ./kubernetes-sigs/metrics-server
54M     ./victoriametrics
63M     ./kubernetes-sigs
7.0M    ./library
8.6M    ./victoriametrics/vmauth
80M     ./kubesphere/ks-installer
933M    .
qiangxuhui commented 3 months ago

本地可以缩小：

# du -h ./dockerfiles.git/
4.0K    ./dockerfiles.git/refs/pull
4.0K    ./dockerfiles.git/refs/heads
4.0K    ./dockerfiles.git/refs/tags
16K ./dockerfiles.git/refs
4.0K    ./dockerfiles.git/branches
64K ./dockerfiles.git/hooks
119M    ./dockerfiles.git/objects/pack
56K ./dockerfiles.git/objects/info
119M    ./dockerfiles.git/objects
24K ./dockerfiles.git/info
119M    ./dockerfiles.git/

但是无法覆盖远程仓库，因为pr会对于rebase合并的commit存在引用，这些commit在github上的存储是只读的，所以无法删除。

wojiushixiaobai commented 3 months ago

对，就是这样，本地 rebase 处理好后通过强推覆盖掉分支。要项目管理员权限，无法通过创建 PR 解决此问题。

本地可以缩小：

# du -h ./dockerfiles.git/
4.0K  ./dockerfiles.git/refs/pull
4.0K  ./dockerfiles.git/refs/heads
4.0K  ./dockerfiles.git/refs/tags
16K   ./dockerfiles.git/refs
4.0K  ./dockerfiles.git/branches
64K   ./dockerfiles.git/hooks
119M  ./dockerfiles.git/objects/pack
56K   ./dockerfiles.git/objects/info
119M  ./dockerfiles.git/objects
24K   ./dockerfiles.git/info
119M  ./dockerfiles.git/

但是无法覆盖远程仓库，因为pr会对于rebase合并的commit存在引用，这些commit在github上的存储是只读的，所以无法删除。

qiangxuhui commented 3 months ago

我是管理员权限，但是强推会报错，推不上去，这是github的官方回答https://github.com/orgs/community/discussions/73155, 官方的人建议重新建立一个仓库。

wojiushixiaobai commented 3 months ago

git rebase 配合 git push --force-with-lease 就行

如果分支被保护了，可以创建一个不受保护的分支在上面进行验证。

我重新看了下 https://learn.microsoft.com/zh-cn/azure/devops/repos/git/remove-binaries?view=azure-devops 这个文档是 ok 的

wojiushixiaobai commented 3 months ago 
# 注意，不推荐使用 git filter-branch，官方推荐使用 git-filter-repo
git filter-branch --tree-filter "rm -f loongson/loongnix-server/8.4/Loongnix-server-8.4.0.rootfs.loongarch64.tar.gz" -- --all
WARNING: git-filter-branch has a glut of gotchas generating mangled history
         rewrites.  Hit Ctrl-C before proceeding to abort, then use an
         alternative filtering tool such as 'git filter-repo'
         (https://github.com/newren/git-filter-repo/) instead.  See the
         filter-branch manual page for more details; to squelch this warning,
         set FILTER_BRANCH_SQUELCH_WARNING=1.
Proceeding with filter-branch...

以下是使用 git-filter-repo 操作的过程

注意事项：

具体执行过程：

# 以 ubuntu 24.04 操作系统为例，其他操作系统请自行安装 git-filter-repo 即可，python 代码
sudo apt update
sudo apt install git git-filter-repo

cd ~
git clone https://github.com/Loongson-Cloud-Community/dockerfiles

cd ~/dockerfiles

# 找出需要清理的二进制文件，然后通过 `git filter-repo --invert-paths --path filename` 清理即可
# 比如清理 loongson/loongnix-server/8.4/Loongnix-server-8.4.0.rootfs.loongarch64.tar.gz
git filter-repo --invert-paths --path loongson/loongnix-server/8.4/Loongnix-server-8.4.0.rootfs.loongarch64.tar.gz

# 清理其他已经处理的二进制文件
# 最后通过 `git push --force-with-lease` 提交即可
git push --force-with-lease

处于保护状态的分支是无法进行强制推送的，请检查仓库设置的 Branchs 和 Rules 是否有规则保护。

qiangxuhui commented 3 months ago

目前仓库总大小389M， 需要下载的大小为120M，目前只是清理了比较大的可执行文件：

# du -h | sort -rh | head -n 10
389M    .
128M    ./kubesphere
120M    ./.git
119M    ./.git/objects/pack
119M    ./.git/objects
82M ./kubesphere/ks-installer
63M ./kubernetes-sigs
52M ./kubesphere/ks-installer/v3.3.0/roles
52M ./kubesphere/ks-installer/v3.3.0
52M ./kubernetes-sigs/metrics-server/0.5.0
wojiushixiaobai commented 3 months ago

已经比之前好很多了，👍。