friendlyarm / Actions-FriendlyWrt

Build FriendlyWrt using GitHub Actions
296 stars 127 forks source link

R5S Docker 默认 Storage Driver 用的是 vfs 如何修改成overlay2 #3

Closed Ahaxzh closed 1 year ago

Ahaxzh commented 1 year ago

RT. 目前使用的是 R5S-FriendlyWrt-22.03-docker.img.gz 11月1号自动构建的系统。

Docker info 如下:

-----------------------------------------------------
 FriendlyWrt 22.03.2, r19803-9a599fee93
 -----------------------------------------------------
root@R5S:~# docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  compose: Docker Compose (Docker Inc., v2.3.3)

Server:
 Containers: 7
  Running: 7
  Paused: 0
  Stopped: 0
 Images: 7
 Server Version: 20.10.18
 Storage Driver: vfs
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 
 runc version: 
 init version: de40ad0
 Kernel Version: 5.10.110
 Operating System: OpenWrt 22.03.2
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 3.811GiB
 Name: R5S
 ID: SCZU:BFNQ:YXOS:3VD7:BEB3:VYD7:QME7:VT2Y:5NAW:CI4V:KHRY:TBA5
 Docker Root Dir: /opt/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://docker.mirrors.ustc.edu.cn/
 Live Restore Enabled: false

root@R5S:~# 

问题如下: 1、Docker 默认 Storage Driver 用的是 vfs 如何修改成overlay2,目前vfs并不是一个适合生产使用的驱动,会非常的占用空间。

 -----------------------------------------------------
 FriendlyWrt 22.03.2, r19803-9a599fee93
 -----------------------------------------------------
root@R5S:~# du -sh /opt/docker/vfs/dir/*
220K    /opt/docker/vfs/dir/01feb36ff3803e756ffe27e25b260fa4b79f4842f4a0adea3ab7617db2275abd
266M    /opt/docker/vfs/dir/17ea9706b09ee5509aa674c602ee835dfd59bb9c4fa593edf662e2bc921085fe
82M   /opt/docker/vfs/dir/18e732d280f0f92ac16a550ef01810a4a25b7594724d0a5a7e7245d963986ead
354M    /opt/docker/vfs/dir/1a0139678a6b4615e0622f17c9cb6f2c7fbf09dbbe3ae158a064ae956c32ac17
87M   /opt/docker/vfs/dir/1d4afbca0e8ed0b50ed640a83ebc534eb39ff4d0f5bdb49ee3de709222cbdea2
91M   /opt/docker/vfs/dir/1ee0b5126a9308be7839e5950c7286f7a672a76337e686aecb56d8fe7c798e37
120M    /opt/docker/vfs/dir/247575e10e21edd20260b1a9990bd0a1239529c338f35d5ef70141eec1090b8a
5.7M    /opt/docker/vfs/dir/2608b76a970a1b03c9e50bc1f3a15f47e2f1c749dea21c6774d9452a45c3dd88
13M   /opt/docker/vfs/dir/2a89eeefaeea41a8b926a7e95a1fe5b63a93aadc664e7ddf6a42e01fd9d9042b
13M   /opt/docker/vfs/dir/2a89eeefaeea41a8b926a7e95a1fe5b63a93aadc664e7ddf6a42e01fd9d9042b-init
13M   /opt/docker/vfs/dir/2c5f3dd4434dc2e508876b10db409854871f8a085ff461338cb94d5afae69370
354M    /opt/docker/vfs/dir/2edffe91cda0b922c37472cb5c45a7c7891cd40da3c3fab7405f936599f635e4
163M    /opt/docker/vfs/dir/3689d4d121409263ed39bd488f25b48f5792ec3284fab86d99063b0853a35b66
354M    /opt/docker/vfs/dir/3caa8d12bc5b62463c831d4566f0deaa5f0835546288d2669fb191faa659640d
289M    /opt/docker/vfs/dir/40e3ed72ff6426d56fabd7084aee6cc1ac29c4b55e15454ca295d3e0f7d62542
280M    /opt/docker/vfs/dir/427b002f87f08bd6cb4743a0633cf67696db9ba1df010dac22cd272d9792d3d5
117M    /opt/docker/vfs/dir/47b3d2da9571ed65854ec2eca1936d525331af4f8998964e4557d4e02b21a2b1
87M   /opt/docker/vfs/dir/496f9efc8f486844f0549062438c17c0fbf0ffb670a7a85fd093e5a3cfb354e0
5.7M    /opt/docker/vfs/dir/4f8506242b3c097491634f1b22d5bbb3a6e6d05675fc9d08491a5c74a85aa2e8
5.7M    /opt/docker/vfs/dir/4fe697a1bf7d023ca43dd1cfdcb4da81bf3fed40d4c302c6a32f35f31e6b537f
31M   /opt/docker/vfs/dir/51b14d5c5a2929077c7be1ea98ed793b78f9241c5b5b5740985df6f202f60c23
289M    /opt/docker/vfs/dir/52b8faa96b8fafb02963132be8b8a3c0649edd2ee92b0069d8c17832fb2b4a0f
92M   /opt/docker/vfs/dir/57aa3206a95d01daad29df03ab674d9b93387cae0d0ddfb3f810460f311ba55a
266M    /opt/docker/vfs/dir/5a06ba1060a9578203c78a13d2b68d95a00a52e506b0d655fc17d06056cae7c1
87M   /opt/docker/vfs/dir/5a0eb988f62f50e45bc252dc67b324783e1789a3eee942eec229735610633b1f
73M   /opt/docker/vfs/dir/5aa3b22fd2bccab3bf51c9ef1f27928cf914e492ffd6312b0e32fdc260e0c6d1
354M    /opt/docker/vfs/dir/5c8e18628181cfb6391e44f289bf7949b44dd83df2096e3c8163eccf5ed97211
354M    /opt/docker/vfs/dir/5c8e18628181cfb6391e44f289bf7949b44dd83df2096e3c8163eccf5ed97211-init
289M    /opt/docker/vfs/dir/64d040edc283aea0bb83c27136a45e9211f78c7053870da4a470c7440c422240
87M   /opt/docker/vfs/dir/6528447a2b7bed2261562cc729b05fd29556f79dd64a37e30cccf852e94efb50
289M    /opt/docker/vfs/dir/66ec8b4877e30cd69dec4a1d61feb825d0efba2c0050645bed1757a6fad7ef5c
280M    /opt/docker/vfs/dir/67b777d7770f7aca6ddb71976cd63985d9946177c3a5f081dc54ccd0d768189b
13M   /opt/docker/vfs/dir/7512117b7008d5fd9a34e74c8747b9e3dda753f49d2e92bc207e75ffbd4168d3
73M   /opt/docker/vfs/dir/7c59cead923e484374f102c8e518a9655ed8039816431ad9e319d54d1e005478
45M   /opt/docker/vfs/dir/7c99ab2b41b451954354dc88695017936e6f4fd04c7fcd612fbc4c80816664b1
82M   /opt/docker/vfs/dir/7de081ed3d8c07ee867cbc130a24dfa32bf7d6da2e8ba240a34c7c77625b6640
38M   /opt/docker/vfs/dir/821f680f2b074ce29b3aa27fe69372f5ec3f6e129fd3ddfba67d2f49f0bf1a5c
216K    /opt/docker/vfs/dir/84ec410bd99b1fae3f8fa249f7090d9b3aa6345263399cd172db8e90c8fbdf3a
5.7M    /opt/docker/vfs/dir/88e9a044bff868596abe06f3f7844c0b750a9eb7504add447a8115939f731f78
13M   /opt/docker/vfs/dir/8d0fd5dbfc7982ded64ae4549289313b2d5594813d1dfb3e7f57e762a1ae4888
354M    /opt/docker/vfs/dir/9e5cb9ed55ffbd63dcb270daf3cfe015e2fa61cafb70e151ade024ae628aac52
354M    /opt/docker/vfs/dir/a1dfce4d9484c7aaefbe29605408e4ffeb1464bbfbfd3e5f16dd3cfe90b585cd
280M    /opt/docker/vfs/dir/a1e4405c17fff00a23ad503e2f40e64efd60bdcad2d94ddec7194c6485203e94
120M    /opt/docker/vfs/dir/aae70c9c3d7d4d3b563cac5861c84857f93904ca9224a532e5ae1b923a70e994
289M    /opt/docker/vfs/dir/adadf20700fdbdbac15ba565c8b94ca0d722f6598c6949effaf9bea0983571cd
94M   /opt/docker/vfs/dir/b15b0e0fb7521262ca6a92d5b934e9757f10e496f5d439b7051ead6c2ba5ece8
280M    /opt/docker/vfs/dir/b397c99cce404e119ca10dff6a7780c63237b23839ee50769a72c49d0981ed1f
289M    /opt/docker/vfs/dir/b40cffe371ee5b09cc5249bd72a768100be8fdb2ac6d75b47c9e150ac2fb9e84
289M    /opt/docker/vfs/dir/b41530801edb2706d57d9c3aff2fcdb76f9dcfdad32bf7e8d7cf9d145bdc1bee
289M    /opt/docker/vfs/dir/b92e0f6be99c28e344ab3cb1bdbef3c848a675c4f7c036a7b88e6b9a2bfb4b2d
82M   /opt/docker/vfs/dir/bffab546106e8f0920915aa2e21af5c0935cef190c031bd639d71f7f05113795
187M    /opt/docker/vfs/dir/c2abddbd734ae50d8c53b1aec934d7168bbae5ac59fb09dabb4b2788c16b292a
163M    /opt/docker/vfs/dir/c2abddbd734ae50d8c53b1aec934d7168bbae5ac59fb09dabb4b2788c16b292a-init
219M    /opt/docker/vfs/dir/c423e2668f5a502a5afce09a8ae19c512ed2346cd4601ca3693391a381b4c878
87M   /opt/docker/vfs/dir/c423e2668f5a502a5afce09a8ae19c512ed2346cd4601ca3693391a381b4c878-init
289M    /opt/docker/vfs/dir/cb73120a369527e977a92e4c2f697e0d6399117464a660e7ead0696601448c78
289M    /opt/docker/vfs/dir/cb73120a369527e977a92e4c2f697e0d6399117464a660e7ead0696601448c78-init
13M   /opt/docker/vfs/dir/d16f4e68fac73bca2bb7911475eaea3b7ef98cb4f019217507b6f89ea07f8541
13M   /opt/docker/vfs/dir/d1ae55e543d54329e588451128c067d42036091f010dd795908347df5a2c08ff
289M    /opt/docker/vfs/dir/d43ba3f2e84a8e2c6cc4f25c40c334867c0818f8b52ab21683fc98c0cf46c77a
266M    /opt/docker/vfs/dir/d502c0f0c18b5ca31fbb0eb6fb0fe57c037133de89a6dda3ee13b71a3faaddd7
266M    /opt/docker/vfs/dir/d502c0f0c18b5ca31fbb0eb6fb0fe57c037133de89a6dda3ee13b71a3faaddd7-init
4.0K    /opt/docker/vfs/dir/d6ad573e9410facd5b4f22eaf6ec7e6fdb1436a876d124ee2d4a04ec294b99c5-init
280M    /opt/docker/vfs/dir/da6961a945fe5ffb2632c369f542d749013352c7a31e3a56a5e4431661738231
45M   /opt/docker/vfs/dir/e9ef6aa40124e6b299f7857989e022ff1a9c62558b3b64e5fffc4f06f706c79b
45M     /opt/docker/vfs/dir/e9ef6aa40124e6b299f7857989e022ff1a9c62558b3b64e5fffc4f06f706c79b-init
5.7M    /opt/docker/vfs/dir/ee379c357cfaeb1bc6944eb4603f615e5340e73ceb1617a8b1787cddf0d3fb5b
4.0K    /opt/docker/vfs/dir/f483612907bc9514d7d6e5e7afee05e94ea04421b2505ccc1e7eb9af771fa060-init
31M   /opt/docker/vfs/dir/f523a192b92a4346ee9bb86f2ef4fe28f48fc653d9f3f3f62cb2b00ca99c90c4
14M   /opt/docker/vfs/dir/f84446abf227096b94e226134275176bab107a4342968ba32f3e0268ca75569a
119M    /opt/docker/vfs/dir/f9e0a83c79f400215b833ff870219399fd8c5b81fb86d2b3ae2e0d76be9802be
163M    /opt/docker/vfs/dir/fa0d24a770f3939bc0977959d12f19ae7003bedb6cddcd25ab2649fb5114fb01
root@R5S:~# 

2、Docker这个版本默认用的iptables,但是22.03默认用的 nftables,现在系统中会提示混用有问题。 image

查看系统进程可以看到Docker调用的配置文件地址在/tmp/docker/daemon.json ,我没有尝试修改该文件,因为这个位置看起来会觉得是根据/etc/config/dockerd 临时生成的。 image

friendlyarm commented 1 year ago

你可以尝试通过编辑文件 /etc/init.d/dockerd, 让dockerd加载你自已定义的配置文件: /usr/bin/dockerd --config-file="${DOCKERD_CONF}"

Ahaxzh commented 1 year ago

所以依旧无法解决么。我现在是插了一张256G的SD卡在用了,但是早知如此不如买R4S了。

lawrencetg commented 1 year ago

可能是因为我们的系统默认已经使用了overlayfs, 所以docker无法开启overlay2, 试了下可以用以下方法解决:

1) 先禁用系统的 overlayfs, 方法是刷完系统后, 用 fdisk命令删除掉最好一个userdata分区 (默认大小为100M左右),

SD卡的系统, 在电脑上直接操作SD卡即可: sudo fdisk /dev/sdX 命令(输入 m 获取帮助): d 分区号 (1-9, 默认 9): 9 命令(输入 m 获取帮助): w

eMMC的系统, 则在使用eMMC刷机助手刷完系统后, 先不重启, 进入终端, 删除eMMC里的最后一个分区: fdisk /dev/mmcblk2 Command (m for help): d Partition number (1-9, default 9): 9 Partition 9 has been deleted. Command (m for help): w

拨出SD卡, 重新上电, 从eMMC启动系统;

2) 启动系统后, 使用df查看/分区, 显示为/dev/mmcblk2p8说明已经没有在使用overlayfs: root@FriendlyWrt:~# df Filesystem 1K-blocks Used Available Use% Mounted on tmpfs 512 0 512 0% /dev tmpfs 785668 156 785512 1% /run /dev/mmcblk2p8 30039536 552428 27962408 2% / tmpfs 3928332 21856 3906476 1% /tmp cgroup 3928332 0 3928332 0% /sys/fs/cgroup

3) docker配置文件的修改: cp /tmp/dockerd/daemon.json /etc/docker/ 编辑 /etc/docker/daemon.json, 内容如下 { "storage-driver": "overlay2", "data-root": "\/opt\/docker\/", "log-level": "warn", "iptables": true } 编辑 /etc/config/dockerd, 去掉 option alt_config_file '/etc/docker/daemon.json' 前面的注释 重启

4) 重启后用docker info查看, 已经是overlay2了:

root@FriendlyWrt:~# docker info Client: Context: default Debug Mode: false

Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 0 Server Version: 20.10.18 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: false userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: runc version: init version: de40ad0 Kernel Version: 5.10.110 Operating System: OpenWrt 22.03.2 OSType: linux Architecture: aarch64 CPUs: 8 Total Memory: 7.493GiB Name: FriendlyWrt ID: BMXH:2XGN:O233:YAL6:TPST:OVDV:YUZR:V6EM:UIBB:S5LB:DOJC:SOS3 Docker Root Dir: /opt/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

root@FriendlyWrt:~#

Ahaxzh commented 1 year ago

感谢,我试试。建议集成到官方固件本身。