happyfish100 / fastdfs

FastDFS is an open source high performance distributed file system (DFS). It's major functions include: file storing, file syncing and file accessing, and design for high capacity and load balance. Wechat/Weixin public account (Chinese Language): fastdfs
GNU General Public License v3.0
8.91k stars 1.97k forks source link

reserved_storage_space 在多storage_path的情况下是每一个storage_path单独计算吗? #676

Closed ql2023 closed 5 months ago

ql2023 commented 7 months ago

FastDFS 版本 V6.10

4个storage_path的其中三个,分别是M01, M02, M03

/mapper/vg_lv_1  7.0G  5.7G  1.4G  81% /nfs1
/mapper/vg_lv_2   15G  5.7G  9.4G  38% /nfs2
/mapper/vg_lv_3  8.0G  5.7G  2.4G  71% /nfs3

配置:

reserved_storage_space = 20%

循环上传情况:

i=0 id=group1/M00/00/03/wKglnGVkD0CAdbQNAAAABNKBnuw183.txt
i=1 id=group1/M02/00/03/wKglnGVkD0CAQ8DpAAAABNKBnuw079.txt
i=2 id=group1/M03/00/03/wKglnGVkD0CAEHLcAAAABNKBnuw424.txt
i=3 id=group1/M00/00/03/wKglnGVkD0CAY0WcAAAABNKBnuw046.txt
i=4 id=group1/M02/00/03/wKglnGVkD0CABUcvAAAABNKBnuw697.txt
i=5 id=group1/M03/00/03/wKglnGVkD0CAVoo4AAAABNKBnuw033.txt

配置比例的情况下,会跳过M01,将reserved_storage_space调整位容量

reserved_storage_space = 2g

循环上传情况:

i=0 id=group1/M00/00/03/wKglnGVkEb6AZwDjAAAABNKBnuw586.txt
i=1 id=group1/M01/00/03/wKglnGVkEb6AGixMAAAABNKBnuw324.txt
i=2 id=group1/M02/00/03/wKglnGVkEb6AOVB9AAAABNKBnuw856.txt
i=3 id=group1/M03/00/03/wKglnGVkEb6AYyjqAAAABNKBnuw574.txt
i=4 id=group1/M00/00/03/wKglnGVkEb6AR1WsAAAABNKBnuw476.txt
i=5 id=group1/M01/00/03/wKglnGVkEb6AcqZ0AAAABNKBnuw420.txt

不会跳过M01,上传更大的文件测试:

i=0 id=group1/M00/00/03/wKglnGVkEuCAQpEqEdsfwJtJqy0114.mp4
i=1 id=group1/M01/00/03/wKglnGVkEuOACs0xEdsfwJtJqy0583.mp4
i=2 id=group1/M02/00/03/wKglnGVkEuaARYSXEdsfwJtJqy0822.mp4
i=3 id=group1/M03/00/03/wKglnGVkEumAaT9mEdsfwJtJqy0567.mp4

也不会跳过M01,最终容量:

/mapper/vg_lv_1  7.0G  5.9G  1.1G  85% /nfs1
/mapper/vg_lv_2   15G  5.9G  9.1G  40% /nfs2
/mapper/vg_lv_3  8.0G  5.9G  2.1G  74% /nfs3

配置比例会跳过,配置容量不会跳过?

ql2023 commented 7 months ago

tracker.conf 配置

store_path = 2
reserved_storage_space = 2g

存储从

/mapper/vg_lv_1  7.0G  5.9G  1.1G  85% /nfs1
/mapper/vg_lv_2   15G  5.9G  9.1G  40% /nfs2
/mapper/vg_lv_3  8.0G  5.9G  2.1G  74% /nfs3

写到

/mapper/vg_lv_1  7.0G  5.9G  1.1G  85% /nfs1
/mapper/vg_lv_2   15G   15G  176M  99% /nfs2
/mapper/vg_lv_3  8.0G  5.9G  2.1G  74% /nfs3

最后

i=10    id=group1/M01/00/04/wKglnGVkS8-AdNvVEdsfwJtJqy0929.mp4
i=11    id=group1/M01/00/04/wKglnGVkS9OAAaMoEdsfwJtJqy0893.mp4
close socket[192.168.37.156:23000] error ,emsg:Connection reset by peer: socket write error
java.net.SocketException: Connection reset by peer: socket write error
ql2023 commented 7 months ago

tracker.conf配置

store_path = 2
reserved_storage_space = 20%

第一次测试:

存储从

/mapper/vg_lv_1  7.0G  5.9G   7.0G  1% /nfs1
/mapper/vg_lv_2   15G  5.9G 15.0G  1% /nfs2
/mapper/vg_lv_3  8.0G  5.9G   8.0G  1% /nfs3

写到

/mapper/vg_lv_1  7.0G  5.9G   7.0G    1% /nfs1
/mapper/vg_lv_2   15G  5.9G   7.5G  51% /nfs2
/mapper/vg_lv_3  8.0G  5.9G   8.0G    1% /nfs3

所有文件写入到M01中,

第二次测试:

/mapper/vg_lv_1  7.0G  5.9G   7.0G    1% /nfs1
/mapper/vg_lv_2   15G  5.9G   7.5G  51% /nfs2
/mapper/vg_lv_3  8.0G  5.9G   8.0G    1% /nfs3

----

/mapper/vg_lv_1  7.0G  5.9G   7.0G    1% /nfs1
/mapper/vg_lv_2   15G  5.9G   6.6G  57% /nfs2
/mapper/vg_lv_3  8.0G  5.9G   6.9G    4% /nfs3

----

/mapper/vg_lv_1  7.0G  5.9G   7.0G    1% /nfs1
/mapper/vg_lv_2   15G  5.9G   6.6G  57% /nfs2
/mapper/vg_lv_3  8.0G  5.9G   2.4G   71% /nfs3

先写了M01,之后写了M02,M01没有到达20%阈值(57%)

第三次测试:

/mapper/vg_lv_1  7.0G  5.9G   7.0G    1% /nfs1
/mapper/vg_lv_2   15G  5.9G   6.6G  57% /nfs2
/mapper/vg_lv_3  8.0G  5.9G   2.4G  71% /nfs3

---

/mapper/vg_lv_1  7.0G  5.9G   6.5G      9% /nfs1
/mapper/vg_lv_2   15G  5.9G   1.6G    90% /nfs2
/mapper/vg_lv_3  8.0G  5.9G   147M  99% /nfs3

先写了部分M00,然后M01,最后M02(写到最后异常):

error ,emsg:Connection reset by peer: socket write error
happyfish100 commented 7 months ago

reserved_storage_space 配置为具体的预留空间,是总的预留空间。 上述例子中,预留空间2G会分摊到 单个存储路径,即单个存储路径的预留空间为 2GB / 3 = 682 MB。

happyfish100 commented 7 months ago

另外,storage server报告磁盘空间的时间间隔(配置项 stat_report_interval)默认为 300秒,配置示例中配置为 60秒,可以酌情调小一些。

ql2023 commented 7 months ago

再找时间测试一下。

ql2023 commented 6 months ago

比例配置是否有总体限制? tracker_server.conf 配置 5%:

reserved_storage_space = 5%

实际磁盘使用空间:

容量    已用    可用  已用%
7.3T      7.3T    34G   100%
7.3T      7.3T    34G   100%
 16T      15T  632G     96%
 16T      15T  780G     96%
5.9T      4.9T  994G    84%
 16T      15T  421G     98%
 16T      15T  376G     98%
5.9T     4.9T 1004G    84%

总容量:约93.4T,剩余可用约:4275G。 实际上不管tracker_server.conf配置store_path=0还是store_path=2,都无法上传文件。从配置5%的阈值看,上面的6个path仍有两个84%的未达到5%的阈值。

使用java-client,简单的获取StorageServer也报 errno=28:

StorageServer storageServer = trackerClient.getStoreStorage(trackerServer);

org.csource.fastdfs.TrackerClient#getStoreStorage(org.csource.fastdfs.TrackerServer)

happyfish100 commented 6 months ago

比例配置是否有总体限制? tracker_server.conf 配置 5%:

reserved_storage_space = 5%

实际磁盘使用空间:

容量    已用    可用  已用%
7.3T      7.3T    34G   100%
7.3T      7.3T    34G   100%
 16T      15T  632G     96%
 16T      15T  780G     96%
5.9T      4.9T  994G    84%
 16T      15T  421G     98%
 16T      15T  376G     98%
5.9T     4.9T 1004G    84%

总容量:约93.4T,剩余可用约:4275G。 实际上不管tracker_server.conf配置store_path=0还是store_path=2,都无法上传文件。从配置5%的阈值看,上面的6个path仍有两个84%的未达到5%的阈值。

使用java-client,简单的获取StorageServer也报 errno=28:

StorageServer storageServer = trackerClient.getStoreStorage(trackerServer);

org.csource.fastdfs.TrackerClient#getStoreStorage(org.csource.fastdfs.TrackerServer)

是的,目前是按总体空间使用情况计算的。