Closed cr0c0dylus closed 1 year ago
added a fix. Need to wait for a new image.
I installed 3.54 but my files are still not visible neither in the Filer web UI nor via fs.meta.save.
I deployed chrislusf/seaweedfs-csi-driver:v1.1.7, then restarted all the jobs that are associated with the seaweedfs volumes and they appeared in the Filer according to the path value:
parameters {
# Available options: https://github.com/seaweedfs/seaweedfs-csi-driver/blob/master/pkg/driver/mounter_seaweedfs.go
collection = "nomad"
replication = "010"
path = "/buckets/volume_app_content"
}
So now I have /buckets/app_content (old) and /buckets/volume_app_content (new). Should I delete /buckets/app_content manually ?
Not clear how the nomad template works here.
Full volume template (volume_app_content.hcl):
namespace = "app-stage"
id = "app_content"
name = "app_content"
type = "csi"
plugin_id = "seaweedfs"
capability {
access_mode = "multi-node-multi-writer"
attachment_mode = "file-system"
}
parameters {
# Available options: https://github.com/seaweedfs/seaweedfs-csi-driver/blob/master/pkg/driver/mounter_seaweedfs.go
collection = "nomad"
replication = "010"
path = "/buckets/volume_app_content"
}
It was passed to nomad with the command:
nomad volume create volume_app_content.hcl
So, after the 1.1.7 deployment, I didn't make any changes in nomad. Just restarted jobs via UI. And volumes became visible in Filer, in weed mount and in fs.meta.save
But now I have a strange error with fs.meta.save (after it does some work):
error: rpc error: code = Unknown desc = list /buckets/volume_app_content/shop/products/9993/photos : dial tcp 192.168.3.22:33306: connect: cannot assign requested address
33306 - is a maxscale port (MariaDB proxy). There are no other errors in syslog, maxscale log or mariadb log.
-rw-r--r-- 1 root root 82683835 Jul 18 16:53 localhost:8888-20230718-165135.meta
-rw-r--r-- 1 root root 84280725 Jul 18 16:56 localhost:8888-20230718-165516.meta
-rw-r--r-- 1 root root 84108176 Jul 18 19:34 localhost:8888-20230718-193305.meta
It seems like mysql driver has unlimited connections by default. Initial state:
# netstat -ntp|wc -l
262
20 seconds after fs.meta.save:
# netstat -ntp|wc -l
29728
When I updated Filer config to
connection_max_idle = 4
connection_max_open = 100
it started working without errors.
So finally I changed storage from mysql to redis3 (keydb active-active replication) and imported the meta.
Performance measurements:
MariaDB:
# time du -hs buckets
56G buckets
real 0m13.762s
user 0m0.483s
sys 0m3.013s
KeyDB 1-st run after meta load:
56G buckets
real 1m53.215s
user 0m0.555s
sys 0m4.862s
KeyDB 2-nd run after meta load:
56G buckets
real 0m12.589s
user 0m0.396s
sys 0m2.863s
The redis3 speed is almost the same as MariaDB. Is that OK?
BTW, it would be great if -v=false
were set for fs.meta.load by default :)
Thank you for the excellent product!
I decided to exclude possible cache influence and tested again with MariaDB. The first run after Filer restart:
# time du -hs buckets
56G buckets
real 2m44.354s
user 0m0.618s
sys 0m4.762s
2-nd run:
# time du -hs buckets
56G buckets
real 0m12.850s
user 0m0.434s
sys 0m3.032s
So it seems like redis3/KeyDB actually faster than MariaDB (1m53.215s <= 2m44.354s).
It seems like redis have a problem: We have a dump:
# ls -l localhost:8888-20230720-130532.meta
-rw-r--r-- 1 root root 185486297 Jul 20 13:06 localhost:8888-20230720-130532.meta
We loading it to redis (keydb):
> fs.meta.load -v=false localhost:8888-20230720-130532.meta
total 66957 directories, 727763 files
localhost:8888-20230720-130532.meta is loaded.
then we immediately saving it:
> fs.meta.save
total 58026 directories, 628694 files
meta data for http://localhost:8888/ is saved to localhost:8888-20230720-132557.meta
it saves less files and directories that were loaded.
redis2 doesn't have such a problem:
total 66957 directories, 727763 files
localhost:8888-20230720-145411.meta is loaded.
> fs.meta.save
total 66957 directories, 727763 files
meta data for http://localhost:8888/ is saved to localhost:8888-20230720-151526.meta
redis2: fs.meta.load was finished in 7 minutes redis3: fs.meta.load was finished in 54 minutes
second test with redis3 didn't show that problem:
total 66957 directories, 727763 files
localhost:8888-20230720-145411.meta is loaded.
> fs.meta.save
total 66957 directories, 727763 files
meta data for http://localhost:8888/ is saved to localhost:8888-20230720-161653.meta
MySQL also was tested on a clean database - fs.meta.load completed in 1 hour.
So redis2 is the fastest storage.
I can see only volume names but not their contents. Same issue is with weed mount or weed shell fs.meta.save. I use mysql Filer backend and meta data is available for all uploaded files: