Files uploaded to a CSI volume are not visible in Filer

[cr0c0dylus]

I can see only volume names but not their contents. Same issue is with weed mount or weed shell fs.meta.save. I use mysql Filer backend and meta data is available for all uploaded files:

MariaDB [seaweedfs]> select count(*) from filemeta \G
*************************** 1. row ***************************
count(*): 795832

> fs.meta.save
total 8 directories, 0 files
meta data for http://localhost:8888/ is saved to localhost:8888-20230707-184657.meta

job "seaweedfs-csi" {
  type = "system"
  datacenters = ["fsn1"]
  priority = 90

  group "nodes" {
    ephemeral_disk {
      migrate = false
      size    = 1100
      sticky  = false
    }

    task "plugin" {
      driver = "docker"

      config {
        image = "chrislusf/seaweedfs-csi-driver:latest"
        force_pull = "true"
        network_mode = "host"
        args = [
          "--endpoint=unix://csi/csi.sock",
          "--filer=seaweedfs-filer.service.consul:8888",
          "--nodeid=${node.unique.name}",
          "--cacheCapacityMB=1024",
          "--cacheDir=${NOMAD_ALLOC_DIR}/cache_dir",
        ]
        privileged = true
      }

      csi_plugin {
        id        = "seaweedfs"
        type      = "monolith"
        mount_dir = "/csi"
      }

      resources {
        cpu    = 1000
        memory = 512
        memory_max = 3072
      }
    }
  }
}

[chrislusf]

added a fix. Need to wait for a new image.

[cr0c0dylus]

I installed 3.54 but my files are still not visible neither in the Filer web UI nor via fs.meta.save.

[cr0c0dylus]

I deployed chrislusf/seaweedfs-csi-driver:v1.1.7, then restarted all the jobs that are associated with the seaweedfs volumes and they appeared in the Filer according to the path value:

parameters {
  # Available options: https://github.com/seaweedfs/seaweedfs-csi-driver/blob/master/pkg/driver/mounter_seaweedfs.go
  collection = "nomad"
  replication = "010"
  path = "/buckets/volume_app_content"
}

So now I have /buckets/app_content (old) and /buckets/volume_app_content (new). Should I delete /buckets/app_content manually ?

[chrislusf]

Not clear how the nomad template works here.

[cr0c0dylus]

Full volume template (volume_app_content.hcl):

namespace = "app-stage"

id = "app_content"
name = "app_content"

type = "csi"

plugin_id = "seaweedfs"

capability {
  access_mode     = "multi-node-multi-writer"
  attachment_mode = "file-system"
}

parameters {
  # Available options: https://github.com/seaweedfs/seaweedfs-csi-driver/blob/master/pkg/driver/mounter_seaweedfs.go
  collection = "nomad"
  replication = "010"
  path = "/buckets/volume_app_content"
}

It was passed to nomad with the command:

nomad volume create volume_app_content.hcl

So, after the 1.1.7 deployment, I didn't make any changes in nomad. Just restarted jobs via UI. And volumes became visible in Filer, in weed mount and in fs.meta.save

But now I have a strange error with fs.meta.save (after it does some work):

error: rpc error: code = Unknown desc = list /buckets/volume_app_content/shop/products/9993/photos : dial tcp 192.168.3.22:33306: connect: cannot assign requested address

33306 - is a maxscale port (MariaDB proxy). There are no other errors in syslog, maxscale log or mariadb log.

-rw-r--r--  1 root root 82683835 Jul 18 16:53 localhost:8888-20230718-165135.meta
-rw-r--r--  1 root root 84280725 Jul 18 16:56 localhost:8888-20230718-165516.meta
-rw-r--r--  1 root root 84108176 Jul 18 19:34 localhost:8888-20230718-193305.meta

[cr0c0dylus]

It seems like mysql driver has unlimited connections by default. Initial state:

# netstat -ntp|wc -l
262

20 seconds after fs.meta.save:

# netstat -ntp|wc -l
29728

When I updated Filer config to

connection_max_idle = 4
connection_max_open = 100

it started working without errors.

[cr0c0dylus]

So finally I changed storage from mysql to redis3 (keydb active-active replication) and imported the meta.

Performance measurements:

MariaDB:

# time du -hs buckets 
56G     buckets

real    0m13.762s
user    0m0.483s
sys     0m3.013s

KeyDB 1-st run after meta load:

56G     buckets

real    1m53.215s
user    0m0.555s
sys     0m4.862s

KeyDB 2-nd run after meta load:

56G     buckets

real    0m12.589s
user    0m0.396s
sys     0m2.863s

The redis3 speed is almost the same as MariaDB. Is that OK?

BTW, it would be great if -v=false were set for fs.meta.load by default :)

Thank you for the excellent product!

[cr0c0dylus]

I decided to exclude possible cache influence and tested again with MariaDB. The first run after Filer restart:

# time du -hs buckets 
56G     buckets

real    2m44.354s
user    0m0.618s
sys     0m4.762s

2-nd run:

# time du -hs buckets 
56G     buckets

real    0m12.850s
user    0m0.434s
sys     0m3.032s

So it seems like redis3/KeyDB actually faster than MariaDB (1m53.215s <= 2m44.354s).

[cr0c0dylus]

It seems like redis have a problem: We have a dump:

# ls -l localhost:8888-20230720-130532.meta
-rw-r--r-- 1 root root 185486297 Jul 20 13:06 localhost:8888-20230720-130532.meta

We loading it to redis (keydb):

> fs.meta.load -v=false localhost:8888-20230720-130532.meta

total 66957 directories, 727763 files
localhost:8888-20230720-130532.meta is loaded.

then we immediately saving it:

> fs.meta.save
total 58026 directories, 628694 files
meta data for http://localhost:8888/ is saved to localhost:8888-20230720-132557.meta

it saves less files and directories that were loaded.

[cr0c0dylus]

redis2 doesn't have such a problem:

total 66957 directories, 727763 files
localhost:8888-20230720-145411.meta is loaded.
> fs.meta.save
total 66957 directories, 727763 files
meta data for http://localhost:8888/ is saved to localhost:8888-20230720-151526.meta

redis2: fs.meta.load was finished in 7 minutes redis3: fs.meta.load was finished in 54 minutes

second test with redis3 didn't show that problem:

total 66957 directories, 727763 files
localhost:8888-20230720-145411.meta is loaded.
> fs.meta.save
total 66957 directories, 727763 files
meta data for http://localhost:8888/ is saved to localhost:8888-20230720-161653.meta

MySQL also was tested on a clean database - fs.meta.load completed in 1 hour.

So redis2 is the fastest storage.