NVIDIA / aistore

AIStore: scalable storage for AI applications
https://aistore.nvidia.com
MIT License
1.21k stars 160 forks source link

Error: fs=/dev/mapper/mpathfh has no disks #130

Closed Hastyrush closed 1 year ago

Hastyrush commented 1 year ago

Hi, I set up the minimal production docker using the guide from https://github.com/NVIDIA/aistore/blob/master/deploy/prod/docker/single/README.md

However, I am facing an issue when trying to deploy the docker using docker run -d -p 51080:51080 --name AIStore -v /mnt/disk16:/ais/ILSVRC

The fatal error encountered is mp[/ais/ILSVRC, fs=/dev/mapper/mpathfh] has no disks Attached is a screenshot of the error image

My setup is using the Seagate 5U84 storage enclosure, with 2 disks mapped to 1 logical volume through 4 initiators, hence the 'mpath' storage type, mounted at /mnt/disk16. My question is, I can use the /mnt/disk as any normal storage just fine on the host machine, but it seems that AIStore is not recognizing it as a mounted disk. Is there any way to solve this issue?

I have attached the output of lsblk as reference.

Thanks in advance! lsblk.txt

alex-aizman commented 1 year ago

It's likely a parsing bug but I'd need lsblk -Jt output to figure it out. Generally, the code is trying to match the "name" field (see example below) with the value that in your case would be "mpathfh". Not finding it.

...
            },{
               "name": "nvme0n1p3",
               "alignment": 0,
               "min-io": 512,
               "opt-io": 0,
               "phy-sec": 512,
               "log-sec": 512,
               "rota": false,
               "sched": "none",
               "rq-size": 1023,
               "ra": 128,
               "wsame": "0B"
            },{
...
Hastyrush commented 1 year ago

Hello, I'm not sure if this is an issue with my Linux distro (Centos 7), but the lsblk command has no -J (json) option. Are there any alternatives to retrieving the name field? Thanks!

image image

alex-aizman commented 1 year ago

versus

(-J must work)

Hastyrush commented 1 year ago

Thanks for the information. I guess till someone adds support for the -J option for Centos 7, this issue can't be resolved directly. Will be closing this, thank you!