search

openvstorage / volumedriver

The Open vStorage VolumeDriver is the core of the Open vStorage solution: a high performance distributed block layer. It converts block storage into objects (Storage Container Objects).
Other
37 stars 18 forks source link

Vpool not mount after some days #275

Closed hoannv46 closed 7 years ago

hoannv46 commented 7 years ago

HI all. I create a cluster on openvstorage fargo. after some days i don't work with openvstorage, vpool not mount. But process vpool still work. Thanks. Hoan

jeroenmaelbrancke commented 7 years ago

Hi Hoan,

Is it possible to send us the volumedriver log files?

regards, Jeroen M

wimpers commented 7 years ago

hi @hoanhdo,

any update about the logs? Is the issue solved?

Kind regards, Wim Provoost Product Manager Open vStorage

hoannv46 commented 7 years ago

Sorry, i am busy. This is my log file log_volumedriver.txt Thanks.

jeroenmaelbrancke commented 7 years ago

Hi @hoanhdo,

Can you run following command and give me the new logfile (correct me if i use the wrong vpool name). I need more information about the volumedriver, your current log file is only for 1 minute.

journalctl -u ovs-volumedriver_vpooldemo.service > test.txt

Previous logfile start and end time.

-- Logs begin at Tue 2017-03-07 09:16:48 ICT, end at Tue 2017-03-07 09:17:40 ICT. --

Regards, Jeroen M

hoannv46 commented 7 years ago

Yes. This is my log test.txt Thanks

jeroenmaelbrancke commented 7 years ago

What is the output from cat /proc/mounts and df -h? The command i provided you should give me all the logging from the volumedriver but i only see 2 hours. How is your journalctl configured (cat /etc/systemd/journald.conf)?

Regards, Jeroen M

hoannv46 commented 7 years ago

I just start machine. This is my disk info. disk.txt

jeroenmaelbrancke commented 7 years ago

@hoanhdo

Can you go to the ovs gui --> Administration --> Support --> Remote access enabled = true --> Save settings.

Can you send me the root credentials + cluster id ( section Support information) to jeroen.maelbrancke@openvstorage.com?

i'll try to login on your environment and have a look.

Regards, Jeroen M

jeroenmaelbrancke commented 7 years ago

Hi @hoanhdo,

After taking a look at your environment we would like to install gdb and the matching debug symbols. Is this okay for you?

We see in the volumedriver that restarting a volume hangs but we do not know where, that's why we would like to install the gdb and debug symbols.

Journalctl also skipped some messages from the volumedriver but we found the logs in /var/log/syslog. That is why the logging you sent earlier was only for 2 hours.

In the logging we saw that some processes from the volumedriver stopped logging. Gdb would help us here why those processes are hanging.

A second concern are the disks. This server is a virtual machine on VMware and we see some performance issues on some disk especially sda .

root@openvstorage:~# hdparm -tT /dev/sdb

/dev/sdb:
 Timing cached reads:   17934 MB in  2.00 seconds = 8971.98 MB/sec
 Timing buffered disk reads: 1646 MB in  3.00 seconds = 548.24 MB/sec
root@openvstorage:~# hdparm -tT /dev/sdc

/dev/sdc:
 Timing cached reads:   15730 MB in  2.00 seconds = 7869.87 MB/sec
 Timing buffered disk reads: 1156 MB in  3.00 seconds = 385.24 MB/sec
root@openvstorage:~# hdparm -tT /dev/sdd

/dev/sdd:
 Timing cached reads:   14976 MB in  2.00 seconds = 7491.70 MB/sec
 Timing buffered disk reads: 3992 MB in  3.00 seconds = 1329.50 MB/sec
root@openvstorage:~# hdparm -tT /dev/sde

/dev/sde:
 Timing cached reads:   14682 MB in  2.00 seconds = 7345.97 MB/sec
 Timing buffered disk reads: 4832 MB in  3.00 seconds = 1610.22 MB/sec
root@openvstorage:~# hdparm -tT /dev/sda

/dev/sda:
 Timing cached reads:   13536 MB in  2.00 seconds = 6770.91 MB/sec
 Timing buffered disk reads: 250 MB in  3.04 seconds =  82.26 MB/sec

Before doing anything on the second concern can you please give us a go to install the necessarily packages?

Kind regards, Jeroen M

hoannv46 commented 7 years ago

Hi jeroenmaelbrancke. What package i need install : gdb, volumedriver-no-dedup-base-dbgsym and volumedriver-no-dedup-server-dbgsym?

jeroenmaelbrancke commented 7 years ago

Is it okay that we install the package?

hoannv46 commented 7 years ago

I installed these packages. You can install any packages to debug.

jeroenmaelbrancke commented 7 years ago

@redlicha voldrv-hang-evidence.tar.gz

jeroenmaelbrancke commented 7 years ago

@redlicha If you have some time please do not forget to check the coredumps of this ticket.

redlicha commented 7 years ago

Inspection of the gdb log / core dump shows a number of threads waiting for a condition variable internal to RocksDB (one of the threads being the main thread trying to restart a volume). At the same time the expected internal RocksDB threads (rocksdb:bgX) are not present - reason is unclear and the attached volumedriver log indicates dropped (rate-limited?) messages (seq num jumps from 0x301 to 0x4b1, 32 seconds timestamp gap):

Mar 07 09:17:06 openvstorage volumedriver_fs.sh[2362]: 2017-03-07 09:17:05 907224 +0700 - openvstorage - 2362/0x00007f0b61ed7900 - volumedriverfs/RocksLogger - 0000000000000301 - info - /mnt/hdd1/vpooldemo_db_mds_1:              Options.max_background_compactions: 0
Mar 07 09:17:38 openvstorage volumedriver_fs.sh[2362]: 2017-03-07 09:17:38 799546 +0700 - openvstorage - 2362/0x00007f0b3affd700 - volumedriverfs/MetaDataServerTable - 00000000000004b1 - info - work_: 2e6e5d69-acc8-4939-8332-d2b6de9f4a27: running periodic action

@wimpers : since this was never reproduced / has never been reported again, I suggest to close this ticket as "not reproducible" and to reopen it if necessary.

wimpers commented 7 years ago

See comment from @redlicha , please re-open if we can reproduce or if we have more evidence.