Samba shares are not persistent over restart or upgrade

[tobiashochguertel]

1. I created an samba share "share-backup".
2. I've got an e-mail from Anacron with notification about an package upgrade, which was not succesfully because of missing public-keys in package management system (yum).
3. I done the "yum upgrade" manually over ssh and answered the import questions about public-keys with "yes".
4. I tought I would be a great idea to restart the machine, so apply all upgrade package.

After these steps I recognized that my mounted CIFS (samba) share in Windows (10) wasn't not longer available - and reconnecting failed. I checked samba shares via web-ui and there was not longer my share "share-backup" listed.

It could be that the samba share isn't persistent in the database or after upgrade previous persist shares are removed?

E-Mail from Anarcron (bullet-point number 2):

/etc/cron.daily/0yum-daily.cron:

warning: /var/cache/yum/x86_64/$releasever/updates/packages/NetworkManager-team-1.4.0-17.el7_3.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY Public key for NetworkManager-team-1.4.0-17.el7_3.x86_64.rpm is not installed
warning: /var/cache/yum/x86_64/$releasever/epel/packages/epel-release-7-9.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY Public key for epel-release-7-9.noarch.rpm is not installed
warning: /var/cache/yum/x86_64/$releasever/rockstor/packages/rockstor-release-3-8.16.el7.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 08795b09: NOKEY Public key for rockstor-release-3-8.16.el7.x86_64.rpm is not installed Importing GPG key 0xF4A80EB5:
 Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"
 Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 Package    : rockstor-release-3-8.10.el7.x86_64 (@anaconda/3)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
The following updates will be applied on rockstor.hochguertel.local:
================================================================================
 Package                     Arch    Version                    Repository
                                                                           Size ================================================================================
Updating:
 NetworkManager              x86_64  1:1.4.0-17.el7_3           updates   2.5 M
 NetworkManager-glib         x86_64  1:1.4.0-17.el7_3           updates   385 k
 NetworkManager-libnm        x86_64  1:1.4.0-17.el7_3           updates   443 k
 NetworkManager-team         x86_64  1:1.4.0-17.el7_3           updates   147 k
 NetworkManager-tui          x86_64  1:1.4.0-17.el7_3           updates   223 k
 audit                       x86_64  2.6.5-3.el7_3.1            updates   233 k
 audit-libs                  x86_64  2.6.5-3.el7_3.1            updates    85 k
 audit-libs-python           x86_64  2.6.5-3.el7_3.1            updates    70 k
 bind-libs-lite              x86_64  32:9.9.4-38.el7_3.2        updates   730 k
 bind-license                noarch  32:9.9.4-38.el7_3.2        updates    83 k
 device-mapper               x86_64  7:1.02.135-1.el7_3.3       updates   269 k
 device-mapper-libs          x86_64  7:1.02.135-1.el7_3.3       updates   333 k
 epel-release                noarch  7-9                        epel       14 k
 firewalld                   noarch  0.4.3.2-8.1.el7_3.2        updates   386 k
 firewalld-filesystem        noarch  0.4.3.2-8.1.el7_3.2        updates    44 k
 kpartx                      x86_64  0.4.9-99.el7_3.1           updates    67 k
 libgudev1                   x86_64  219-30.el7_3.7             updates    76 k
 libnl3                      x86_64  3.2.28-3.el7_3             updates   278 k
 libnl3-cli                  x86_64  3.2.28-3.el7_3             updates   160 k
 libpciaccess                x86_64  0.13.4-3.el7_3             updates    26 k
 libsemanage                 x86_64  2.5-5.1.el7_3              updates   144 k
 libsemanage-python          x86_64  2.5-5.1.el7_3              updates   104 k
 libsmbclient                x86_64  4.4.4-12.el7_3             updates   126 k
 libtiff                     x86_64  4.0.3-27.el7_3             updates   170 k
 libwbclient                 x86_64  4.4.4-12.el7_3             updates   100 k
 microcode_ctl               x86_64  2:2.1-16.3.el7_3           updates   745 k
 nfs-utils                   x86_64  1:1.3.0-0.33.el7_3         updates   377 k
 ntp                         x86_64  4.2.6p5-25.el7.centos.1    updates   547 k
 ntpdate                     x86_64  4.2.6p5-25.el7.centos.1    updates    85 k
 openssh                     x86_64  6.6.1p1-33.el7_3           updates   438 k
 openssh-clients             x86_64  6.6.1p1-33.el7_3           updates   642 k
 openssh-server              x86_64  6.6.1p1-33.el7_3           updates   440 k
 openssl                     x86_64  1:1.0.1e-60.el7_3.1        updates   713 k
 openssl-libs                x86_64  1:1.0.1e-60.el7_3.1        updates   959 k
 policycoreutils             x86_64  2.5-11.el7_3               updates   841 k
 policycoreutils-python      x86_64  2.5-11.el7_3               updates   445 k
 polkit                      x86_64  0.112-11.el7_3             updates   167 k
 python-firewall             noarch  0.4.3.2-8.1.el7_3.2        updates   301 k
 rockstor-release            x86_64  3-8.16.el7                 rockstor   22 k
 samba                       x86_64  4.4.4-12.el7_3             updates   610 k
 samba-client                x86_64  4.4.4-12.el7_3             updates   547 k
 samba-client-libs           x86_64  4.4.4-12.el7_3             updates   4.6 M
 samba-common                noarch  4.4.4-12.el7_3             updates   191 k
 samba-common-libs           x86_64  4.4.4-12.el7_3             updates   161 k
 samba-common-tools          x86_64  4.4.4-12.el7_3             updates   451 k
 samba-libs                  x86_64  4.4.4-12.el7_3             updates   260 k
 samba-winbind               x86_64  4.4.4-12.el7_3             updates   485 k
 samba-winbind-clients       x86_64  4.4.4-12.el7_3             updates   124 k
 samba-winbind-krb5-locator  x86_64  4.4.4-12.el7_3             updates    79 k
 samba-winbind-modules       x86_64  4.4.4-12.el7_3             updates   108 k
 selinux-policy              noarch  3.13.1-102.el7_3.15        updates   414 k
 selinux-policy-targeted     noarch  3.13.1-102.el7_3.15        updates   6.4 M
 systemd                     x86_64  219-30.el7_3.7             updates   5.2 M
 systemd-libs                x86_64  219-30.el7_3.7             updates   369 k
 systemd-sysv                x86_64  219-30.el7_3.7             updates    63 k
 vim-minimal                 x86_64  2:7.4.160-1.el7_3.1        updates   436 k
 wpa_supplicant              x86_64  1:2.0-21.el7_3             updates   788 k
 xfsprogs                    x86_64  4.5.0-9.el7_3              updates   895 k

Transaction Summary
================================================================================
Upgrade  58 Packages
Updates failed to install with the following error message: 
Didn't install any keys

Output of the manual yum upgrade via ssh (bullet-point number 3)

  Verifying  : 1:NetworkManager-1.4.0-13.el7_3.x86_64                                                                                                                                                                                                                   72/116
  Verifying  : openssh-6.6.1p1-31.el7.x86_64                                                                                                                                                                                                                            73/116
  Verifying  : 1:NetworkManager-team-1.4.0-13.el7_3.x86_64                                                                                                                                                                                                              74/116
  Verifying  : rockstor-release-3-8.10.el7.x86_64                                                                                                                                                                                                                       75/116
  Verifying  : systemd-libs-219-30.el7_3.6.x86_64                                                                                                                                                                                                                       76/116
  Verifying  : xfsprogs-4.5.0-8.el7.x86_64                                                                                                                                                                                                                              77/116
  Verifying  : firewalld-0.4.3.2-8.el7.noarch                                                                                                                                                                                                                           78/116
  Verifying  : 32:bind-license-9.9.4-38.el7_3.noarch                                                                                                                                                                                                                    79/116
  Verifying  : samba-winbind-4.4.4-9.el7.x86_64                                                                                                                                                                                                                         80/116
  Verifying  : libnl3-3.2.28-2.el7.x86_64                                                                                                                                                                                                                               81/116
  Verifying  : audit-libs-python-2.6.5-3.el7.x86_64                                                                                                                                                                                                                     82/116
  Verifying  : 1:NetworkManager-glib-1.4.0-13.el7_3.x86_64                                                                                                                                                                                                              83/116
  Verifying  : 32:bind-libs-lite-9.9.4-38.el7_3.x86_64                                                                                                                                                                                                                  84/116
  Verifying  : selinux-policy-targeted-3.13.1-102.el7_3.7.noarch                                                                                                                                                                                                        85/116
  Verifying  : samba-common-4.4.4-9.el7.noarch                                                                                                                                                                                                                          86/116
  Verifying  : samba-libs-4.4.4-9.el7.x86_64                                                                                                                                                                                                                            87/116
  Verifying  : ntpdate-4.2.6p5-25.el7.centos.x86_64                                                                                                                                                                                                                     88/116
  Verifying  : samba-client-libs-4.4.4-9.el7.x86_64                                                                                                                                                                                                                     89/116
  Verifying  : samba-common-libs-4.4.4-9.el7.x86_64                                                                                                                                                                                                                     90/116
  Verifying  : samba-winbind-modules-4.4.4-9.el7.x86_64                                                                                                                                                                                                                 91/116
  Verifying  : polkit-0.112-9.el7.x86_64                                                                                                                                                                                                                                92/116
  Verifying  : ntp-4.2.6p5-25.el7.centos.x86_64                                                                                                                                                                                                                         93/116
  Verifying  : 2:vim-minimal-7.4.160-1.el7.x86_64                                                                                                                                                                                                                       94/116
  Verifying  : 1:NetworkManager-tui-1.4.0-13.el7_3.x86_64                                                                                                                                                                                                               95/116
  Verifying  : samba-winbind-krb5-locator-4.4.4-9.el7.x86_64                                                                                                                                                                                                            96/116
  Verifying  : openssh-clients-6.6.1p1-31.el7.x86_64                                                                                                                                                                                                                    97/116
  Verifying  : 7:device-mapper-libs-1.02.135-1.el7_3.1.x86_64                                                                                                                                                                                                           98/116
  Verifying  : systemd-sysv-219-30.el7_3.6.x86_64                                                                                                                                                                                                                       99/116
  Verifying  : samba-client-4.4.4-9.el7.x86_64                                                                                                                                                                                                                         100/116
  Verifying  : kpartx-0.4.9-99.el7.x86_64                                                                                                                                                                                                                              101/116
  Verifying  : 7:device-mapper-1.02.135-1.el7_3.1.x86_64                                                                                                                                                                                                               102/116
  Verifying  : libpciaccess-0.13.4-2.el7.x86_64                                                                                                                                                                                                                        103/116
  Verifying  : python-firewall-0.4.3.2-8.el7.noarch                                                                                                                                                                                                                    104/116
  Verifying  : libwbclient-4.4.4-9.el7.x86_64                                                                                                                                                                                                                          105/116
  Verifying  : 1:nfs-utils-1.3.0-0.33.el7.x86_64                                                                                                                                                                                                                       106/116
  Verifying  : policycoreutils-python-2.5-9.el7.x86_64                                                                                                                                                                                                                 107/116
  Verifying  : policycoreutils-2.5-9.el7.x86_64                                                                                                                                                                                                                        108/116
  Verifying  : audit-2.6.5-3.el7.x86_64                                                                                                                                                                                                                                109/116
  Verifying  : epel-release-7-8.noarch                                                                                                                                                                                                                                 110/116
  Verifying  : samba-common-tools-4.4.4-9.el7.x86_64                                                                                                                                                                                                                   111/116
  Verifying  : libtiff-4.0.3-25.el7_2.x86_64                                                                                                                                                                                                                           112/116
  Verifying  : 1:NetworkManager-libnm-1.4.0-13.el7_3.x86_64                                                                                                                                                                                                            113/116
  Verifying  : libgudev1-219-30.el7_3.6.x86_64                                                                                                                                                                                                                         114/116
  Verifying  : 2:microcode_ctl-2.1-16.el7.x86_64                                                                                                                                                                                                                       115/116
  Verifying  : libsemanage-2.5-4.el7.x86_64                                                                                                                                                                                                                            116/116

Updated:
  NetworkManager.x86_64 1:1.4.0-17.el7_3            NetworkManager-glib.x86_64 1:1.4.0-17.el7_3               NetworkManager-libnm.x86_64 1:1.4.0-17.el7_3       NetworkManager-team.x86_64 1:1.4.0-17.el7_3             NetworkManager-tui.x86_64 1:1.4.0-17.el7_3
  audit.x86_64 0:2.6.5-3.el7_3.1                    audit-libs.x86_64 0:2.6.5-3.el7_3.1                       audit-libs-python.x86_64 0:2.6.5-3.el7_3.1         bind-libs-lite.x86_64 32:9.9.4-38.el7_3.2               bind-license.noarch 32:9.9.4-38.el7_3.2
  device-mapper.x86_64 7:1.02.135-1.el7_3.3         device-mapper-libs.x86_64 7:1.02.135-1.el7_3.3            epel-release.noarch 0:7-9                          firewalld.noarch 0:0.4.3.2-8.1.el7_3.2                  firewalld-filesystem.noarch 0:0.4.3.2-8.1.el7_3.2
  kpartx.x86_64 0:0.4.9-99.el7_3.1                  libgudev1.x86_64 0:219-30.el7_3.7                         libnl3.x86_64 0:3.2.28-3.el7_3                     libnl3-cli.x86_64 0:3.2.28-3.el7_3                      libpciaccess.x86_64 0:0.13.4-3.el7_3
  libsemanage.x86_64 0:2.5-5.1.el7_3                libsemanage-python.x86_64 0:2.5-5.1.el7_3                 libsmbclient.x86_64 0:4.4.4-12.el7_3               libtiff.x86_64 0:4.0.3-27.el7_3                         libwbclient.x86_64 0:4.4.4-12.el7_3
  microcode_ctl.x86_64 2:2.1-16.3.el7_3             nfs-utils.x86_64 1:1.3.0-0.33.el7_3                       ntp.x86_64 0:4.2.6p5-25.el7.centos.1               ntpdate.x86_64 0:4.2.6p5-25.el7.centos.1                openssh.x86_64 0:6.6.1p1-33.el7_3
  openssh-clients.x86_64 0:6.6.1p1-33.el7_3         openssh-server.x86_64 0:6.6.1p1-33.el7_3                  openssl.x86_64 1:1.0.1e-60.el7_3.1                 openssl-libs.x86_64 1:1.0.1e-60.el7_3.1                 policycoreutils.x86_64 0:2.5-11.el7_3
  policycoreutils-python.x86_64 0:2.5-11.el7_3      polkit.x86_64 0:0.112-11.el7_3                            python-firewall.noarch 0:0.4.3.2-8.1.el7_3.2       rockstor-release.x86_64 0:3-8.16.el7                    samba.x86_64 0:4.4.4-12.el7_3
  samba-client.x86_64 0:4.4.4-12.el7_3              samba-client-libs.x86_64 0:4.4.4-12.el7_3                 samba-common.noarch 0:4.4.4-12.el7_3               samba-common-libs.x86_64 0:4.4.4-12.el7_3               samba-common-tools.x86_64 0:4.4.4-12.el7_3
  samba-libs.x86_64 0:4.4.4-12.el7_3                samba-winbind.x86_64 0:4.4.4-12.el7_3                     samba-winbind-clients.x86_64 0:4.4.4-12.el7_3      samba-winbind-krb5-locator.x86_64 0:4.4.4-12.el7_3      samba-winbind-modules.x86_64 0:4.4.4-12.el7_3
  selinux-policy.noarch 0:3.13.1-102.el7_3.15       selinux-policy-targeted.noarch 0:3.13.1-102.el7_3.15      systemd.x86_64 0:219-30.el7_3.7                    systemd-libs.x86_64 0:219-30.el7_3.7                    systemd-sysv.x86_64 0:219-30.el7_3.7
  vim-minimal.x86_64 2:7.4.160-1.el7_3.1            wpa_supplicant.x86_64 1:2.0-21.el7_3                      xfsprogs.x86_64 0:4.5.0-9.el7_3

Complete!
[root@rockstor ~]# yum update
Loaded plugins: changelog, fastestmirror
Rockstor-Stable                                                                                                                                                                                                                                         | 2.9 kB  00:00:00
rockstor                                                                                                                                                                                                                                                | 2.9 kB  00:00:00
Loading mirror speeds from cached hostfile
 * base: centosmirror.netcup.net
 * epel: mirror.netcologne.de
 * extras: mirror.softaculous.com
 * updates: mirror.softaculous.com
No packages marked for update
[root@rockstor ~]# clear
[root@rockstor ~]# ls
anaconda-ks.cfg
[root@rockstor ~]# ls
anaconda-ks.cfg
[root@rockstor ~]# reboot

[tobiashochguertel]

the samba share "share-backup" was sharing the share "backup" from the "Shares" menu from Storage. The "Share" with name "backup" is also not longer available.

I notice yesterday that I wasn't able to create an Storage->Samba-Share with the same name as a Storage->Share. The Web-UI told me that it isn't possible to create an Share with same name. That's the reason why my samba-share got the name "share-backup".

I saw that I've got an rockstor update via yum, but the UI version doesn't change, may be the underling backend got an new version which applies a fix in this area where I get this issue?

[tobiashochguertel]

My try to add again the Storage->Share with name "backup" again, is not possible.

            Traceback (most recent call last):
  File "/opt/rockstor/eggs/gunicorn-0.16.1-py2.7.egg/gunicorn/workers/sync.py", line 34, in run
    client, addr = self.socket.accept()
  File "/usr/lib64/python2.7/socket.py", line 202, in accept
    sock, addr = self._sock.accept()
error: [Errno 11] Resource temporarily unavailable

[tobiashochguertel]

my try to edit an existing Storage->pool shows me an additional error:

            Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py", line 40, in _handle_exception
    yield
  File "/opt/rockstor/src/rockstor/storageadmin/views/pool_balance.py", line 49, in get_queryset
    self._balance_status(pool)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 145, in inner
    return func(*args, **kwargs)
  File "/opt/rockstor/src/rockstor/storageadmin/views/pool_balance.py", line 73, in _balance_status
    cur_status = balance_status(pool)
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 950, in balance_status
    mnt_pt = mount_root(pool)
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 238, in mount_root
    run_command(mnt_cmd)
  File "/opt/rockstor/src/rockstor/system/osi.py", line 109, in run_command
    raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = ['/bin/mount', '/dev/disk/by-label/backup', '/mnt2/backup', '-o', ',compress=no']. rc = 32. stdout = ['']. stderr = ['mount: wrong fs type, bad option, bad superblock on /dev/sdb,', '       missing codepage or helper program, or other error', '', '       In some cases useful info is found in syslog - try', '       dmesg | tail or so.', '']

[tobiashochguertel]

My try to create an Storage->Share shows me the following error:

            Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py", line 40, in _handle_exception
    yield
  File "/opt/rockstor/src/rockstor/storageadmin/views/share.py", line 170, in post
    pqid = qgroup_create(pool)
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 668, in qgroup_create
    mnt_pt = mount_root(pool)
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 238, in mount_root
    run_command(mnt_cmd)
  File "/opt/rockstor/src/rockstor/system/osi.py", line 109, in run_command
    raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = ['/bin/mount', '/dev/disk/by-label/backup', '/mnt2/backup', '-o', ',compress=no']. rc = 32. stdout = ['']. stderr = ['mount: wrong fs type, bad option, bad superblock on /dev/sdb,', '       missing codepage or helper program, or other error', '', '       In some cases useful info is found in syslog - try', '       dmesg | tail or so.', '']

                Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py", line 40, in _handle_exception
    yield
  File "/opt/rockstor/src/rockstor/storageadmin/views/share.py", line 170, in post
    pqid = qgroup_create(pool)
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 668, in qgroup_create
    mnt_pt = mount_root(pool)
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 238, in mount_root
    run_command(mnt_cmd)
  File "/opt/rockstor/src/rockstor/system/osi.py", line 109, in run_command
    raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = ['/bin/mount', '/dev/disk/by-label/backup', '/mnt2/backup', '-o', ',compress=no']. rc = 32. stdout = ['']. stderr = ['mount: wrong fs type, bad option, bad superblock on /dev/sdb,', '       missing codepage or helper program, or other error', '', '       In some cases useful info is found in syslog - try', '       dmesg | tail or so.', '']

[tobiashochguertel]

I think it would be helpfully to attach th rockstor.log

rockstor.log.txt

[tobiashochguertel]

The Samba Share is still existing in the samba.conf file:

[root@rockstor samba]# cat smb.conf
# See smb.conf.example for a more detailed config file or
# read the smb.conf manpage.
# Run 'testparm' to verify the config is correct after
# you modified it.

[global]
        workgroup = SAMBA
        security = user

        passdb backend = tdbsam

        printing = cups
        printcap name = cups
        load printers = yes
        cups options = raw

[homes]
        comment = Home Directories
        valid users = %S, %D%w%S
        browseable = No
        read only = No
        inherit acls = Yes

[printers]
        comment = All Printers
        path = /var/tmp
        printable = Yes
        create mask = 0600
        browseable = No

[print$]
        comment = Printer Drivers
        path = /var/lib/samba/drivers
        write list = root
        create mask = 0664
        directory mask = 0775
####BEGIN: Rockstor SAMBA CONFIG####
[share-backup]
    root preexec = "/opt/rockstor/bin/mnt-share share-backup"
    root preexec close = yes
    comment =
    path = /mnt2/share-backup
    browseable = yes
    read only = no
    guest ok = no
    admin users = admin hochguertelto
####END: Rockstor SAMBA CONFIG####
[root@rockstor samba]#

[MFlyer]

Hi @tobiashochguertel , usually never had issues with samba shares through system reboots.

Back to your update:

It could be that the samba share isn't persistent in the database or after upgrade previous persist shares are removed? No

Sometimes samba updates try to overwrite current samba config, but usually you get asked about this

Reading issues both on shares & pools I think this is on a lower level, had a quick read to your Rockstor logs and it seems about disks too:

[04/Mar/2017 19:39:02] ERROR [storageadmin.views.disk:255] Error running a command. cmd = ['/usr/sbin/smartctl', '--info', '/dev/disk/by-id/usb-SanDisk_Ultra_Fit_4C530001110405102123-0:0']. rc = 1. stdout = ['smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.8.7-1.el7.elrepo.x86_64] (local build)', 'Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org', '', '/dev/disk/by-id/usb-SanDisk_Ultra_Fit_4C530001110405102123-0:0: Unknown USB bridge [0x0781:0x5583 (0x100)]', 'Please specify device type with the -d option.', '', 'Use smartctl -h to get a usage summary', '', '']. stderr = ['']
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/disk.py", line 253, in _update_disk_state
    do.name, do.smart_options)
  File "/opt/rockstor/src/rockstor/system/smart.py", line 316, in available
    [SMART, '--info'] + get_dev_options(device, custom_options))
  File "/opt/rockstor/src/rockstor/system/osi.py", line 109, in run_command
    raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = ['/usr/sbin/smartctl', '--info', '/dev/disk/by-id/usb-SanDisk_Ultra_Fit_4C530001110405102123-0:0']. rc = 1. stdout = ['smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.8.7-1.el7.elrepo.x86_64] (local build)', 'Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org', '', '/dev/disk/by-id/usb-SanDisk_Ultra_Fit_4C530001110405102123-0:0: Unknown USB bridge [0x0781:0x5583 (0x100)]', 'Please specify device type with the -d option.', '', 'Use smartctl -h to get a usage summary', '', '']. stderr = ['']
[04/Mar/2017 19:39:03] ERROR [storageadmin.views.command:75] Exception while refreshing state for Pool(backup). Moving on: Error running a command. cmd = ['/bin/mount', '/dev/disk/by-label/backup', '/mnt2/backup', '-o', ',compress=no']. rc = 32. stdout = ['']. stderr = ['mount: wrong fs type, bad option, bad superblock on /dev/sdb,', '       missing codepage or helper program, or other error', '', '       In some cases useful info is found in syslog - try', '       dmesg | tail or so.', '']
[04/Mar/2017 19:39:03] ERROR [storageadmin.views.command:76] Error running a command. cmd = ['/bin/mount', '/dev/disk/by-label/backup', '/mnt2/backup', '-o', ',compress=no']. rc = 32. stdout = ['']. stderr = ['mount: wrong fs type, bad option, bad superblock on /dev/sdb,', '       missing codepage or helper program, or other error', '', '       In some cases useful info is found in syslog - try', '       dmesg | tail or so.', '']
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/command.py", line 65, in _refresh_pool_state
    mount_root(p)
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 238, in mount_root
    run_command(mnt_cmd)
  File "/opt/rockstor/src/rockstor/system/osi.py", line 109, in run_command
    raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = ['/bin/mount', '/dev/disk/by-label/backup', '/mnt2/backup', '-o', ',compress=no']. rc = 32. stdout = ['']. stderr = ['mount: wrong fs type, bad option, bad superblock on /dev/sdb,', '       missing codepage or helper program, or other error', '', '       In some cases useful info is found in syslog - try', '       dmesg | tail or so.', '']
[04/Mar/2017 19:40:03] ERROR [storageadmin.views.disk:255] Error running a command. cmd = ['/usr/sbin/smartctl', '--info', '/dev/disk/by-id/usb-SanDisk_Ultra_Fit_4C530001110405102123-0:0']. rc = 1. stdout = ['smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.8.7-1.el7.elrepo.x86_64] (local build)', 'Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org', '', '/dev/disk/by-id/usb-SanDisk_Ultra_Fit_4C530001110405102123-0:0: Unknown USB bridge [0x0781:0x5583 (0x100)]', 'Please specify device type with the -d option.', '', 'Use smartctl -h to get a usage summary', '', '']. stderr = ['']
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/disk.py", line 253, in _update_disk_state
    do.name, do.smart_options)
  File "/opt/rockstor/src/rockstor/system/smart.py", line 316, in available
    [SMART, '--info'] + get_dev_options(device, custom_options))
  File "/opt/rockstor/src/rockstor/system/osi.py", line 109, in run_command
    raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = ['/usr/sbin/smartctl', '--info', '/dev/disk/by-id/usb-SanDisk_Ultra_Fit_4C530001110405102123-0:0']. rc = 1. stdout = ['smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.8.7-1.el7.elrepo.x86_64] (local build)', 'Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org', '', '/dev/disk/by-id/usb-SanDisk_Ultra_Fit_4C530001110405102123-0:0: Unknown USB bridge [0x0781:0x5583 (0x100)]', 'Please specify device type with the -d option.', '', 'Use smartctl -h to get a usage summary', '', '']. stderr = ['']
[04/Mar/2017 19:40:04] ERROR [storageadmin.views.command:75] Exception while refreshing state for Pool(backup). Moving on: Error running a command. cmd = ['/bin/mount', '/dev/disk/by-label/backup', '/mnt2/backup', '-o', ',compress=no']. rc = 32. stdout = ['']. stderr = ['mount: wrong fs type, bad option, bad superblock on /dev/sdb,', '       missing codepage or helper program, or other error', '', '       In some cases useful info is found in syslog - try', '       dmesg | tail or so.', '']
[04/Mar/2017 19:40:04] ERROR [storageadmin.views.command:76] Error running a command. cmd = ['/bin/mount', '/dev/disk/by-label/backup', '/mnt2/backup', '-o', ',compress=no']. rc = 32. stdout = ['']. stderr = ['mount: wrong fs type, bad option, bad superblock on /dev/sdb,', '       missing codepage or helper program, or other error', '', '       In some cases useful info is found in syslog - try', '       dmesg | tail or so.', '']
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/command.py", line 65, in _refresh_pool_state
    mount_root(p)
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 238, in mount_root
    run_command(mnt_cmd)
  File "/opt/rockstor/src/rockstor/system/osi.py", line 109, in run_command
    raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = ['/bin/mount', '/dev/disk/by-label/backup', '/mnt2/backup', '-o', ',compress=no']. rc = 32. stdout = ['']. stderr = ['mount: wrong fs type, bad option, bad superblock on /dev/sdb,', '       missing codepage or helper program, or other error', '', '       In some cases useful info is found in syslog - try', '       dmesg | tail or so.', '']

@phillxnet ?

M.

[tobiashochguertel]

@MFlyer @phillxnet tell me if I shoudl provide more informations. The system is fresh, yesterday installed. Just started to play around and to evaluate rockstor as an solution for my nas to storage backups until these get archive on bluray discs.

Is the database of rockstor an mysql or an sqlite database, would it be helpfully to have the current state of the database?

[MFlyer]

@tobiashochguertel Rockstor runs over Django+PostgreSQL so working with Django ORM

Can you provide a disks page screenshot? M.

[tobiashochguertel]

The current disk page screenshot:

I have to tell, that I have yesterday played around with the case that one drive is failed and get replaced. My drives are in an physical hot-swap case.

1. I started to copy data via Samba share to share "share-backup" via Samba.
2. While the copy process I physical removed (un-plugged via hot-swap case) one hard-disk of the backup pool (Raid 1), when 10 GB of 80 GB was copied.
3. I added an new physical hard-disk in the hot-swap case.
4. I started an rescan in Storage->pool.
5. Added the new drive (new physcical hard-drisk) to the Storage->Pool "backup" and waited until the process of syncronisation was done to 100%. (the backup-date transfer was at 40 GB of 80 GB until this time.)
6. The pool "backup" had three drives. I removed the failed drive.
7. I removed the failed drive from the pool.
8. I want sleep until the data copy process was finished.
9. I've got an mail with the information that yum wasn't able to apply an upgrade, because of missing public-keys. I started the ssh sesion and done the upgrade manual and answers the questions about importing the missing public-keys with "yes".

Name    Serial  Capacity    Pool    Power Status    APM Model   Transport   Vendor  S.M.A.R.T
 ata-WDC_WD1001FALS-00E8B0_WD-WMATV3930586  WD-WMATV3930586         931.50 GB   backup   active/idle  20 minutes    ??? WDC WD1001FALS-0    sata    ATA  ON OFF
 ata-WDC_WD1001FALS-00E8B0_WD-WMATV3931443  WD-WMATV3931443         931.50 GB   backup   active/idle  20 minutes    ??? WDC WD1001FALS-0    sata    ATA  ON OFF
 usb-SanDisk_Ultra_Fit_4C530001110405102123-0:0-part3   4C530001110405102123        53.60 GB    rockstor_rockstor   unknown     ??? Ultra Fit   usb SanDisk  Not Supported

[phillxnet]

@MFlyer I see 2 disk issues reported (from a quick look): The first is harmless and could be hushed by maybe adding a custom SMART option of "-d sat".

'/dev/disk/by-id/usb-SanDisk_Ultra_Fit_4C530001110405102123-0:0: Unknown USB bridge [0x0781:0x5583 (0x100)]', 'Please specify device type with the -d option.'

But the second is an indication of a failed mount and could simply be an instance of issue #1547 where on rare occasions a mount by label (which is what we do) can fail. It's a know btrfs issue as explained and referenced in that issue.

['/bin/mount', '/dev/disk/by-label/backup', '/mnt2/backup', '-o', ',compress=no']. rc = 32. stdout = ['']. stderr = ['mount: wrong fs type, bad option, bad superblock on /dev/sdb ...

See my second posting in that issue, ie

btrfs device scan

Or it could be an indication of an unhealthy pool which btrfs currently defaults to not mounting. With the caveat that a repair is only possible on the first degraded mount as there after only read only mounts are permitted. It's a hotly debated default for btrfs which I hope will soon receive more attention: see: https://mail-archive.com/linux-btrfs@vger.kernel.org/msg61995.html for the most recent discussion on the list involving this default behaviour.

OK, while typing this response I see @tobiashochguertel has added a new post that fills in some info that may well relate to the know issue re 'btrfs device scan' requirements with live drive changes as drives were changed which I hadn't originally realise but fits.

So looks like mount not share issues as @MFlyer suspected.

[tobiashochguertel]

The USB Disk usb-SanDisk_Ultra_Fit_4C530001110405102123-0:0: is the drive with rockstor operation sytem installed. The other two drive's are both to store data - to store only backup data.

I don't really understand the keyword "degraded", what does it mean - or what have I to do?

Should it not be that the created shares and samba shares are always available also when a drive fails?

[MFlyer]

Hi @tobiashochguertel , your post about drive change probably explains your issue: Rockstor has some background routines checking disks/shares/pools state (those failing in your logs) and although your system supports hot swap, btrfs needs some specific steps to replace a disk. My suggestion is to have a new Rockstor installation to flush all db entries and start with a clean system.

Mirko

[tobiashochguertel]

@MFlyer these steps to replace a disk are found in the following article of the rockstor documentation? http://rockstor.com/docs/data_loss.html#data-loss-prevention-and-recovery-in-raid1-pools

Reinstalling is okay, but I need to understand the way and steps to work with an harddrive fail in case of an real emergency. Currently I simulating an emergency case for raid1 fail.

Result of my first simulation is: My system is in an state of no way back - compelete data lose. Done by working with the Web-UI. I can't recover my data.

I was expecting: I tought that rockstor with the Web-Gui provides an human-friendly way to done these task via clicking around in the web-ui.

When I reinstalled the system, wiped all 3 drive's again and start again with an my cases of emergency simulation to train the reallity. Is the documentation up to date from above to replace an failed harddrive in raid1 (pool)?

[MFlyer]

Hi @tobiashochguertel ,

I was expecting: I tought that rockstor with the Web-Gui provides an human-friendly way to done these task via clicking around in the web-ui.

Currently data recovery over CLI, not via WebUI

[tobiashochguertel]

@MFlyer Thanks for clarify.

For the posterity, the the steps for hard drive replace when failed is writen down in the documentation of rockstor http://rockstor.com/docs/data_loss.html#data-loss-prevention-and-recovery-in-raid1-pools.

I think we can close this issue now?

[MFlyer]

Hi @tobiashochguertel , if you agree you can close this issue :)

Obviously our plans are to have a nice recovery system via WebUI :wink:

[phillxnet]

@tobiashochguertel and @MFlyer Linking to partly relevant (as just disk replace) existing open issue:

Implement a disk replacement UI #1611

[phillxnet]

@tobiashochguertel Sorry forgot about:

"I don't really understand the keyword "degraded", what does it mean - or what have I to do?"

"degraded" is a mandatory mount option in order to successfully mount a pool which has, for example, a missing member, or other flagged issue. This option is not implemented by Rockstor and is generally only required during manual repair procedures as otherwise all mount efforts are rejected, which from Rockstor's perspective leaves empty shares as there is no backing pool to present data to those shares (sub-volumes).

My previously referenced link to the linux-btrfs mailing list shows some disagreement and acknowledgement on how this places btrfs aside from pretty much all other raid systems, where a degraded mount is default. Rockstor currently adheres to these defaults but that may change when we come to implement better disk and pool health reporting.

@MFlyer apologies if I'm butting in here.

[tobiashochguertel]

@phillxnet thanks for answer my question. By the way, I've read the mailing list discussion. Was quite intressting :) I alos notice the updated btrfs.wiki article to yellow state for raid1. Yes I was expecting that a raid is a raid - so if something in a mirror raid went wrong the raid system will start the work from self to replicate the data to the new drive.

It bothers me that in the Ui all related entries disappear - but they are in the background even there. I can't access these configurations in Storage->shares and Storage->samba-shares any longer. So they are still there but not shown until the behind error (pool error, degraded state) is fixed as I

Would it be possible to highlight these entries "red" or with an different ui-behavior instead of complete not showing them?

If I can see these entries also when a error is behind them I will have a good user experience. In large environmens I can see quick which shares are effected by an error.

[MFlyer]

@schakrava : disk replacement related - there's another open issue -, I think we can close this not samba related

Mirko

[phillxnet]

@tobiashochguertel and @MFlyer I agree that this issue need to be close really. Useful to have highlighted known current limitations with btrfs and current requirement for degraded mount option and the also known limitation of pool status reporting. @tobiashochguertel If you are still in attendance with this issue then please feel free to close it (if @schakrava hasn't done so already) as I think we have now established that your expectation is not in line with current btrfs defaults and that we are currently falling short on pool health / mount status reporting. This is definitely on the road map and given recent disk management / reporting work the pool health / status reporting shouldn't be too far down the line.

Thanks for you contributions in this issue.

@schakrava I think we can close this one now as not well defined in itself and base Rockstor function re pool state reporting is understood as a future feature requirement but is not currently part of our existing (and near end) milestone.

[tobiashochguertel]

I have closed the issue - but I'm not sure if this issue is not fixed or will be fixed because there are no related (linked) issues with this issue?

[phillxnet]

@tobiashochguertel Thanks. The problem with this issue was that is was not very focused. Ie started with samba shares not preserved over reboot (as well as repo key issues), which they are. And the initial title suggestion took @MFlyer, our samba person (among many other talents), off on a goose chase, however this turned out to be as an artefact of your tests, which you didn't initially introduce but showed up in the logs, re removing drives. This lead to links to btrfs default behaviour, which we currently follow: ergo the degraded mount option requirement discussion, pointers to btrfs mailing list etc.

As this is a very long and meandering tale involving a number of issues I thought it best that we 'tidy up' by attempting to have only more focused issues open as otherwise it takes quite a while for each developer to re-read what has gone before when it is already very well known by all developers that there is no pool health reporting and that once we have more of the fundamentals sorted we can move up to that level. By which time maybe the current work at facebook on multi-drive btrfs (one of this years focus points) and the long pending work by Anand Jain on adding drive health awareness to btrfs will help us on our way, ie btrfs has no current concept of a bad drive.

If you want to open a new clean issue with a focus on one item. Or a few issues, each focused on a single element of your original report, each with reproducible steps, then that would be a further contribution. There is no single issue here other than re-describing the behaviour of the test you performed but did not initially report: ie removed disks and pool didn't mount. But there is also repo key issue, which I haven't seen myself for a bit either.

So sorry to potentially come across as a little short but I think your contribution would be more valuable and accessible if it was distributed across maybe several more focused issues as the winding nature of this one (partly through your impression of how btrfs should work and our adhearing to current default bahaviour which is different from most current raid behaviour) has already take quite a while and hasn't really helped with development as much as it might have done.

You requested links to other issues, yet my attempted focus on another issue and it's link to this issue was what brought me to re-address and re-read this entire thread. That issue was "insufficient use of btrfs device scan" #1547 which I had linked to this issue on 4th march, the day you opened this issue and @MFlyer asked that I look to the disk side of your log entries. That issue in turn links to known behaviour and btrfs wiki entries confirming that behaviour and reproducible steps. The issue in turn linked to an only recently closed issue #550 which in turn also was related to the reproducible steps. The originally linked issue also contains the exact same report of drive mount failure I saw in your logs, hence the link (and it remains open). And as of #550 (closed as merged as of 15 days ago) also contains pool mount failure logging improvements that would further assist diagnosing what happened in your reported scenario specifically commit: https://github.com/phillxnet/rockstor-core/commit/3045722e8d85b3363dc6f10cac2c391dd7ebc332 which was one of 76 commits related to drive management, which follows on from another rather large pull request which itself deals with better drive categorisation / management #1622 (93 commits). The two referenced issues / pull requests in turn link to multiple other issues for context.

My suggestion in order to avoid these less focused issues is to first open a forum thread as it will one, get more eyes, and two, hopefully lead to the required focus in the GitHub issues themselves. All current members of the development team are very active on the forum so hopefully between us we can tease out what is already know and what is not so that we either open issues on that basis or request that particular reports be transferred to issues by the reporters.

Hope that helps and sorry if this is terse sounding, but focused issues are a must really. I've been picked up on the same myself in the past so appreciate how this could be coming across.

Thanks again for your efforts and rest assured that you are not being ignored. I think the many more eyes of the forum is the place to start with whittling down what is know and unknown behaviour though. And again feel free to open new issues on single topics with reproducible steps using current testing channel update releases.

Thanks for you understanding and efforts in making Rockstor better, bit by bit.