search

rockstor / rockstor-core

Linux/BTRFS based Network Attached Storage(NAS)
http://rockstor.com/docs/contribute_section.html
GNU General Public License v3.0
548 stars 137 forks source link

Config restore: samba share exports not consistently restored #2847

Open Hooverdan96 opened 3 weeks ago

Hooverdan96 commented 3 weeks ago

After installing the 5.0.9.0 version and restoring from a configuration exported in a 4.x system, i noticed 2 things.

Samba Shares were not created:

[16/Jun/2024 13:27:43] ERROR [storageadmin.views.config_backup:60] Exception occurred while creating resource: https://localhost/api/samba. Payload: [{'share': 5, 'path': ...

after the payload this message shows:

 Exception: 500 Server Error: Internal Server Error for url: https://localhost/api/samba. Moving on.

The samba service was actually updated with all the parameters from the previous install and was running.

Once the restore was finished, triggering it again, yielded the same error messages and no samba shares created.

The Rockons behaved a bit differently. I had attached the Rockon service to its prior RockonRoot manually (which was on one of the disk pools as a share).

None of the Rockons were showing in the UI (despite multiple refreshes. However, when running docker ps all containers were up and running,

But the error messages I saw in the log all indicated that docker was not running. Here is an example for the netdata Rockon:

[16/Jun/2024 13:12:48] ERROR [storageadmin.views.config_backup:60] Exception occurred while creating resource: https://localhost/api/rockons/41/stop. Payload: {'new_rid': 41, 'rname': 'Netdata (official)', 'containers': [161], 'shares': {}, 'ports': {19999: 19999}, 'devices': {},                                      'environment': {'PGID': '476'}, 'cc': {}, 'labels': {'com.centurylinklabs.watchtower.enable=true': 'netdata_official'}}. Exception: 500 Server Error: Internal Server Error for url: https://localhost/api/rockons/41/stop. Moving on.
[16/Jun/2024 13:12:48] INFO [storageadmin.views.config_backup:370] Send update command to the rock-ons api for the following rock-on: Netdata (official)
[16/Jun/2024 13:12:48] ERROR [storageadmin.util:45] Exception: Docker service is not running. Start it and try again.
NoneType: None

Even though the service seems to be running.

After the restarting of the restore operation (as described above) the installed Rockons were now visible in the WebUI. So that's good news, but I am stumped why the api was initially not recognizing that the docker service was already running during the first restore.

FroggyFlox commented 2 weeks ago

Hi @Hooverdan96 ,

Thanks a lot for yet another critical report...

I'm trying to think about this one and wanted to verify something first: the import of the Pool(s) that contained the problematic Samba shares in question went fine? All shares are listed and without error?

Hooverdan96 commented 2 weeks ago

@FroggyFlox, yes all Samba corresponding shares were imported correctly. The one that I forgot to recreate was a transcode directory on the root drive. But for that one, I also didn't have a Samba share created (so there wasn't anything in the backup for that). None of the Samba shares were connected to the root drive originally, only for the pool related shares.

FroggyFlox commented 2 weeks ago

Thanks for the confirmation... Sorry for all the questions, but that one is puzzling and we indeed do not seem to have good feedback in the error returned by Rockstor here.

Would you be able to paste the full payload or was it truncated as well? I'm interested in the following:

[16/Jun/2024 13:27:43] ERROR [storageadmin.views.config_backup:60] Exception occurred while creating resource: https://localhost/api/samba. Payload: [{'share': 5, 'path': ...

Alternatively, do you see more debug information in the logs if you turn ON debug mode?

Hooverdan96 commented 2 weeks ago

I can paste the full payload. It didn't seem important at the time :). Let me find it again in the logs. I now created some of them manually, since I needed to restore my "prod" system.

FroggyFlox commented 2 weeks ago

Thanks a lot... Just curious in case there is something that jumps out of the ordinary with it.

Hooverdan96 commented 2 weeks ago 
[16/Jun/2024 13:27:43] ERROR [storageadmin.views.config_backup:60] Exception occurred while creating resource: https://localhost/api/samba.
Payload: [{'share': 5, 'path': '/mnt2/RockStor', 'comment': 'RockStor Files', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [5]}, 
{'share': 10, 'path': '/mnt2/minecraft', 'comment': 'Minecraft Server data', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [10]}, 
{'share': 2, 'path': '/mnt2/HDHomeRun_config', 'comment': 'HDHomeRun Configuration Share', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [2]}, 
{'share': 3, 'path': '/mnt2/lms-config', 'comment': 'Logitech Media Server Configuration', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [3]}, 
{'share': 13, 'path': '/mnt2/transcode', 'comment': 'Plex transcoding - on SSD', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [13]}, 
{'share': 9, 'path': '/mnt2/Sick-config', 'comment': 'to delete later', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [9]}, 
{'share': 6, 'path': '/mnt2/tautulli_config', 'comment': 'Tautulli Configuration', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [6]}, 
{'share': 11, 'path': '/mnt2/plex-config', 'comment': 'Plex Configuration Directory', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [11]}, 
{'share': 7, 'path': '/mnt2/scrutiny_config', 'comment': 'Scrutiny config', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [7]}, 
{'share': 4, 'path': '/mnt2/ha_config', 'comment': 'Home Assistant Config', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [4]}, 
{'share': 8, 'path': '/mnt2/test-config', 'comment': 'test Rockon installs', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [8]}, 
{'share': 14, 'path': '/mnt2/RockonRoot', 'comment': 'Rockon Root Directory', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [14]}, 
{'share': 15, 'path': '/mnt2/WGEasy_config', 'comment': 'Samba-Export', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [15]}, 
{'share': 17, 'path': '/mnt2/dnsmasq-config', 'comment': 'pihole - dnsmasq-configuration', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [17]}, 
{'share': 18, 'path': '/mnt2/pihole-config', 'comment': 'pi-hole configuration', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [18]}, 
{'share': 21, 'path': '/mnt2/fb-conf', 'comment': 'Samba-Export', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [21]}, 
{'share': 20, 'path': '/mnt2/fb-db', 'comment': 'Samba-Export', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [20]}]. 
Exception: 500 Server Error: Internal Server Error for url: https://localhost/api/samba. Moving on.

Only change I made to the output from the log is to insert line breaks between the shares. In the log it was one long line.

Hooverdan96 commented 2 weeks ago

could it be that the samba service had not started yet (since there were custom settings to be updated by the time it gets to the creation of shares? Though, I always felt that a samba server startup doesn't take that long ...

FroggyFlox commented 2 weeks ago

If I understand correctly your first message...

Once the restore was finished, triggering it again, yielded the same error messages and no samba shares created.

... I would think that the status of the samba service does not come into play here (or at least is not chasing the problem). On the other hand, the intermittent error with the docker service you observed may underline some timeouts or alike.

I'd like to focus on the samba but first, though, as that seems to be the one consistently problematic. Can you confirm that creating a new Samba export works just fine? Also, it would be great to see what the logs say after a new restore attempt while debug mode is on. I understand this is your "production" machine, though, so I understand if you don't want to mess with it more. I'll keep thinking about it some more as that is a troubling one. Do the id of the shares in the payload match those you see in the database?

Hooverdan96 commented 2 weeks ago

I can confirm creating a Samba share manually worked just fine (multiple ones in fact).

Can you remind me again where I set the debug flag?

Is this the one:

buildvm:~ # cd /opt/rockstor
buildvm:~ # export DJANGO_SETTINGS_MODULE=settings
buildvm:~ # poetry run debug-mode ON

I'll see that I can get to that in a little bit and can then report back.

FroggyFlox commented 2 weeks ago

Yes, that's it. Thank you so much, and sorry for the incessant questions... hopefully we'll have a better idea of what is happening here.

Hooverdan96 commented 2 weeks ago

Interestingly, when I turn debug=on then I am getting this error message screen:

image

This is what I see in the log:

[18/Jun/2024 13:14:01] ERROR [storageadmin.views.config_backup:612] [Errno 2] No such file or directory: '/opt/rockstor/static/config-backups/quick_test.json.gz'
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/config_backup.py", line 609, in get_queryset
    with gzip.open(fp, "rb") as f:
         ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/gzip.py", line 58, in open
    binary_file = GzipFile(filename, gz_mode, compresslevel)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/gzip.py", line 174, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/opt/rockstor/static/config-backups/quick_test.json.gz'
[18/Jun/2024 13:14:01] INFO [storageadmin.views.config_backup:613] The file quick_test.json.gz is not gzipped, so compress it now.
[18/Jun/2024 13:14:01] DEBUG [system.osi:261] Running command: /usr/bin/gzip /opt/rockstor/static/config-backups/quick_test.json.gz
[18/Jun/2024 13:14:01] ERROR [system.osi:287] non-zero code(1) returned by command: ['/usr/bin/gzip', '/opt/rockstor/static/config-backups/quick_test.json.gz']. output: [''] error: ['gzip: /opt/rockstor/static/config-backups/quick_test.json.gz: No such file or directory', '']
[18/Jun/2024 13:14:01] ERROR [storageadmin.middleware:34] Exception occurred while processing a request. Path: /api/config-backup method: GET
[18/Jun/2024 13:14:01] ERROR [storageadmin.middleware:35] ConfigBackup object can't be deleted because its id attribute is set to None.
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/config_backup.py", line 609, in get_queryset
    with gzip.open(fp, "rb") as f:
         ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/gzip.py", line 58, in open
    binary_file = GzipFile(filename, gz_mode, compresslevel)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/gzip.py", line 174, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/opt/rockstor/static/config-backups/quick_test.json.gz'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/config_backup.py", line 620, in get_queryset
    o, err, rc = run_command(["/usr/bin/gzip", fp], log=True)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/src/rockstor/system/osi.py", line 289, in run_command
    raise CommandException(cmd, out, err, rc)
system.exceptions.CommandException: Error running a command. cmd = /usr/bin/gzip /opt/rockstor/static/config-backups/quick_test.json.gz. rc = 1. stdout = ['']. stderr = ['gzip: /opt/rockstor/static/config-backups/quick_test.json.gz: No such file or directory', '']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
    return view_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/django/views/generic/base.py", line 104, in view
    return self.dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/rest_framework/generics.py", line 243, in get
    return self.list(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/rest_framework/mixins.py", line 38, in list
    queryset = self.filter_queryset(self.get_queryset())
                                    ^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/src/rockstor/storageadmin/views/config_backup.py", line 638, in get_queryset
    cbo.delete()
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/django/db/models/base.py", line 1125, in delete
    raise ValueError(
ValueError: ConfigBackup object can't be deleted because its id attribute is set to None.
[18/Jun/2024 13:14:05] DEBUG [storageadmin.views.home:68] context={'request': <WSGIRequest: GET '/'>, 'current_appliance': <Appliance: Appliance object (1)>, 'setup_user': True, 'page_size': 15, 'update_channel': 'Testing'}
[18/Jun/2024 13:14:05] DEBUG [storageadmin.views.home:70] ABOUT TO RENDER INDEX
[18/Jun/2024 13:14:06] ERROR [storageadmin.views.config_backup:612] [Errno 2] No such file or directory: '/opt/rockstor/static/config-backups/backup-2024-06-14-121421.json.gz'
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/config_backup.py", line 609, in get_queryset
    with gzip.open(fp, "rb") as f:
         ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/gzip.py", line 58, in open
    binary_file = GzipFile(filename, gz_mode, compresslevel)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/gzip.py", line 174, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/opt/rockstor/static/config-backups/backup-2024-06-14-121421.json.gz'
[18/Jun/2024 13:14:06] INFO [storageadmin.views.config_backup:613] The file backup-2024-06-14-121421.json.gz is not gzipped, so compress it now.
[18/Jun/2024 13:14:06] DEBUG [system.osi:261] Running command: /usr/bin/gzip /opt/rockstor/static/config-backups/backup-2024-06-14-121421.json.gz
[18/Jun/2024 13:14:06] ERROR [system.osi:287] non-zero code(1) returned by command: ['/usr/bin/gzip', '/opt/rockstor/static/config-backups/backup-2024-06-14-121421.json.gz']. output: [''] error: ['gzip: /opt/rockstor/static/config-backups/backup-2024-06-14-121421.json.gz: No such file or directory', '']
[18/Jun/2024 13:14:06] ERROR [storageadmin.middleware:34] Exception occurred while processing a request. Path: /api/config-backup method: GET
[18/Jun/2024 13:14:06] ERROR [storageadmin.middleware:35] ConfigBackup object can't be deleted because its id attribute is set to None.
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/config_backup.py", line 609, in get_queryset
    with gzip.open(fp, "rb") as f:
         ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/gzip.py", line 58, in open
    binary_file = GzipFile(filename, gz_mode, compresslevel)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/gzip.py", line 174, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/opt/rockstor/static/config-backups/backup-2024-06-14-121421.json.gz'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/config_backup.py", line 620, in get_queryset
    o, err, rc = run_command(["/usr/bin/gzip", fp], log=True)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/src/rockstor/system/osi.py", line 289, in run_command
    raise CommandException(cmd, out, err, rc)
system.exceptions.CommandException: Error running a command. cmd = /usr/bin/gzip /opt/rockstor/static/config-backups/backup-2024-06-14-121421.json.gz. rc = 1. stdout = ['']. stderr = ['gzip: /opt/rockstor/static/config-backups/backup-2024-06-14-121421.json.gz: No such file or directory', '']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
    return view_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/django/views/generic/base.py", line 104, in view
    return self.dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/rest_framework/generics.py", line 243, in get
    return self.list(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/rest_framework/mixins.py", line 38, in list
    queryset = self.filter_queryset(self.get_queryset())
                                    ^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/src/rockstor/storageadmin/views/config_backup.py", line 638, in get_queryset
    cbo.delete()
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/django/db/models/base.py", line 1125, in delete
    raise ValueError(
ValueError: ConfigBackup object can't be deleted because its id attribute is set to None.

Once I turn it off again, no error message but it doesn't show any backups (I had 2 in my list yesterday) ... the only difference to yesterday is that I ran the upgrade to 5.0.10-0 to make the fix "official". So, it might be because of that. I will continue to observe that before I open an issue.

Now, when I turn the debug back on, the error messsage disappears. I can now upload a backup file again and start the restoration process.

And well, it seems that it attempts to create the smb shares before the samba service is configured and started?

[18/Jun/2024 13:34:33] ERROR [storageadmin.views.config_backup:60] Exception occurred while creating resource: https://localhost/api/samba. Payload: [{'share': 5, 'path': '/mnt2/RockStor', 'comment': 'RockStor Files', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [5]}, {'share': 10, 'path': '/mnt2/minecraft', 'comment': 'Minecraft Server data', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [10]}, {'share': 2, 'path': '/mnt2/HDHomeRun_config', 'comment': 'HDHomeRun Configuration Share', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [2]}, {'share': 3, 'path': '/mnt2/lms-config', 'comment': 'Logitech Media Server Configuration', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [3]}, {'share': 13, 'path': '/mnt2/transcode', 'comment': 'Plex transcoding - on SSD', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [13]}, {'share': 9, 'path': '/mnt2/Sick-config', 'comment': 'to delete later', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [9]}, {'share': 6, 'path': '/mnt2/tautulli_config', 'comment': 'Tautulli Configuration', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [6]}, {'share': 11, 'path': '/mnt2/plex-config', 'comment': 'Plex Configuration Directory', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [11]}, {'share': 7, 'path': '/mnt2/scrutiny_config', 'comment': 'Scrutiny config', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [7]}, {'share': 4, 'path': '/mnt2/ha_config', 'comment': 'Home Assistant Config', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [4]}, {'share': 8, 'path': '/mnt2/test-config', 'comment': 'test Rockon installs', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [8]}, {'share': 14, 'path': '/mnt2/RockonRoot', 'comment': 'Rockon Root Directory', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [14]}, {'share': 15, 'path': '/mnt2/WGEasy_config', 'comment': 'Samba-Export', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [15]}, {'share': 17, 'path': '/mnt2/dnsmasq-config', 'comment': 'pihole - dnsmasq-configuration', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [17]}, {'share': 18, 'path': '/mnt2/pihole-config', 'comment': 'pi-hole configuration', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [18]}, {'share': 21, 'path': '/mnt2/fb-conf', 'comment': 'Samba-Export', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [21]}, {'share': 20, 'path': '/mnt2/fb-db', 'comment': 'Samba-Export', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [20]}]. Exception: 500 Server Error: Internal Server Error for url: https://localhost/api/samba. Moving on.
[18/Jun/2024 13:34:33] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:33] INFO [storageadmin.views.config_backup:116] Started restoring NFS exports.
[18/Jun/2024 13:34:33] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:33] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:33] INFO [storageadmin.views.config_backup:56] Successfully created resource: https://localhost/api/adv-nfs-exports. Payload: {'entries': []}
[18/Jun/2024 13:34:33] INFO [storageadmin.views.config_backup:136] Finished restoring NFS exports.
[18/Jun/2024 13:34:33] INFO [storageadmin.views.config_backup:140] Started restoring services.
[18/Jun/2024 13:34:33] INFO [storageadmin.views.config_backup:151] Restore the following service: replication
[18/Jun/2024 13:34:33] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:33] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:33] INFO [storageadmin.views.config_backup:56] Successfully created resource: https://localhost/api/sm/services/replication/config. Payload: {'config': {'listener_port': '10002', 'network_interface': 'Wired connection 1'}}
[18/Jun/2024 13:34:33] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:33] INFO [storageadmin.views.config_backup:151] Restore the following service: smb
[18/Jun/2024 13:34:33] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:33] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:33] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:33] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:34] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:34] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:34] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:34] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:34] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:34] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:34] DEBUG [system.osi:261] Running command: /usr/bin/systemctl restart smb
[18/Jun/2024 13:34:34] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:34] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:34] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:34] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:34] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:34] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:34] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:34] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:34] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:34] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:34] DEBUG [system.osi:261] Running command: /usr/bin/systemctl restart nmb
[18/Jun/2024 13:34:34] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:34] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:34] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:34] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:34] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:34] INFO [storageadmin.views.config_backup:56] Successfully created resource: https://localhost/api/sm/services/smb/config. Payload: {'config': {'fruit:veto_appledouble': 'no', 'netbios name': 'ROCKSTORW', 'fruit:wipe_intentionally_left_blank_rfork': 'yes', 'min protocol': 'SMB2', 'os level': '255', 'fruit:posix_rename': 'yes', 'fruit:metadata': 'stream', 'local master': 'yes', 'preferred master': 'yes', 'fruit:nfs_aces': 'no', 'fruit:model': 'MacSamba', 'domain master': 'yes', 'vfs objects': 'catia fruit streams_xattr', 'fruit:delete_empty_adfiles': 'yes', 'workgroup': 'VOLWIL'}}
[18/Jun/2024 13:34:34] INFO [storageadmin.views.config_backup:151] Restore the following service: ntpd
[18/Jun/2024 13:34:34] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5
[18/Jun/2024 13:34:34] DEBUG [system.osi:261] Running command: /usr/sbin/btrfs subvolume list /mnt2/4xRAID5

After it creates the samba service it moves on to ntp, crony and the other services apparently (I didn't include that here). If you want the entire log to sort through I can get that to you, too.

FroggyFlox commented 2 weeks ago

Do the id of the shares in the payload match those you see in the database?

@Hooverdan96 , so far my best guess is on that point above. I've indeed had a quick look at the restore code and we don't seem to have a mechanism in place to update the share id from the old system (the one present on the backup being restored) and the share id on the new system. I thought we had that as I remember some issue like that not too long ago. I'll have to dig deeper but so far that's my best guess.

FroggyFlox commented 2 weeks ago

I implemented that kind of id conversion for pools and shares related to scheduled tasks but that did not touch samba shares, etc... https://github.com/rockstor/rockstor-core/commit/16edb0714187049b74222ac61153832185fbdffd

@Hooverdan96, if you can confirm that the shares indeed do not match, then it's very likely to be the culprit. One could verify that be manually fixing the share IDs in the config backup file but that's probably too cumbersome for your system as you have a lot of samba shares. It'll need to be confirmed and tested on a test machine so that we can have a nice reproducer.

Hooverdan96 commented 2 weeks ago

It took a little bit to remember how to look into the table :).

For my own recollection down the road, I used:

 psql -U rocky -d storageadmin -c "SELECT * FROM storageadmin_share;"

If these are the corresponding IDs that the Samba restore process is looking for, then they are indeed mismatched. I took a couple of examples.

image

Or am I looking in the wrong place?

Hooverdan96 commented 2 weeks ago

And, I took the dive in and adjusted the share tags for each of the Samba Shares with the one listed in the current Rockstor's share ids in the above shown table (did not change anything else from the original file). Uploaded and executed. Et voila, all shares are now showing up ...

image

So your suspicion of a similar problem that you mentioned above seems to be confirmed this way.

FroggyFlox commented 2 weeks ago

Thank you so much for testing all of that! I'm glad we seem to have pinpointed the cause of the original issue.

FroggyFlox commented 1 week ago

For reference, one should look at the restore scheduled tasks logic to update the share id. In particular, we already have a simple helper of interest: https://github.com/rockstor/rockstor-core/blob/0f70e46f4cbb74781adec01b6921afc3d548db05/src/rockstor/storageadmin/views/config_backup.py#L560

I'm not sure about the quality of error handling here, but it should at least return us to a working state and further improvements to error handling (if/where needed) could be the focus of future PR(s).

Hooverdan96 commented 1 week ago

Would this also be necessary for NFS and SFTP exports?

FroggyFlox commented 1 week ago

@Hooverdan96, Good point, indeed: I do not believe so as we use share names and not IDs for the NFS and SFTP exports. This would benefit from confirmation but I quickly looked at the code and it seems to confirm my memory on this. This would mean that we need to convert these IDs for the samba share restore procedure only.

FroggyFlox commented 3 days ago

As expected, I can reproduce the failure to restore samba shares if their IDs in the config backup mismatch the existing share ids in the target system. I simply edited the config backup file to change the share ids to something else:

{
  "model": "storageadmin.sambashare",
  "pk": 1,
  "fields": {
    "share": 11,
    "path": "/mnt2/test_share01",
    "comment": "Samba-Export",
    "browsable": "yes",
    "read_only": "no",
    "guest_ok": "no",
    "shadow_copy": false,
    "time_machine": false,
    "snapshot_prefix": null
  }
},
{
  "model": "storageadmin.sambashare", 
  "pk": 2, 
  "fields": {
    "share": 22, 
    "path": "/mnt2/test_share02",
    "comment": "Samba-Export",
    "browsable": "yes", 
    "read_only": "no", 
    "guest_ok": "no", 
    "shadow_copy": false, 
    "time_machine": false, 
    "snapshot_prefix": null
  }
}

Upon restore, we see the following error in the logs:

[04/Jul/2024 06:17:58] INFO [storageadmin.views.config_backup:104] Started restoring Samba exports.
[04/Jul/2024 06:17:58] ERROR [storageadmin.util:45] Exception: Share id (11) does not exist.
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/share.py", line 93, in _validate_share
    share = Share.objects.get(id=sid)
            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/django/db/models/query.py", line 637, in get
    raise self.model.DoesNotExist(
storageadmin.models.share.Share.DoesNotExist: Share matching query does not exist.
[04/Jul/2024 06:17:58] ERROR [storageadmin.views.config_backup:60] Exception occurred while creating resource: https://localhost/api/samba. Payload: [{'share': 11, 'path': '/mnt2/test_share01', 'comment': 'Samba-Export', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [11]}, {'share': 22, 'path': '/mnt2/test_share02', 'comment': 'Samba-Export', 'browsable': 'yes', 'read_only': 'no', 'guest_ok': 'no', 'shadow_copy': False, 'time_machine': False, 'snapshot_prefix': None, 'shares': [22]}]. Exception: 500 Server Error: Internal Server Error for url: https://localhost/api/samba. Moving on.
phillxnet commented 3 days ago

@FroggyFlox Thanks for the simple reproducer here. I was about to assign myself to this issue and see if I could find the same as a starting point. Shall I proceed (with self assignment) - or were you about to do the same? My vague plan, thus far, - based on @Hooverdan96 and your exposition to date, was to hopefully resource your prior code, already referenced, and hopefully apply the same work-around to help make this restore function more robust. We still have to deal with changing share / pool names. But give we don't yet support that, we can cross that bridge when we get to it. I.e. advice on every future share/pool name-change that a refresh config-backup be created. As we have to maintain our config-backup file format backward support capability.

Using the id was always going to fail but it's where we find ourselves and have already made some in-roads to addressing the short-falls.

FroggyFlox commented 3 days ago

@phillxnet , please go for it if you were about to as I don't know when I would be able to for sure.

My vague plan, thus far, - based on @Hooverdan96 and your exposition to date, was to hopefully resource your prior code, already referenced, and hopefully apply the same work-around to help make this restore function more robust.

That would have been my plan as well.

phillxnet commented 3 days ago

@FroggyFlox OK, I'll have a go - as this is a show-stopper for our next Stable release so wanted to help nudge this along if I can. I'll report progress here as usual: assuming I make any :).

phillxnet commented 2 days ago

@Hooverdan96 I've narrowed the scope a little in the associated PR re comment: https://github.com/rockstor/rockstor-core/pull/2864#issuecomment-2211056177

As your & @FroggyFlox's investigation/exposition/reproducer similarly narrowed to the SMB share exports restore failure.

After the restarting of the restore operation (as described above) the installed Rockons were now visible in the WebUI. So that's good news,

I think it takes a little while for each Rock-on to be installed and then show-up in the Web-UI: but I'm not convinced that is what you saw there. So I'm similarly narrowing the title on this issue - give there were failures in the samba share exports restore that could have, in-turn, messed up some other restore element. They are meant to each fail through but alas we have, obviously, a little more robustness work to do as of yet.

But all-in I'm pretty sure we have at least now addressed (in the linked PR) the main focus here: SMB exports in config-backup file failed to restore due to DB dump ID's mismatching the native (new install DB) share IDs. But as per @FroggyFlox prior work in this area - I've extended some of that into what we needed for the SMB export restore.

I'd like to restore each SMB export under it's own API call: but that also should be for a future PR was my thinking. I.e. we need first to handle any restores with associated Share ID mis-match first. Later we can then enhance the robustness once we have it actually working under said conditions. As this then enables more robust testing anyway.

Bit-by-bit and all in good time: hopefully.