Docker Volume folders not recursing.

[Morphy99]

Describe the problem I've added an exclude for the docker addon_data folder and included volumes inside it however I only get one file backed up and none of sub dirs.

Platform and Kodi version Libreelec 9.2.6 RPi 4

Here's my json which is correct AFAIK:

{
  "addon_data": {
    "dirs": [
      {
        "path": "special://home/userdata/addon_data/",
        "type": "include",
        "recurse": true
      },
      {
        "path": "special://home/userdata/addon_data/service.system.docker/",
        "type": "exclude"
      },
      {
        "path": "special://home/userdata/addon_data/service.system.docker/docker/volumes/",
        "type": "include",
        "recurse": true
      }
    ],
    "root": "special://home/userdata/addon_data/"
  },
  "playlists": {
    "dirs": [
      {
        "path": "special://home/userdata/playlists/",
        "type": "include",
        "recurse": true
      }
    ],
    "root": "special://home/userdata/playlists/"
  },
  "thumbnails": {
    "dirs": [
      {
        "path": "special://home/userdata/Thumbnails/",
        "type": "include",
        "recurse": true
      }
    ],
    "root": "special://home/userdata/Thumbnails/"
  },
  "database": {
    "dirs": [
      {
        "path": "special://home/userdata/Database/",
        "type": "include",
        "recurse": true
      }
    ],
    "root": "special://home/userdata/Database/"
  },
  "game_saves": {
    "dirs": [
      {
        "path": "special://home/userdata/Savestates/",
        "type": "include",
        "recurse": true
      }
    ],
    "root": "special://home/userdata/Savestates/"
  },
  "profiles": {
    "dirs": [
      {
        "path": "special://home/userdata/profiles/",
        "type": "include",
        "recurse": true
      }
    ],
    "root": "special://home/userdata/profiles/"
  },
  "addons": {
    "dirs": [
      {
        "path": "special://home/addons/",
        "type": "include",
        "recurse": true
      },
      {
        "path": "special://home/addons/packages/",
        "type": "exclude"
      },
      {
        "path": "special://home/addons/temp/",
        "type": "exclude"
      },
      {
        "path": "special://home/addons/service.system.docker/",
        "type": "exclude"
      }
    ],
    "root": "special://home/addons/"
  },
  "config": {
    "dirs": [
      {
        "path": "special://home/userdata/",
        "type": "include",
        "recurse": false
      },
      {
        "path": "special://home/userdata/keymaps/",
        "type": "include",
        "recurse": true
      },
      {
        "path": "special://home/userdata/peripheral_data/",
        "type": "include",
        "recurse": true
      },
      {
        "path": "special://home/userdata/library/",
        "type": "include",
        "recurse": true
      }
    ],
    "root": "special://home/userdata/"
  }
}

Here's what I have in the archive:

[robweber]

Could it be an ordering thing? Take a look at the addons directory definition. The includes happen before the excludes. Try re-ordering and see if that makes a difference.

I'm not familiar with the files within docker addon but one thing to keep in mind is that if the files are in-use or unavailable to the Kodi application due to permissions they'll get skipped as well. I only mention this as the folder you have is called "volumes" - guessing these files might be in use by a process or could be owned by a different user.

[Morphy99]

Thanks for getting back! I'll check the ordering. I'm on Libreeelec so I'm not sure it's a permissions issue as it's all root user. I've just checked and they're 755 drwxr-xr-x not drwx------ weirdly. Not sure that would effect anything.

The volumes are just the config directories of the docker containers but some of them may be in use, but not all. I might have to look at a script to stop them and run your addon.

[Morphy99]

So I tried adding the include of the volumes dir in before the exclude and still didn't work. I also tried just adding the volumes dir on its own and it started to back them up however threw an error on one of the files, I'm guessing as it was in use.

2021-01-10 21:40:10.348 T:2465690480   ERROR: EXCEPTION: Unknown exception thrown from the call "read"
2021-01-10 21:40:10.350 T:2465690480   ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
                                             - NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
                                            Error Type: <type 'exceptions.RuntimeError'>
                                            Error Contents: Unknown exception thrown from the call "read"
                                            Traceback (most recent call last):
                                              File "/storage/.kodi/addons/script.xbmcbackup/default.py", line 87, in <module>
                                                backup.backup()
                                              File "/storage/.kodi/addons/script.xbmcbackup/resources/lib/backup.py", line 171, in backup
                                                filesCopied = self._copyFiles(fileGroup['files'],self.xbmc_vfs,self.remote_vfs)
                                              File "/storage/.kodi/addons/script.xbmcbackup/resources/lib/backup.py", line 415, in _copyFiles
                                                wroteFile = dest.put(aFile,destFile)
                                              File "/storage/.kodi/addons/script.xbmcbackup/resources/lib/vfs.py", line 100, in put
                                                self.zip.writestr(utils.encode(dest),aFile.read())
                                            RuntimeError: Unknown exception thrown from the call "read"
                                            -->End of Python script error report<--

[robweber]

That would be my guess as well. Does this addon actually run docker containers, like as a service or something? I recall a similar problem being reported with other addons that get accessed by system libraries.

[Morphy99]

Yep it runs containers managed by systemd. It's just a bit weird it didn't even try to backup those dirs with the config above. I'll have another play tomorrow. In the meantime I've a couple more questions if you don't mind:

Can a backup be triggered by another way other than in the add-on or via schedule? I.e. something like Web API ?

When restoring does the add-on recreate the directory as it was backed up (removing additional files if they exist) or does it just extract the files backed up keeping anything else that may exist?

[Morphy99]

Is there a file size limit? The error above was whilst reading a large (3.8Gb) db file. I stopped the container so it shouldn't be in use.

[robweber]

I suspect the size of the file is the main problem here. On two fronts really. When calling aFile.read() it attempts to read the entire file into memory. Above you indicated an Rpi4 so I'm guessing like 4GB memory max on the whole board? Kodi does allow for reading a file in chunks, however the Python zip writestr() method requires the entire file dumped so even if I modified the code to read large files in chunks the zip library wouldn't handle it. It looks like there may be a method to support this in Python 3 but I'll have to read up more on that. Looks we're limited by both hardware and software constraints on this one.

To answer your other questions though...... Yes you can trigger a backup with something like the Web API. You can construct a JSON string to trigger Kodi to run an addon. You use curl or some other program to send the POST request. An example is on the wiki.

When restoring the the addon just dumps the files that exist in the archive back to their locations - overwriting versions that may currently exist. If there is a file, say foo.txt that exists in the directory that didn't at the time of backup it will remain unchanged (option 2). No one has ever asked for a purge of the directory first before, is that something you have a use case for? Thinking through the edge cases on something like that I could see potential issues since you're doing this while Kodi is running. Imagine like the entire addon_data directory. If you purge that prior to a restore and while the restore is happening a running addon attempts to read a file it might crash Kodi, killing the restore with it. The file by file restore process negates this possibility somewhat as a file would need to be in use at the exact time it's written to cause an error. Just "wondering aloud" I suppose you could do a diff on the directories and purge old files post-restore to avoid issues from a pre-restore purge of the whole directory.

[Morphy99]

Yep it's only 4gb but I don't think it even attempts to put it in memory from what I can see: That's unless the sensor monitoring the memory doesn't update quickly enough to catch the high usage. Still, I've tried it at least 3 times today so you would have thought it would've caught it once!

I may be switching databases so I'll see if I can find one which uses smaller files but it's not the end of the world if I can't back this up as it's just a history log. What would be useful is a file filter, being able to exclude/include certain files by wildcards?

No one has ever asked for a purge of the directory first before, is that something you have a use case for?

Not really. I'm just thinking in terms of taking snapshots in case a restore back in time is needed. Tbh it's probably more something I might find with a docker container as it's more system operations than just backing up kodi files. I can totally see how things could get messed up running a restore while things are running.

[robweber]

This might be getting beyond my pay grade! I tried going through the C code for the xbmcvfs.File class, which is the underlying structure here being used for the read() method. I found the contents of the read method but nothing is jumping out at me for the cause of the error - ie where the actual file being read is too much for the system. It could even be that it checks for an available memory buffer and bombs out before it even starts. I'd need someone on the Kodi team to confirm any of this since it's core Kodi code at that point.

[Morphy99]

Sorry! I appreciate the help and if something is too big of an ask just say. I didn't realise that this was using Kodi's file system operations, I guess it makes sense seeing as it's a Kodi add-on. Don't worry about the error, I'll find another way.

I take it there isn't a filter for individual files then?

[robweber]

Trying to push the boundary of what it can currently do is a good way to make sure the addon keeps improving. A lot of use cases others have are things that never occurred to me using my own setup. A lot of the low level file ops are handed off to Kodi libs as they are guaranteed to work across platforms. The Kodi devs have already done the heavy lifting on that stuff.

Filtering individual files or using like a wildcard for a directory is a great idea. Currently the matching isn't sophisticated enough for the directory walking function to handle it but I do think it could be layered in. I'll make a separate issue for it as an enhancement.

[Morphy99]

Sounds great! Give me a mention when you create it I'll keep my eye on it :grin:

Going back to this issue so we can close it off, I think I figured it out by creating a separate folder set:

{
  "addon_data": {
    "dirs": [
      {
        "path": "special://home/userdata/addon_data/",
        "type": "include",
        "recurse": true
      },
      {
        "path": "special://home/userdata/addon_data/service.system.docker/",
        "type": "exclude"
      }
     ],
    "root": "special://home/userdata/addon_data/"
  },
  "volumes": {
    "dirs": [
      {
        "path": "special://home/userdata/addon_data/service.system.docker/docker/volumes/",
        "type": "include",
        "recurse": true
      },
      {
        "path": "special://home/userdata/addon_data/service.system.docker/docker/volumes/HASS/",
        "type": "exclude"
      }
    ],
    "root": "special://home/userdata/addon_data/service.system.docker/volumes/"
  }
}

I excluded the HASS folder as it had the offending db file in it. Weird thing now is the folder is created as "volumes\olumes" in the zip file :open_mouth:

[robweber]

The volumes name might be an error - not sure. I'll have to see if I can reproduce that. For now I'll close this and have created a new issue for the wildcard filtering.