Serve books from network share

[wenestvedt]

My HA server has a small internal drive, and my books are stored on a NAS.

Can the HA Add-On gain the ability to read the metadata.db and e-book files from a share (SMB or NFS) instead of copying the files locally?

Thanks!

[dunxd]

Currently the add-on works like this when it is initialised:

1. If it doesn't already exist, create a directory at /media/books on the HA hosting the add-on
2. Create a symlink to /media/books called library inside the add-on container - this link is used by the add-on to access the book files and metadata.db

I haven't tried it, but you could delete the symlink at books in /media on your HA server (if the add-on already made it) then mount an NFS or SMB share at /media/books then I believe it will work. This doc looks like it might be helpful if you aren't sure how to mount an NFS share in the media folder in Home Assistant. I would be really interested to hear whether this works or not.

Another option, which I could add to a future version of the add-on, is to make the path to where the Library is stored configurable, allowing you to specify a NFS or SMB path - and have the default as/media/books. I don't have an NFS or SMB server to easily test this against, but could make a test release to allow you to try it. What do you think?

[wenestvedt]

Dang, I never even thought to try that!! Thank you!

My Synology NAS has a share named "calibre" which I used for COPS in the past (until the DSM7 upgrade killed it) -- so it has a full set of book files & catalog files. I maintain the library on my Mac and rsync it up to the Synology for access around the house.

I followed the directions in the linked document, and used this command:

shell_command:
  mount_nas_folder: mkdir -p /media/books;mount -t cifs -o vers=3.0,noserverino,username=USERNAME_GOES_HERE,password=PASSWORD_GOES_HERE,domain=WORKGROUP //MY.IP.ADDRESS.HERE/calibre /media/books

Viola! Now in my HA server's Media window, I see four folders named "Camera," "Local Media," "Synology Photos," and "Text to Speech." (Only the first one was there before this.) Inside "Local Media" there is a folder named "Books," which is where I told HA to mount the Synology's export. All the e-book folders are there!

Unfortunately, launching HA COPS produces the "COPS Configuration Check" web page which complains:

Check if Calibre database file exists and is readable

File ./library/metadata.db not found, Please check Value of $config['calibre_directory'] in config_local.php (Does it end with a '/'?) Value of open_basedir in your php.ini The access rights of the Calibre Database Synology users please read this Note that hosting your Calibre Library in /home is almost impossible due to access rights restriction

This is familiar from years ago when I installed COPS directly on the Synology. Do I need to somehow edit the PHP setup files, like I did back then? I am mounting the share with an account with r/w permissions to the files, so that's probably not it.

[wenestvedt]

The error specifically says File ./library/metadata.db not found -- but I don't have a subdirectory within /calibre named "library" -- it's just the metadata.db and folders full of books. Assuming that this this path cannot be specified somewhere, I changed the mount command to go to /media/books/library -- but it doesn't help.

Is this related to the fact that I can only see folders and cover images (.jpg files), but no book files (neither .epub nor .pdf nor .anything else) when I browse via the HA web interface's Media page?

[dunxd]

The reference to ./library/ is from the perspective of PHP running in the add-on container. This is symlinked to /media/books/ in the HA host. COPS expects to find metadata.db in ./library/ - i.e. on the HA host you should be able to get to /media/books and find metadata.db in that folder.

Note that the web front end for HA only shows folders and audio/video files - but all the files should show up if you ssh in. Can you see metadata.db in /media/books/ via SSH?

If not, maybe try adding a / to the end if the command where you map the share:

shell_command:
  mount_nas_folder: mkdir -p /media/books;mount -t cifs -o vers=3.0,noserverino,username=USERNAME_GOES_HERE,password=PASSWORD_GOES_HERE,domain=WORKGROUP //MY.IP.ADDRESS.HERE/calibre /media/books/

[dunxd]

I just read your message again, and can see that the books are visible in /media/books, so the command looks to have done what was intended. Check to see if metadata.db is in there.

[dunxd]

Another thing to be aware of - if you load the add-on in your browser and there is an error, the URL ends in /checkconfig.php?err=1. That ?err=1 triggers an error message at the top of the page. If you refresh that checkconfig.php page with err=1 still in the URL it will say there is an error even if you just fixed the problem! This caught me out a few times when setting up. Room for improvement!

Always try going to the root of the site when troubleshooting (e.g. http://hassio.home.arpa:8000).

[wenestvedt]

I just read your message again, and can see that the books are visible in /media/books, so the command looks to have done what was intended. Check to see if metadata.db is in there.

Two things: from a Terminal window, no books at all are visible when I use ls -- but in the web interface, clicking on "Media" in the sidebar shows me a "Local Media" folder that then contains the "books" folder and all its subdirectories. Weird.

Second, here's the latest HA COPS log:

[Wed May 10 11:40:01 2023] 172.30.32.2:37428 [302]: GET /index.php - Uncaught Exception: Database <> not found. in /cops/lib/Base.php:135
Stack trace:
#0 /cops/lib/Base.php(151): Base::error()
#1 /cops/lib/Base.php(165): Base::getDb()
#2 /cops/index.php(31): Base::checkDatabaseAvailability()
#3 {main}
  thrown in /cops/lib/Base.php on line 135

Even when looking at the page itself, not the error page, in a browser. So that ain't good. :7(

(Thanks for your help!)

[dunxd]

That is pretty odd, but searching around it appears there are some complications with accessing a CIFS share from a docker container (which both HA COPS and the SSH add-on are). That might explain why you can navigate to the folder and view contents from within HA but not via SSH. I suspect that the same issue would occur with NFS.

It would be cool to figure out how to get this working, but I wonder if you would be better served by installing COPS directly on your NAS. There is documentation on how to get the version compatible with PHP 5-7 on a Synology NAS at https://github.com/seblucas/cops/wiki/Howto---Synology.

If you want/need to use PHP 8 and have a nicer interface, you could download the updated version of COPS as used in this add-on from https://github.com/mikespub-org/seblucas-cops/releases and adapt the instructions for the newer filenames.

If you want to help me figure out how to get this working in HA add-on that would be awesome - do let me know if that is the case!

[wenestvedt]

If you are game, I would love to help figure this out!

I agree that the two Docker Containers aren't allowed to see each others' storage -- which is why the one for Terminal & SSH cannot see the mounted storage. (And also why I got "permission denied" errors when trying to mount that Synology share from within the Terminal & SSH session.)

I left all the old COPS files on the Synology when I upgraded to DSM7, so simply fixing that is my fallback plan -- but this is a fun puzzle!

Probably we need to get the mount made by the Supervisor before the Containers are launched. (Do I have that right?)

[dunxd]

I think the way to do it is to add cifs-utils to the packages installed in the Dockerfile (i.e. add cifs-utils after rsync on line 8 of dockerfile.)

Then you should be able to change some lines in run.sh to create the share instead of symlinking to /media/books. E.g. replace lines 20-23

mkdir -p /media/books
cd /cops || return
# Create a link to the books directory called library
ln -s /media/books library

With

cd /cops || return
mkdir library
mount -t cifs -o username=<win_share_user> //WIN_SHARE_IP/<share_name> library

I might not have the mount command right as I can't test.

Are you able to fork my repo and try it in your HA with the above changes?

[wenestvedt]

Are you able to fork my repo and try it in your HA with the above changes?

I haven't before, so this is a perfect time to learn! (I happen to have a comp.sci. major handy, who just returned from college for the summer, and I will ask him for help when I get stuck.)

[dunxd]

Great. Fork my whole repo, make some changes to repository.yaml in the root so you can tell the difference in HA, then add the new fork's URL to your repositories in HA.

Your repository with any changes you make to the COPS add-on should then be available for installation. You should uninstall my version of the add-on as they will clash with ports etc.

One thing to note is to change the version: in COPS/config.yaml after making changes you want to try out. If you don't do that HA won't offer to update the add-on after you make a change.

I have some ideas about how to change the add-on to allow specifying where Calibre library files are, including network shares, so let me know if you get it working.

On Thu, 11 May 2023, 2:32 pm wenestvedt, @.***> wrote:

Are you able to fork my repo and try it in your HA with the above changes?

I haven't before, so this is a perfect time to learn! (I happen to have a comp.sci. major handy, who just returned from college for the summer, and I will ask him for help when I get stuck.)

— Reply to this email directly, view it on GitHub https://github.com/dunxd/HomeAssistantAddons/issues/10#issuecomment-1544018632, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD6W2HP2ECZDGIRJNNRPWVTXFTS6ZANCNFSM6AAAAAAX3ZRC3I . You are receiving this because you were assigned.Message ID: @.***>

[wenestvedt]

Almost done. This is fun!

Is there a way to do a prompt for the server IP address & credentials, or a config file -- or should I just hard code in some "anonymous" local user creds (to avoid disclosing my actual -- albeit local -- credentials)?

• Will

On May 10, 2023, at 4:55 PM, Duncan Drury @.***> wrote:

I think the way to do it is to add cifs-utils to the packages installed in the Dockerfile (i.e. add cifs-utils after rsync on line 8 of dockerfile.)

Then you should be able to change some lines in run.sh to create the share instead of symlinking to /media/books. E.g. replace lines 20-23

mkdir -p /media/books cd /cops || return

Create a link to the books directory called library

ln -s /media/books library With

cd /cops || return mkdir library mount -t cifs -o username= //WIN_SHARE_IP/ library I might not have the mount command right as I can't test.

Are you able to fork my repo and try it in your HA with the above changes?

— Reply to this email directly, view it on GitHub https://github.com/dunxd/HomeAssistantAddons/issues/10#issuecomment-1542797736, or unsubscribe https://github.com/notifications/unsubscribe-auth/AI7PDHMUM225UH5VXDFMRWTXFP6CXANCNFSM6AAAAAAX3ZRC3I. You are receiving this because you authored the thread.

[dunxd]

It is possible to set up config variables that appear in the configuration page for the Add-on - you set them in config.yaml.

You can use those variables in run.sh - they are available like this:

$(bashio::config 'variable_name')

[wenestvedt]

I actually have the four variables working now. Unfortunately, I use a Synology NAS, and I am finding that connecting to it is...tricky.

The NAS reports in its logs that the HA host connected, but the COPS code returns no data when I load up its URL.

Is there a way to increase the verbosity of the logs created by the COPS PHP code?

• Will

[dunxd]

I don't think so - it uses the PHP built in web server and you are limited to the options detailed in the docs. I don't see any that increase the verbosity of the logs, but I could be mistaken.

I included an info.php file - if you go to https://cops_url/info.php you can see all the PHP settings and extensions included. I don't know if that is of any help, as this doesn't run any other code. If it doesn't output anything, then it may be that your run.sh code has an error that stops the code before it reaches the php -S command.

Another place to look is the supervisor logs in HA. These may show any errors that occur during initialisation of the add-on.

[wenestvedt]

The single thing in the Supervisor log is "permission denied," which is worthless -- especially when the mount command in configuration.yaml works fine to put the NAs share onto the HA host...just at the wrong mount point!

The big stumbling block right now is that I don't know what I am doing. :7) The second-biggest one is that the mounted folder appears in "/Media/Local Assets" instead of right under "/Media" -- and I don't know how to either get that "Local Assets" out of the path or else convince COPS to look there instead.

[dunxd]

Note that in the Web Interface of Home Assistant when you click on Media in the left menu, it takes you to an interface where you can browse different sources. The "Local Media" "folder" in here maps to /media on the HA host.

I just took a look at your Fork. You aren't creating your share in /media/library on the host server. You are creating it in /cops/library in the add-on container. This is actually a good thing, as there is no need for the add-on to read /media just to get to your share. But you won't see any files in /media on the host.

This leads me to suspect that you might want to check the permissions for the share on the NAS. Are you able to read the files using the same credentials?

This could be the blind leading the blind - this is my first add-on for HA, the first time I am using Docker containers for anything, I've never configured SMB and I haven't coded anything in PHP since version 5!

[dunxd]

I have got SMB on my development machine now, so can look a little deeper. I tried setting up the share on the HA host in /media/books and was able to browse it via the Media > Local Media web interface, but not from command line - it just showed as an empty directory. I suspect this is because the command line and add-on are running in child containers.

I then took the approach you took in your fork of creating the mount within the fork using cifs-utils. I changed the command slightly to:

mount -t cifs -v -r -o username=$WIN_USER_NAME,password=$WIN_PASSWORD "//$WIN_SHARE_IP/$WIN_SHARE_NAME" library

This now throws the following error message in the add-on logs:

Unable to set current process capabilities: Operation not permitted
mount: permission denied (are you root?)

So it looks like containers aren't allowed to access a share on their host, and they aren't allowed to create their own shares. I'm trying to get some assistance on how to get this working - given that plenty of HA users are running off tiny SD cards, there has to be a way to allow CIFS/SMB/NFS share access to add-ons!

[dunxd]

Bingo! Add this to your config.yaml:

privileged:
  - SYS_ADMIN

This allows mounting within the container, and for me it is now working with the configuration options you added in your fork.

Not my favourite way of doing it, but should get you up and running.

If it is ok with you, I would like to add your fork as an extra add-on in my repository, so people can choose between addons that use SMB and local storage of their book files.

[dunxd]

Hey @wenestvedt I noticed that today's release of Home Assistant allows mounting of NAS directories into Home Assistant, for use as a media source etc. If you find this works for this Add-on, please let me know. It would be better to make use of built in HA features rather than create a separate Add-on dedicated to using NAS as a source. Cheers!

[wenestvedt]

I saw the same thing, so I applied the upgrade. I will try agaim to get the book share working.Would be great if this works!- WillOn Jun 7, 2023, at 5:09 PM, Duncan Drury @.***> wrote: Hey @wenestvedt I noticed that today's release of Home Assistant allows mounting of NAS directories into Home Assistant, for use as a media source etc. If you find this works for this Add-on, please let me know. It would be better to make use of built in HA features rather than create a separate Add-on dedicated to using NAS as a source. Cheers!

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

[dunxd]

Just checked, and if setting up a network storage with Usage set to Media it exposes it as a directory in /media. However I wonder whether this will be accessible from the add-on, given the issues we found. Would be very grateful if you share what you find!

If it does work, would you be able to try without the SYS_ADMIN privileged setting too?

[dunxd]

I think we are on as I can access the folder from the SSH add-on which didn't work with the previous mounted share if I recall correctly.

I haven't been able to test with this add-on yet as I need to upgrade my test environment to 2023.6 and last time that turned into a lengthy exercise I don't have time for today. Fingers crossed it works as it will then be easier to make a single add-on version that works with both CIFS and local storage.

[dunxd]

Ideas for version that supports the new CIFS support in Home Assistant:

Toggle to enable rsync

If enabled, install rsync (in run.sh), create /media/books start rsync server If disabled rsync port setting is ignored?

Allow configuration of library path/directory name?

Initially require that Network Storage is set up with name books and Usage selected is Media. Need good documentation.

[dunxd]

@wenestvedt I have released a new version which works with network shares. I actually needed to change very little - it appears the way that shares are mounted allow them to be accessed from add-ons now. Much simpler. Do let me know if you try it and if you find any errors! Cheers!