Hardware mismatch, the maximum of 2 computer hardware_id's is already registered

[alno74d]

Expected Behavior

No hardware mismatch, Hardware ID should not change across docker service restarts.

Current Behavior

Error: _Hardware mismatch, the maximum of 2 computer hardwareid's is already registered. It looks like every time the docker service is restarted, webgrabplus detects a new hardware id whereas the container was not recreated. After 2 new IDs registered, the third hardware registration gets queued for a 12 hours, preventing us from using the service if the docker server is restarted every day for instance.

Steps to Reproduce

1. Setup container properly, run guide update, xmltv guide OK
2. Stop docker service and start back docker service
3. Run guide update, xmltv guide OK
4. Stop docker service and start back docker service
5. Run guide update, issue with license, here is the log:

   me:~# cat WGLicense.log.txt
   [        ] WebGrab++ license request and update log
   [        ] 2020/07/23 17:43
   [  Info  ] New copy of 'custom license' data downloaded
   [        ]
   [  Info  ] License request/update
   [  Info  ] Username '********'
   [Error   ] Hardware mismatch, the maximum of 2 computer hardware_id's is already registered
   [        ]
   [Warning ] The program cannot run right now on this computer,
   [Warning ] but it's hardware_id will be ADDED and ready to use after 2020/07/24 05:43 ,
   [Warning ] and it will REPLACE one of the previously registered computers as soon as you run on the ONE YOU WANT TO KEEP !!
   [  Info  ] Custom License status : 'True'
   [  Info  ] The program will run with performance settings : 'custom'
   [  Info  ] License check .. done
   [        ]
   [        ] Actual settings of the custom license :
   [        ]
   [        ] License   started   on :2020/05/09
   [        ] License will expire on :2021/03/07
   [        ]
   [        ] channels/ini      1000
   [        ] channels total    1000
   [        ] siteinis          100
   [        ] decryption keys   enabled
   [        ] decryption mode   new (V3) & (V2)
   [        ] index only        no
   [        ] postprocess MDB   enabled
   [        ] postprocess REX   enabled
   [        ] debug             False
   [        ] show details *    full
   [        ] update mode       all
   [        ] channel delay     0 secs
   [        ] index delay       0 secs
   [        ] show delay        0 secs
   [        ]
   [        ] * showdetails : 'tt = times & title, 's' = subtitle, 'd' = description
   [        ]
   [        ] Hardware_ID matching **       enabled
   [        ]
   [        ] ** 'enabled' : allows to run on 2 computers max, 'disabled' : allows to run on any computer

Environment

OS: Ubuntu 20.04 CPU architecture: x86_64 How docker service was installed: From Docker’s repositories

Command used to create docker container (run/create/compose/screenshot)

version: "2.3"
services:
  webgrabplus:
    image: linuxserver/webgrabplus
    container_name: webgrabplus
    environment:
      - PUID=***
      - PGID=***
      - TZ=Europe/Berlin
    volumes:
      - /webgrabplus/config:/config
      - /webgrabplus/data:/data
    restart: unless-stopped

Docker logs

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 01-envfile: executing...
[cont-init.d] 01-envfile: exited 0.
[cont-init.d] 10-adduser: executing...
usermod: no changes

-------------------------------------
          _         ()
         | |  ___   _    __
         | | / __| | |  /  \
         | | \__ \ | | | () |
         |_| |___/ |_|  \__/

Brought to you by linuxserver.io
-------------------------------------

To support LSIO projects visit:
https://www.linuxserver.io/donate/
-------------------------------------
GID/UID
-------------------------------------

User uid:    ***
User gid:    ***
-------------------------------------

[cont-init.d] 10-adduser: exited 0.
[cont-init.d] 30-config: executing...
******** Please use the file wg3-cron to adjust the scheduled time for running WebGrab++. wg-cron can now be deleted. ********
[cont-init.d] 30-config: exited 0.
[cont-init.d] 99-custom-scripts: executing...
[custom-init] no custom files found exiting...
[cont-init.d] 99-custom-scripts: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.

[tobbenb]

I didn't know they tied anything to hardware id. Most likely there is not much we can do with this container change how docker works. Have you checked if there are any options in docker to set a persistent hardware id?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[t-chab]

According to http://www.webgrabplus.com/content/docker-and-hardwareid you should be able to workaround this by using h instead of f for license settings :

<license wg-username="[your_username]" registered-email="[your_email]" password="[your_password]">h</license>

[tobbenb]

Sorry, I haven't been much online lately and have completely forgotten about this issue. If setting it to h works, then I guess the easiest would be to add it in the defaul XML file.

[t-chab]

@tobbenb : no need to be sorry, and thanks for all the great work you make on this project.

Unfortunately, my first tries are not successful, I switch back to f as default value for tag

I'll try to get some support from webgrabplus forums, I'll let you know.

[t-chab]

@tobbenb : according to WebGrabPlus website, a new beta is available

Changelog say it improves license force mode handling.

Is it possible to publish this beta with a beta tagged image ?

[tobbenb]

Is this fixed by the webgrabplus+ update? If not, please reopen the issue.

[pbathuk]

Hi, I'm commenting on this as the issue still persists. The latest version of the beta is 3.1.8 which still has the same issue - if you restart the container as new hardware_ID is created.

I am not sure what it being used as the hardware_ID - and am going to ask. As until it is known what variable is being used it will be impossible to find a work around.

[tobbenb]

Isn't that issue solved by using the h option in the config file?

[pbathuk]

Yes and no. If I change the config XML to have h it will run it once, however it is then changed, by the program, to have a To force a license update; replace this text with the letter f So the next reboot of the PC / docker container will mean that the fix has to be reapplied.

I am hoping a docker command or a change in the container could link what is being used to that of the actual PC, as from what I gather from the forum non-docker users don't have the same issue.

[tobbenb]

If webgrab change it to f after it has run it's something you have to ask them to fix. The same goes for what they use for fingerprint.

[Bl00d-B0b]

Hello,

Using webgrab plus version V3.2.2.0 still have the same issue:

1. with 'f' works only when you delete wglocal.lic file. then next time you see in .xml config: To force a license update; replace this text with the letter f
2. with 'h' works without deleting wglocal.lic file, but next time you still see in .xml config: To force a license update; replace this text with the letter f

I can't say it's fixed, as you need to edit every time .xml conifg file before run

maybe someone have a solution?

[ufo56]

Hello,

Using webgrab plus version V3.2.2.0 still have the same issue:

1. with 'f' works only when you delete wglocal.lic file. then next time you see in .xml config: To force a license update; replace this text with the letter f
2. with 'h' works without deleting wglocal.lic file, but next time you still see in .xml config: To force a license update; replace this text with the letter f

I can't say it's fixed, as you need to edit every time .xml conifg file before run

maybe someone have a solution?

Have you found solution ?

[Bl00d-B0b]

No. Waiting for new update

[argoncoast]

Hello,

Using webgrab plus version V3.2.2.0 still have the same issue:

1. with 'f' works only when you delete wglocal.lic file.  then next time you see in .xml config: **To force a license update; replace this text with the letter f**

2. with 'h' works without deleting wglocal.lic file, but next time you still see in .xml config: **To force a license update; replace this text with the letter f**

I can't say it's fixed, as you need to edit every time .xml conifg file before run

maybe someone have a solution?

Doesn't this imply that wg itself is altering the .xml config and that the solution is for them to change the behaviour upstream? Only way around it I can think of within the docker container itself would be some hacky sed command or deleting the .lic file.

EDIT: I've posted on the webgrabplus forum to get the upstream dev involved.

[tobbenb]

As far as I know, webgrab uses the Mac address for the license, so setting a Mac for the container should make the license persistent through updates.

[t-chab]

For what it's worth, I use the following snippet to force license renewal as a workaround :

xmlstarlet ed -L -u '//license' -v 'f' WebGrab++.config.xml

[argoncoast]

@tobbenb Yeah, that's the conclusion I've come to from testing and discussing with upstream. I just testing the possibility of using host networking as as far as I know docker doesn't check if a mac address is already allocated when you manually set one which I can potentially see some issues with if people have a big stack.

@t-chab I did something similar albeit using grep & sed and a bash script, which worked fine but discussing with upstream they suggest this may be a temporary measure, and it does have the side effect of hitting their servers each time, which for one or two of us probably wouldn't be an issue but if it's a documented or implemented method in the image itself for all installs could potentially be an issue.

[aptalca]

at least try setting a hostname for the container. That's a very common thing used in tracking hardware

[argoncoast]

@aptalca That's another one I'll add to my list to try if I need to.

I think so far that two things are relevant and I seem to have got things to a state where I'm not running into license issues.

1. Running as host (the alternative being setting a MAC address, I've decided that from a deployment point of view it's probably easier to suggest running as host as there is an edge case where docker doesn't check if a MAC address is allocated to another running container when you're setting a MAC which could lead to issues, whereas, security implications aside, as there's no port for webgrabplus running as host would be a quick easy and foolproof fix for deployment.

2. It's been suggested by upstream that UID/GID are relevant which I initially dismissed, however I think I ran into an issue where the cron job was running (as abc and when I opened a terminal in the contaner (as root) I then failed a licence check. Now webgrabplus has a 2 hardware ID limit, so for most cases this should not present itself, as people will run a container and have the cron job run by abc and if they trigger a manual update as root they'll only put themselves a the two machine limit, but I was running two or three instances at the time to try and delve into this a bit more.

I'm hoping that the webgrabplus author replies at some point and just tells us what the licence is derived from as that would put an end to this.

I have also found that by using a license.shscript in /config and running that in cron also works, but it does have the side effect of forcing a licence check each time it's run so not ideal and should they rate limit it up stream it could result in issues.

20 3 * * * /bin/bash /config/license.sh && /bin/bash /defaults/update.sh

#!/bin/bash

if 
  grep -q "To force a license update; replace this text with the letter f" /config/WebGrab++.config.xml 
then
  sed -i 's/To\ force\ a\ license\ update\;\ replace\ this\ text\ with\ the\ letter\ f/h/g' /config/WebGrab++.config.xml && echo "WebGrab++.config.xml updated with "h" flag"
else
  echo "WebGrab++.config.xml does not require update"
fi

[argoncoast]

So had a reply back from the upstream, not entirely clear but, it appears there are three things that are required:

1. A consistent MAC address - I'd advocate for clarity running as host to reduce support burden here.
2. A consistent hostname - this I believe is already covered in the readme.md as container_name and hostname I believe is analogous in docker-compose, and I would assume in docker as well. Running hostname from a terminal within my container returns my host's hostname, so I think on either front it's fine. (Note this is running the container with host networking, however, running it as bridge returns a container ID, I think this is a further good reason to run as host as it removes the need to set both a MAC address and hostname during deployment.
3. A consistent user is required. Again as the update process is run as abc this should be covered, I suppose something in the readme to explain how to run a manual update as the abc user instead of root might be a good idea.

   docker exec --user abc webgrabplus /defaults/update.sh

It doesn't look like the image itself requires any modification and the root cause is definitely lack of clarity on what factors influence a hardware check and understanding from upstream about the particulars of running in a containerised environment.

In summary, a simple change to recommend running as host network fixes the first two issues and is the simplest answer. The alternative would require everyone to set a MAC address and hostname during deployment. The disadvantages of this approach I think are an increased support burden as the MAC address needs to be from within the docker networking subnet and there's no check made by docker that a MAC address isn't already in use, using a random web based MAC generator is going to result in failed deployments.

Point 3, is an edge case as one would assume most of the docker image users are running webgrab solely on a single machine, so wouldn't run into issues should they trigger a manual update as the root user, but for the sake of covering all bases it's probably sensible to mention this.

Would you like me to submit a PR to readme-vars.yml?

[aptalca]

Docker allows you to set the hostname via the hostname directive. Without it, the container id (randomly generated every time the container is recreated) is set as the hostname if using bridge networking.

[argoncoast]

@aptalca agreed, however this is already mitigated by suggesting running as host.

root@HomeNAS:~# docker exec -it webgrabplus hostname
HomeNAS

[aptalca]

What I'm trying to say is, host networking is a bit of a sledgehammer.

All those can be accomplished with bridge networking by setting a hostname and a mac address. The container already uses the same user abc for the operations. We just need to add those vars and a blurb about not changing the puid/pgid.

[argoncoast]

@aptalca Yeah, I agree that running as bridge is a more secure approach, and it's one I am doing myself, I did consider both approaches but thought that host would be an easier method for end users to deploy and would reduce your support burden.

Maybe I'm overthinking it, worst that can happen is either the MAC address parameter is invalid and the container fails to deploy or the slim chance there's a duplicate MAC address in a docker stack and some weird and wonderful issue arises.

I'm sure either of these will seek some help.

It seems you're favouring the bridge approach, and it's entirely understandable why, so given we have a decision I'll submit a PR based on that.

Thanks for the input.

[argoncoast]

Sorry, I'm not sure I can submit a PR that will cover this, looking at your jenkins parameters I don't think there's a mac_address parameter to use in the readme-vars.yml (hostname is covered)

https://github.com/linuxserver/docker-jenkins-builder/blob/master/roles/generate-jenkins/defaults/main.yml

[ufo56]

Just add "--mac-address 02:42:ac:11:ff:ff" and license issue is fixed.

[argoncoast]

Just add "--mac-address 02:42:ac:11:ff:ff" and license issue is fixed.

I've fixed the license issue, I was just trying to submit a PR to improve the documentation for everyone else, but from looking at the link above I don't think the CI which generates the readme currently supports adding the mac-address parameter.

Since my last post had some communication back from the webgrabplus team. They've shared with me a snippet of the license and it's actually made up of three elements.

hardware_id ${USER}#${HOSTNAME}:${MAC} so mine came back as abc#webgrabplus:00:00:00:00:00:00

So it's not as simple as just adding the mac address, it also requires the hostname adding. It also incorporates an email/password check thankfully, otherwise a collection of docker users using a copy/paste would appear as a single user!

For reference

docker run -d \
  --name=webgrabplus \
  --mac-address=00:00:00:00:00:00 \
  --hostname=webgrabplus \
  -e PUID=1000 \
  -e PGID=1000 \
  -e TZ=Europe/London \
  -v /path/to/config:/config \
  -v /path/to/data:/data \
  --restart unless-stopped \
  lscr.io/linuxserver/webgrabplus

---
version: "2.1"
    webgrabplus:
        container_name: webgrabplus
        mac_address: 00:00:00:00:00:00
        hostname: webgrabplus
        environment:
            - PUID=1000
            - PGID=1000
            - TZ=Europe/London
        volumes:
            - /path/to/config:/config
            - /path/to/data:/data
        restart: unless-stopped
        image: lscr.io/linuxserver/webgrabplus

And to ensure that there's a consistent user running webgrabplus a manual update should be run as the abc user:

docker exec --user abc webgrabplus /defaults/update.sh

[tobbenb]

@argoncoast It's possible to make a custom compose example like here: https://github.com/linuxserver/docker-bookstack/blob/master/readme-vars.yml#L41

[argoncoast]

@tobbenb OK, no problem on the compose front, any ideas how to tackle the docker run command?

[tobbenb]

Not really. I have to ask the experts on the Jenkins stuff, but it's probably not possible without adding it to the Jenkins-builder. It will probably be the same for the parameters section. So it might be best to add a note in the setup part about it. The compose part will have to do for now I think.