Invalid Character Error Following "Sending detected devices to API, for filtering & validation"

[Majoraslayer]

I'm using the omnibus Docker container on Ubuntu 23.04 to monitor several drives. When running the command to refresh drive information, the script halts with mention of an invalid character. As this character doesn't exist anywhere in my settings, I suspect this may be a bug in the script. Below is the activity log in question:

---

AnalogJ/scrutiny/metrics dev-0.7.1

time="2023-09-13T11:00:01Z" level=info msg="Verifying required tools" type=metrics time="2023-09-13T11:00:01Z" level=info msg="Executing command: smartctl --scan --json" type=metrics time="2023-09-13T11:00:01Z" level=info msg="Executing command: smartctl --info --json /dev/sdj" type=metrics time="2023-09-13T11:00:01Z" level=info msg="Using WWN Fallback" type=metrics time="2023-09-13T11:00:01Z" level=info msg="Executing command: smartctl --info --json /dev/sdk" type=metrics time="2023-09-13T11:00:01Z" level=info msg="Using WWN Fallback" type=metrics time="2023-09-13T11:00:01Z" level=info msg="Executing command: smartctl --info --json /dev/sdn" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Generating WWN" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Executing command: smartctl --info --json /dev/sdo" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Using WWN Fallback" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Executing command: smartctl --info --json /dev/sdq" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Using WWN Fallback" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Executing command: smartctl --info --json /dev/sdp" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Using WWN Fallback" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Executing command: smartctl --info --json /dev/sdb" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Generating WWN" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Executing command: smartctl --info --json /dev/sdf" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Generating WWN" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Executing command: smartctl --info --json /dev/sdi" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Generating WWN" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Executing command: smartctl --info --json /dev/sdl" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Using WWN Fallback" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Executing command: smartctl --info --json /dev/sdm" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Using WWN Fallback" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Executing command: smartctl --info --json /dev/sdc" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Generating WWN" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Executing command: smartctl --info --json /dev/sdg" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Generating WWN" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Executing command: smartctl --info --json /dev/sdr" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Generating WWN" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Executing command: smartctl --info --json /dev/sds" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Using WWN Fallback" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Executing command: smartctl --info --json --device nvme /dev/nvme1" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Using WWN Fallback" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Executing command: smartctl --info --json /dev/sda" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Generating WWN" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Executing command: smartctl --info --json /dev/sdd" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Generating WWN" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Executing command: smartctl --info --json /dev/sde" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Generating WWN" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Executing command: smartctl --info --json /dev/sdh" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Generating WWN" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Executing command: smartctl --info --json --device nvme /dev/nvme0" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Using WWN Fallback" type=metrics time="2023-09-13T11:00:02Z" level=info msg="Sending detected devices to API, for filtering & validation" type=metrics 2023/09/13 11:00:03 ERROR: invalid character '<' looking for beginning of value

[AnalogJ]

"invalid character '<' looking for beginning of value" usually means you're getting a HTML response instead of a JSON response from the API.

What does your network/reverse proxy setup look like? Did you customize the configuration to support a custom domain?

[Majoraslayer]

I have my Nginx reverse proxy set up to use a subpath, such as mydomain . org/s07 (without spaces). My collector.yaml is configured with a custom endpoint:

api: endpoint: 'http://192.168.1.40:880/s07'

Note that the 880 pot is intentional here, as I remapped it with my docker command because 8080 was already in use.

EDIT: Basepath in scrutiny.yaml is also set to /s07

[AnalogJ]

Yeah, that's your issue. In omnibus mode, the collector is running within the container, so you should be using the internal port information for the service (not the externally accessible host:port combination).

Try http://localhost:8080

[Majoraslayer]

Try http://localhost:8080

Tried this, and I got the same error again. I also tried substituting localhost with 127.0.0.1 just in case, but I still get that same error.

[AnalogJ]

would you be willing to share both your collector.yaml and scrutiny.yaml config files with me? Email me at jason@thesparktree.com

[Majoraslayer]

Any luck, following our email correspondence?

[Majoraslayer]

After months of struggling with this, I found the source of the problem! The issue is specific to using the collector.yaml and scrutiny.yaml config files to set things up on a reverse proxy.

If you set a custom basepath in scrutiny.yaml you have to uncomment the api section of the collector.yaml file and change the endpoint to reflect this basepath.

The problem is that the example endpath given in the the example collector.yaml file isn't formatted correctly. It suggests:

endpoint: 'http://localhost:8080/custombasepath'

The correct setting should be:

endpoint: 'http://localhost:8080/custombasepath/'

Leaving off the forward slash on the address causes this error. Simple fix, the example.collector.yaml file provided just needs to be updated to reflect the correct way to set this.

[AnalogJ]

Strange, there's been code to normalize the endpoint since v0.4.0

Though, now that I think about it, it only applys to endpoints that are overridden via CLI flags. Good catch, sorry about this.

[Moret84]

I have a similar issue. I am using swag as a reverse proxy. I have set up scrutiny omnibus image to hide behind authelia auth. I want to be able to post data from other server too. So I tried to set a custom subpath that would make scrutiny available through basic auth. I can't understand why I get ERROR: invalid character '<' looking for beginning of value. Here is my nginx conf

## Version 2023/05/31
# make sure that your scrutiny container is named scrutiny
# make sure that your dns has a cname set for scrutiny

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name scrutiny.*;

    include /config/nginx/ssl.conf;

    client_max_body_size 0;

    # enable for ldap auth (requires ldap-location.conf in the location block)
    #include /config/nginx/ldap-server.conf;

    # enable for Authelia (requires authelia-location.conf in the location block)
    include /config/nginx/authelia-server.conf;

    # enable for Authentik (requires authentik-location.conf in the location block)
    #include /config/nginx/authentik-server.conf;

    location / {
        # enable for ldap auth (requires ldap-server.conf in the server block)
        #include /config/nginx/ldap-location.conf;

        # enable for Authelia (requires authelia-server.conf in the server block)
        include /config/nginx/authelia-location.conf;

        # enable for Authentik (requires authentik-server.conf in the server block)
        #include /config/nginx/authentik-location.conf;

        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;
        set $upstream_app scrutiny;
        set $upstream_port 8080;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;
    }

    location /collector {
        auth_basic "Restricted";
        auth_basic_user_file /config/nginx/scrutiny-collector-htpasswd.conf;

        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;
        set $upstream_app scrutiny;
        set $upstream_port 8080;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;
    }
}