bakito / adguardhome-sync

🛡️ Synchronize AdGuard Home config to replicas
Apache License 2.0
771 stars 35 forks source link

synch error #3

Closed elRadix closed 3 years ago

elRadix commented 3 years ago

can you please have a look at the error below

2021-04-05T12:57:01.816+0200 ERROR sync sync/sync.go:104 Error getting origin filters {"from": "192.168.1.101:81", "error": "parsing time \"\"\"\" as \"\"2006-01-02T15:04:05Z07:00\"\": cannot parse \"\"\" as \"2006\""} github.com/bakito/adguardhome-sync/pkg/sync.(*worker).sync /root/go/pkg/mod/github.com/bakito/adguardhome-sync@v0.0.6/pkg/sync/sync.go:104 github.com/bakito/adguardhome-sync/pkg/sync.Sync.func1 /root/go/pkg/mod/github.com/bakito/adguardhome-sync@v0.0.6/pkg/sync/sync.go:24 github.com/robfig/cron/v3.FuncJob.Run /root/go/pkg/mod/github.com/robfig/cron/v3@v3.0.1/cron.go:136 github.com/robfig/cron/v3.(*Cron).startJob.func1 /root/go/pkg/mod/github.com/robfig/cron/v3@v3.0.1/cron.go:312

bakito commented 3 years ago

Hi Can you please provide the output of your origin adguard instance?

https:///control/filtering/status

What version of adguard home are you using?

JoshStark commented 3 years ago

I also had this issue when first setting up the sync tool against an existing primary and new (running but empty) replica instance. I found this issue could be caused by bad filter lists (either empty or malformed), but the only way I was able to resolve the issue was to copy the filter section of the primary instance .yaml file over into the replica .yaml file then restart replica. Only then did sync work for filters.

AdGuard Home version: v0.105.2 (primary & replica).

Output of the lists I'm using (via http://:80/control/filtering/status) is:

{
    "enabled": true,
    "interval": 24,
    "filters": [
        {
            "id": 1,
            "enabled": true,
            "url": "https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt",
            "name": "AdGuard DNS filter",
            "rules_count": 37332,
            "last_updated": "2021-04-04T12:23:12+01:00"
        },
        {
            "id": 2,
            "enabled": true,
            "url": "https://adaway.org/hosts.txt",
            "name": "AdAway",
            "rules_count": 8717,
            "last_updated": "2021-04-04T12:23:12+01:00"
        },
        {
            "id": 1600953770,
            "enabled": true,
            "url": "https://raw.githubusercontent.com/PolishFiltersTeam/KADhosts/master/KADhosts.txt",
            "name": "KADhosts",
            "rules_count": 20353,
            "last_updated": "2021-04-04T12:23:13+01:00"
        },
        {
            "id": 1600953771,
            "enabled": true,
            "url": "https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Spam/hosts",
            "name": "FadeMind",
            "rules_count": 73,
            "last_updated": "2021-04-04T12:23:13+01:00"
        },
        {
            "id": 1600953772,
            "enabled": true,
            "url": "https://v.firebog.net/hosts/static/w3kbl.txt",
            "name": "w3kbl",
            "rules_count": 777,
            "last_updated": "2021-04-04T12:23:13+01:00"
        },
        {
            "id": 1600953773,
            "enabled": true,
            "url": "https://v.firebog.net/hosts/Admiral.txt",
            "name": "Admiral",
            "rules_count": 662,
            "last_updated": "2021-04-04T12:23:13+01:00"
        },
        {
            "id": 1600953774,
            "enabled": true,
            "url": "https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt",
            "name": "anudeepND",
            "rules_count": 42296,
            "last_updated": "2021-04-04T12:23:13+01:00"
        },
        {
            "id": 1600953775,
            "enabled": true,
            "url": "https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt",
            "name": "lists.disconnect.me",
            "rules_count": 2701,
            "last_updated": "2021-04-04T12:23:14+01:00"
        },
        {
            "id": 1600953776,
            "enabled": true,
            "url": "https://v.firebog.net/hosts/Easylist.txt",
            "name": "Easylist",
            "rules_count": 11167,
            "last_updated": "2021-04-04T12:23:14+01:00"
        },
        {
            "id": 1600953777,
            "enabled": true,
            "url": "https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext",
            "name": "yoyo",
            "rules_count": 3568,
            "last_updated": "2021-04-04T12:23:14+01:00"
        },
        {
            "id": 1600953778,
            "enabled": true,
            "url": "https://raw.githubusercontent.com/FadeMind/hosts.extras/master/UncheckyAds/hosts",
            "name": "UncheckyAds",
            "rules_count": 10,
            "last_updated": "2021-04-04T12:23:14+01:00"
        },
        {
            "id": 1600953779,
            "enabled": true,
            "url": "https://raw.githubusercontent.com/bigdargon/hostsVN/master/hosts",
            "name": "hostsVN",
            "rules_count": 17483,
            "last_updated": "2021-04-04T12:23:15+01:00"
        },
        {
            "id": 1600953780,
            "enabled": true,
            "url": "https://v.firebog.net/hosts/Easyprivacy.txt",
            "name": "Easyprivacy",
            "rules_count": 6854,
            "last_updated": "2021-04-04T12:23:15+01:00"
        },
        {
            "id": 1600953781,
            "enabled": true,
            "url": "https://v.firebog.net/hosts/Prigent-Ads.txt",
            "name": "Prigent-Ads",
            "rules_count": 3670,
            "last_updated": "2021-04-04T12:23:15+01:00"
        },
        {
            "id": 1600953782,
            "enabled": true,
            "url": "https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt",
            "name": "quidsup",
            "rules_count": 15184,
            "last_updated": "2021-04-04T12:23:15+01:00"
        },
        {
            "id": 1600953783,
            "enabled": true,
            "url": "https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.2o7Net/hosts",
            "name": "add.2o7Net",
            "rules_count": 1286,
            "last_updated": "2021-04-04T12:23:15+01:00"
        },
        {
            "id": 1600953784,
            "enabled": true,
            "url": "https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt",
            "name": "WindowsSpyBlocker",
            "rules_count": 376,
            "last_updated": "2021-04-04T12:23:15+01:00"
        },
        {
            "id": 1600953785,
            "enabled": true,
            "url": "https://hostfiles.frogeye.fr/firstparty-trackers-hosts.txt",
            "name": "frogeye",
            "rules_count": 56977,
            "last_updated": "2021-04-04T12:23:18+01:00"
        },
        {
            "id": 1600953786,
            "enabled": true,
            "url": "https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareHosts.txt",
            "name": "DandelionSprout",
            "rules_count": 326,
            "last_updated": "2021-04-04T12:23:18+01:00"
        },
        {
            "id": 1600953787,
            "enabled": true,
            "url": "https://osint.digitalside.it/Threat-Intel/lists/latestdomains.txt",
            "name": "digitalside",
            "rules_count": 376,
            "last_updated": "2021-04-04T12:23:18+01:00"
        },
        {
            "id": 1600953788,
            "enabled": true,
            "url": "https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt",
            "name": "lists.disconnect.me.malvertising",
            "rules_count": 2736,
            "last_updated": "2021-04-04T12:23:18+01:00"
        },
        {
            "id": 1600953789,
            "enabled": true,
            "url": "https://mirror1.malwaredomains.com/files/justdomains",
            "name": "malwaredomains",
            "rules_count": 26854,
            "last_updated": "2021-04-04T12:23:20+01:00"
        },
        {
            "id": 1600953790,
            "enabled": true,
            "url": "https://v.firebog.net/hosts/Prigent-Crypto.txt",
            "name": "Prigent-Crypto",
            "rules_count": 13580,
            "last_updated": "2021-04-04T12:23:20+01:00"
        },
        {
            "id": 1600953791,
            "enabled": true,
            "url": "https://mirror.cedia.org.ec/malwaredomains/immortal_domains.txt",
            "name": "malwaredomains/immortal_domains",
            "rules_count": 3196,
            "last_updated": "2021-04-04T12:23:21+01:00"
        },
        {
            "id": 1600953793,
            "enabled": true,
            "url": "https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt",
            "name": "Mandiant_APT1_Report_Appendix_D",
            "rules_count": 2046,
            "last_updated": "2021-04-04T12:23:22+01:00"
        },
        {
            "id": 1600953794,
            "enabled": true,
            "url": "https://phishing.army/download/phishing_army_blocklist_extended.txt",
            "name": "phishing.army",
            "rules_count": 24980,
            "last_updated": "2021-04-04T12:23:22+01:00"
        },
        {
            "id": 1600953795,
            "enabled": true,
            "url": "https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt",
            "name": "quidsup/notrack-malware",
            "rules_count": 356,
            "last_updated": "2021-04-04T12:23:22+01:00"
        },
        {
            "id": 1600953796,
            "enabled": true,
            "url": "https://v.firebog.net/hosts/Shalla-mal.txt",
            "name": "Shalla-mal",
            "rules_count": 19233,
            "last_updated": "2021-04-04T12:23:23+01:00"
        },
        {
            "id": 1600953797,
            "enabled": true,
            "url": "https://raw.githubusercontent.com/Spam404/lists/master/main-blacklist.txt",
            "name": "Spam404",
            "rules_count": 8096,
            "last_updated": "2021-04-04T12:23:23+01:00"
        },
        {
            "id": 1600953798,
            "enabled": true,
            "url": "https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts",
            "name": "add.Risk",
            "rules_count": 2556,
            "last_updated": "2021-04-04T12:23:23+01:00"
        },
        {
            "id": 1600953799,
            "enabled": true,
            "url": "https://urlhaus.abuse.ch/downloads/hostfile/",
            "name": "urlhaus.abuse.ch",
            "rules_count": 1188,
            "last_updated": "2021-04-04T12:23:23+01:00"
        },
        {
            "id": 1600953800,
            "enabled": true,
            "url": "https://zerodot1.gitlab.io/CoinBlockerLists/hosts_browser",
            "name": "CoinBlockerLists",
            "rules_count": 3553,
            "last_updated": "2021-04-04T12:23:24+01:00"
        }
    ],
    "whitelist_filters": null,
    "user_rules": [
        "||samba.tv^$important",
        "@@||securemetrics.apple.com^$important",
        "@@||g.live.com^$client='192.168.0.10'",
        "@@||gvt2.com^$client='192.168.0.10'",
        "@@||gvt2.com^$client='192.168.0.23'",
        "@@||erabu.sony.tv^$client='192.168.0.20'",
        ""
    ]
}
elRadix commented 3 years ago

here is my output

bakito commented 3 years ago

I've built a new version https://github.com/bakito/adguardhome-sync/releases/tag/v0.0.7 where the last_updated timestamp is ignored. It's anyway not of any use for the synchronization.

Your replica instance needs to be manually initialized via the adquard home wizard. Did you to this?

JoshStark commented 3 years ago

Your replica instance needs to be manually initialized via the adquard home wizard. Did you to this?

Yes, I went through the initial wizard to bind it and set it up, then restarted, then attempted to sync.

elRadix commented 3 years ago

how can I push the wizard again?

bakito commented 3 years ago

Does the error stil occur with the new Version?

To run the wizard again, you probably have to re-install adguard home.

elRadix commented 3 years ago

updated the app to the latest version its working great now

bakito commented 3 years ago

Cool Thx for the feedback

aayushus commented 3 years ago

Hi @bakito , I am using the latest build from the quay.io repo.

I am getting a similar error -

2021-04-18T01:12:00.083Z ERROR sync sync/sync.go:83 Error getting origin status {"from": "10.0.0.10:8082", "error": "403 Forbidden"} github.com/bakito/adguardhome-sync/pkg/sync.(*worker).sync /go/src/app/pkg/sync/sync.go:83

I am able to get a response from http://10.0.0.10:8082/control/filtering/status via postman.

Can you please help?

bakito commented 3 years ago

Hi @mercplus What's the Version of your origon adguard home? How does your config look ?

aayushus commented 3 years ago

Hi @mercplus What's the Version of your origon adguard home? How does your config look ?

Both of the adguards are docker deployment. I was syncing manually, so I tried switching Origin and Secondary with the same error.

Origin Version - v0.105.2 Secondary Version - v0.105.2

Here's my config - https://pastebin.com/zJ7BQ5KS

Please note - The adguards deployments are on 2 different (self-hosted) servers. The Adguardhome-sync deployment is on a third server which I primarily use for testing. I am able to get a response to the curl from this third server.

bakito commented 3 years ago

Are username and password in your config correct? 403 is returned from adguard home if username / password are incorrect.

You adguard home instances need to be initialized via the Weg UI befor synch is working.

aayushus commented 3 years ago

Are username and password in your config correct? 403 is returned from adguard home if username / password are incorrect.

You adguard home instances need to be initialized via the Weg UI befor synch is working.

Yeah, the username/password is correct. If they were incorrect, I would have gotten 401 unauthorized vs 403 Forbidden?

Is there a way to enable debug mode and see the actual curl request going out to Adguard?

bakito commented 3 years ago

There's currently no verbose mode.

Adguard Home does not return 401 but 403 on invalid credentials. At least, that's what I'm seeing with the version v0.105.2 running in docker.

`curl http://localhost:9090/control/status -v

What can you see in the log of adguard home container?

With the curl above I see 2021/04/18 15:37:58 [info] Auth: invalid Basic Authorization value

bakito commented 3 years ago

I added the possibility to see the requests sent to adguard home. You can see the messages when settintgthe environment variable LOG_LEVEL=debug

You need v0.1.2 for that.

aayushus commented 3 years ago

I added the possibility to see the requests sent to adguard home. You can see the messages when settintgthe environment variable LOG_LEVEL=debug

You need v0.1.2 for that.

Thanks. I get the below log -

2021-04-18T16:47:00.059Z DEBUG client client/client.go:105 do get {"host": "10.0.0.10:8082", "method": "GET", "path": "status", "username": "username"}

2021-04-18T16:47:00.132Z DEBUG client client/client.go:110 got response {"host": "10.0.0.10:8082", "method": "GET", "path": "status", "username": "username", "body": "Forbidden"}

2021-04-18T16:47:00.132Z ERROR sync sync/sync.go:86 Error getting origin status {"from": "10.0.0.10:8082", "error": "403 Forbidden"}

My password is a complete numeric string (no alphabets). Could that be an issue?

aayushus commented 3 years ago

Hi @bakito I was able to get it up and running.

I created a new instance on a new server with the same ID/Pass using the setup wizard, didn't work.

I then changed the username to an email address with an alphanumeric password, it worked. I reinitialized my regular adguards and everything started working!

bakito commented 3 years ago

OK, then it must have been something with your first setup.

It worked for me, when I used the setup with an all numeric password.

I was not able to reproduce this issue.

Ionutsbirnea commented 6 months ago

Hi all. I get the same 403 error, username is an email and password is indeed alphanumeric but with one capital letter. So far no luck logging in. I am running this in a jail on a TrueNas server. The install has been done via plug-in tab. Any help will be much appreciated. Thanks all!

GeneralUser-NonDev commented 6 months ago

Hi all I have the following error trying to setup the sync program (unraid docker)

[migrations] started
[migrations] no migrations found
───────────────────────────────────────

      ██╗     ███████╗██╗ ██████╗ 
      ██║     ██╔════╝██║██╔═══██╗
      ██║     ███████╗██║██║   ██║
      ██║     ╚════██║██║██║   ██║
      ███████╗███████║██║╚██████╔╝
      ╚══════╝╚══════╝╚═╝ ╚═════╝ 

   Brought to you by linuxserver.io
───────────────────────────────────────

To support the app dev(s) visit:
AdGuardHome Sync: https://github.com/sponsors/bakito

To support LSIO projects visit:
https://www.linuxserver.io/donate/

───────────────────────────────────────
GID/UID
───────────────────────────────────────

User UID:    99
User GID:    100
───────────────────────────────────────

[custom-init] No custom files found, skipping...
[ls.io-init] done.
2024-01-06T18:43:08.956-0500    INFO    root    cmd/root.go:125 Using config file:/config/adguardhome-sync.yaml
2024-01-06T18:43:08.957-0500    INFO    sync    sync/sync.go:38 AdGuardHome sync        {"version": "0.5.2", "build": "2023-11-23T07:12:37Z", "os": "linux", "arch": "amd64"}
2024-01-06T18:43:08.958-0500    INFO    sync    sync/sync.go:65 Setup cronjob   {"cron": "*/1 * * * *", "next-execution": "2024-01-06T18:44:00.000-0500"}
2024-01-06T18:43:08.958-0500    INFO    sync    sync/http.go:57 Starting API server     {"port": 8080}
2024-01-06T18:43:08.958-0500    INFO    sync    sync/sync.go:75 Running sync on startup
2024-01-06T18:43:08.959-0500    ERROR   sync    sync/sync.go:153        Error getting origin status     {"from": "192.168.10.248:3000", "error": ": Get \"https://192.168.10.248:3000/control/status\": http: server gave HTTP response to HTTPS client"}
github.com/bakito/adguardhome-sync/pkg/sync.(*worker).sync
        /home/bakito/go/src/github.com/bakito/adguardhome-sync/pkg/sync/sync.go:153
github.com/bakito/adguardhome-sync/pkg/sync.Sync.func3
        /home/bakito/go/src/github.com/bakito/adguardhome-sync/pkg/sync/sync.go:76

Primary Adguard (v0.107.43) is on my unraid server, same as the sync program. 192.168.10.0 secondary is on another machine (v0.107.43, proxmox container, helper script install) on another VLAN. 192.168.1.0 sync seems to be having issue seeing the primary which is on the same machine as itself. Both instances seem to be working (set same filters manually when sync didnt work) and are showing use. Ive even tried putting both adguard and sync on "custom: br0" in unraid as i'd seen a comment on another issue post about different custom docker networks messing with the program.

I can only install things like this via youtube videos so i have no clue what i'm looking at, any assistance is appreciated.

edit: I just found the webui of sync, and logged in. when i clicked on the red "origin" button i got an error ( don't recall it, something about a file/command/something being too long), but when i hovered it said something about returning a non-secure connection on a secure request, no i removed the "s" on the http addresses of both origin and replica, noow when i click the origin button it loads the page, but i still have errors, (now 403 forbidden) but it loads the origin page, i don't see any buttons for the replica(s), i'd seen the extras so I commented them out (multiple setup) and left just the single setup.

Edit again: I figured it out, finally got the sync webui to have an icon for both primary and secondary instances, then i quintuple checked the logins and what do you know, i messed up the password on the primary, fixed it and it appears to be working correctly now from what i can tell.

bartclone commented 6 months ago

Hi,

The error log shows that the AG running in the Proxmox container cannot be reached.

As you state this Proxmoc container is on another VLAN, which separates networks, you only can solve it using a specific route on your proxmox-host and/or firewall/router.

As such, this is beyond AG, and more of a network issue. Have you ticked the box in Proxmox (host) to be VLAN-aware?

Regards, Bart

Op zo 7 jan. 2024 om 00:32 schreef GeneralUser-NonDev < @.***>

Hi all I have the following error trying to setup the sync program (unraid docker)

2024-01-06T18:22:03.975-0500` ERROR sync sync/sync.go:153 Error getting origin status {"from": "192.168.10.248:3000", "error": ": Get \"https://192.168.10.248:3000/control/status\": dial tcp 192.168.10.248:3000: connect: no route to host"}github.com/bakito/adguardhome-sync/pkg/sync.(*worker).sync /home/bakito/go/src/github.com/bakito/adguardhome-sync/pkg/sync/sync.go:153github.com/bakito/adguardhome-sync/pkg/sync.Sync.func2 /home/bakito/go/src/github.com/bakito/adguardhome-sync/pkg/sync/sync.go:59github.com/robfig/cron/v3.FuncJob.Run @./cron.go:136github.com/robfig/cron/v3.(Cron).startJob.func1 **@.***/cron.go:312

Primary Adguard is on my unraid server, same as the sync program. 192.168.10.0 secondary is on another machine (proxmox container, helper script install) on another VLAN. 192.168.1.0 sync seems to be having issue seeing the primary which is on the same machine as itself. Both instances seem to be working (set same filters manually when sync didnt work) and are showing use.

I can only install things like this via youtube videos so i have no clue what i'm looking at, any assistance is appreciated.

— Reply to this email directly, view it on GitHub https://github.com/bakito/adguardhome-sync/issues/3#issuecomment-1879875781, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4W3VHZHN2CSX4VIAR7ZS3YNHNHZAVCNFSM42MT6X6KU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBXHE4DONJXHAYQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

gh57gh commented 3 months ago

I'm having a similar problem. My situation is the following: Replica: DietPi with Adguard installed over the dietpi-software (not docker) Origin: Unraid with Adguard installed over the Docker Compose Manager. Adguard-Sync: Part of the docker-compose of the Unraid Adguard instance.

I get the two errors in the Adguard-Sync webui:

"ERROR  sync    sync/sync.go:136    Error getting origin status"
"ERROR  sync    sync/sync.go:165    Error getting origin status"

my filtering status for the unraid adguard origin is the following:

{"filters":[{"url":"https://adguardteam.github.io/HostlistsRegistry/assets/filter_1.txt","name":"AdGuard DNS filter","last_updated":"2024-04-16T13:09:18+02:00","id":1,"rules_count":66483,"enabled":true},{"url":"https://adguardteam.github.io/HostlistsRegistry/assets/filter_2.txt","name":"AdAway Default Blocklist","id":2,"rules_count":0,"enabled":false}],"whitelist_filters":null,"user_rules":[],"interval":24,"enabled":true}

I can click on both instances of adguard through the adguard-sync webgui. Unfortunately I'm no expert and I think there is an underlying problem with the Unraid Adguard origin.

This is the yaml I created in Unraid:

version: "2"
services:
  adguardhome:
    image: adguard/adguardhome
    container_name: adguardhome
    networks:
      br0:
        ipv4_address: 10.19.19.18
    ports:
    # Plain DNS
      - 53:53/tcp
      - 53:53/udp
    # AdGuard Home Admin Panel as well as DNS-over-HTTPS
      - 8083:80/tcp # Home WebGUI access
      - 488:443/tcp
      - 488:443/udp
      - 3000:3000/tcp # First setup to set user and password
    # DNS-over-TLS
      - 853:853/tcp
    # DNS-over-QUIC
      - 784:784/udp
      - 853:853/udp
      - 8853:8853/udp
    # DNSCrypt
      - 5443:5443/tcp
      - 5443:5443/udp
    environment:
      - TZ=Europe/Berlin
    volumes:
      - /mnt/user/appdata/adguardhome/work:/opt/adguardhome/work
      - /mnt/user/appdata/adguardhome/conf:/opt/adguardhome/conf

  adguardhome-sync:
    image: lscr.io/linuxserver/adguardhome-sync:latest
    container_name: adguardhome-sync
    environment:
      - PUID=99
      - PGID=100
      - TZ=Europe/Berlin
      - CONFIGFILE=/config/adguardhome-sync.yaml
    volumes:
      - /mnt/user/appdata/adguardhome-sync/config:/config
    ports:
      - 8038:8080
    networks:
      - customproxy
    restart: unless-stopped

networks:
  br0:
    name: br0
    external: true
  customproxy:
    driver: bridge
    external: true