Tag blacklist option seems to behave inconsistently

[prof-m]

When I specify a tag or series of tags in the tags.blacklist config options, Hybooru is inconsistent about actually removing the tags and the posts with those tags from the UI. Sometimes, after using Rebuild Database, it removes the tags and all posts with those tags as expected. But sometimes I update the blacklist option in `configs.json), use Rebuild Database again, and suddenly none of the tags in the blacklist seem to be respected - the tags and their posts are back.

I messed around for a few hours trying to see if I could find a consistent trigger that reproduces it, but nothing seems to consistently make the blacklist tags work as expected. Some scenarios I tested:

• Update configs.json while Hybooru is already running and then hit 'Rebuild Database'. (This pretty much always fails to notice the updates in configs.json, I noticed - I'm guessing it doesn't get read again once the app is running in prod mode?)
• Stop Hybooru, update configs.json, run npm run start:prod, hit 'Rebuild Database'
• Stop Hybooru, update configs.json, run npm run build:prod, run npm run start:prod, hit 'Rebuild Database'
• Stop Hybooru, update configs.json, run rm -rf dist/ && npm run build:prod, run npm run start:prod, hit 'Rebuild Database'
• All of the above with both one blacklisted tag and multiple blacklisted tags
• All of the above with tags.resolveRelations set to true, and also set to false.

Apologies for not being able to provide concrete reproduction steps - it really does seem to be inconsistent. @_@ The best I can say is that it seems to stop working more often when the blacklisted tag is a parent tag on a post, rather than a direct tag on a post.

[funmaker]

Yes, you need to restart Hybooru to apply config changes. Based on commands, I assume you are running production Hybooru from development environment. Which configs.json do you update? When you run npm run build:prod, build script prepares another configs.json in dist/ folder for deployment. It's content is a flat merge of configs.json in project root and previous dist/configs.json if it exists.(see webpack/package.babel.js) Make sure you are updating dist/configs.json if you are running production and root configs.json for development(npm start). If you are running production, it's best to only deploy content of dist folder, there is no need to use npm run start:prod in general, unless you are debugging the building process itself.

[prof-m]

Aha, all good to know, thank you! I hadn't noticed the second configs.json file in dist/, but knowing it exists and about the flat merge makes a lot of the behavior I was seeing make sense.

Even confirming that the dist/configs.json is set correctly, I am still seeing the blacklist not be respected when running npm run start:prod, but I'll mess around with serving the contents of dist/ from a webserver instead of using start:prod and see if that changes things.

[prof-m]

Okay, good news - the info you gave me helped lead me to successfully serving Hybooru over https via Caddy for the first time. 🎉

Bad news - the blacklist from the dist/configs.json file still isn't being respected. Both the blacklisted tag and the posts with that tag are still showing up post Hybooru-restart and Rebuild Database.

In case I'm still doing something wrong, here's the steps I'm now following:

• Only put base config info in configs.json (see: base-configs.json below)
• Run npm run build:prod
• cd into dist/
• Update tags.blacklist entry for dist/configs.json (seedist-configs.json below)
• Run npm start (still in dist/)
• Open browser to site, do a hard refresh just to be safe, hit Settings icon, hit Rebuild Database, do another hard refresh once that finishes
• Check for blacklisted tag in Tags page

base-configs.json file (sanitized):

{
  "port": 3939,
  "hydrusDbPath": "/path/to/hydrus/db",
  "appName": "Booru",
  "appDescription": "An imageboard",
  "adminPassword": "Password",
  "isTTY": true,
  "importBatchSize": 8192,
  "pageSize": 72,
  "cachePages": 5,
  "cacheRecords": 1024,
  "filesPathOverride": null,
  "thumbnailsPathOverride": null,
  "maxPreviewSize": 104857600,
  "db": {
    "user": "hybooru",
    "host": "localhost",
    "database": "hybooru",
    "password": "password",
    "port": 5433
  },
  "tags": {
    "motd": "miku",
    "untagged": "-*",
    "ignore": [],
    "blacklist": [],
    "whitelist": null,
    "resolveRelations": true
  },
  "rating": {
    "enabled": false,
    "stars": 5,
    "serviceName": null
  }
}

dist-configs.json file (sanitized):

{
    "port": 3939,
    "hydrusDbPath": "/path/to/hydrus/db",
    "appName": "Booru",
    "appDescription": "An imageboard",
    "adminPassword": "Password",
    "isTTY": true,
    "importBatchSize": 8192,
    "pageSize": 72,
    "cachePages": 5,
    "cacheRecords": 1024,
    "filesPathOverride": null,
    "thumbnailsPathOverride": null,
    "maxPreviewSize": 104857600,
    "db": {
        "user": "hybooru",
        "host": "localhost",
        "database": "hybooru",
        "password": "password",
        "port": 5433
    },
    "tags": {
        "motd": "miku",
        "untagged": "-*",
        "ignore": [
            "exu"
        ],
        "blacklist": [
            "oldp"
        ],
        "whitelist": null,
        "resolveRelations": true
    },
    "rating": {
        "enabled": false,
        "stars": 5,
        "serviceName": null
    }
}

Relevant Caddyfile for reference:

https://profm.url {
    root * /home/ubuntu/Projects/Hybooru/dist
    file_server
    reverse_proxy http://localhost:3939

    tls email@email.com
    encode zstd gzip
}

[funmaker]

@prof-m can you prepare a small hydrus database and list of steps to reproduce this issue? You don't have to include the files, just hydrus database files that will let me find what exactly is wrong.

[prof-m]

@funmaker yup, let me see what I can pull together

[prof-m]

@funmaker How small do you want the hydrus database to be? The current one I'm running Hybooru on has a little over 10k files, with a client.db filesize of 31.5mb or so - would that work, or should I pair it down?

[funmaker]

@prof-m any would be fine. Preferably with only the tags, siblings/parents and posts that represent the issue you are describing. If you can't prepare a database with just that, you can send me your current database.

[funmaker]

Closing this for now. @prof-m once you get reproduction steps ready you can reopen the issue.

[prof-m]

Sounds good! I actually never could reproduce it, so I suspect it probably was just me getting confused about what config files go where. If I ever encounter it again, I'll dig into getting a reproducible copy