Unexpected Filter Behaviour

[NFGman]

I've been using update_all for more than six months. For most of that time I've had a filter in place, so that unwanted cores and .mra files aren't downloaded. A few days ago I ran update_all and more than 400 .mra files were deleted.

Editing the filter, even cutting it down to simply filter=!cheats didn't resolve the issue. No doubt due to my ignorance of the MiSTer system, the only way I found to forcibly re-download all the missing cores and MRAs was to delete the whole Scripts/.config directory.

So, it seems that:

1. my filter - which worked for ~six months - was suddenly being parsed differently, and
2. the filtered-out files will not be re-downloaded until the system forgets which ones it was told to skip.

The filter I was using successfully for many months was this:

[distribution_mister] db_url = https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip filter = !cheats !computer-cores !other !arcade-astrocade !arcade-bankpanic !arcade-battlezone !arcade-blockade !arcade-canyonbomber !arcade-cosmic !arcade-cosmicguerilla !arcade-crazyballoon !arcade-dominos !arcade-dottorikun !arcade-mrjong !arcade-mysticmarathon !arcade-pleiads !arcade-polyplay !arcade-pooyan !arcade-popeye !arcade-riverpatrol !arcade-silverland !arcade-sms !arcade-snk_triplez80 !arcade-spaceinvaders !arcade-spacerace !arcade-sprint1 !arcade-sprint2 !arcade-subs !arcade-superbreakout !arcade-tiamc1 !arcade-tnkiii !arcade-traverseusa !arcade-tropicalangel !arcade-turkeyshoot !arcade-ultratank !arcade-zigzag !S32X !AY-3-8500 !AVision !Arcadia !Astrocade !Atari5200 !Atari7800 !AtariLynx !BBCBridgeCompanion !Casio_PV-1000 !ChannelF !ColecoVision !CreatiVision !Gamate !GBA !GBA2P !GameBoy !Gameboy2P !GameNWatch !Intv !MyVision !NES !Odyssey2 !Pokemon Mini !SGB !SMS !SNES !Super_Vision_8000 !SuperVision !VC4000 !Vectrex !WonderSwan

This worked wonderfully, downloading only the cores I wanted and ignoring the ones I didn't. Until, suddenly, many many cores (including cave (dodonpachi, dangun feveron etc) and robotron (Robotron, Defender etc)) were caught up in the filter.

UPDATE: the problem is !Pokemon Mini. With that in there, the filter goes into aggressive mode. Shadowmasks, fonts, filters, arcade MRAs, alternatives, docs, computers. Many hundreds of deletions.

[theypsilon]

Hey, thanks for opening the issue here on GitHub. I just saw some notifications in Discord, but it is much easier to put everything in context and follow up on solutions here.

Indeed, your problem is considering "Pokemon Mini" as just one filter term when it's indeed 2. With that in mind, filtering follows the following logic:

• First, positive filters are processed to create a selection of files to be downloaded
• Then, negative filters are applied to that selection to filter it further down.

So, in your case, your filter says, "I want all files matching 'Mini' files, and from that selection, remove all the files matching cheats, computer cores, etc...". So it seems to be working correctly.

Why didn't it work like that before for your particular case? I fixed a bug in the latest release when the filter was applied only to a single database (like in your case), and now that is fixed, you are experiencing normal behavior.

Once all of this is clarified, the only question left would be if you or any of the participants in Discord have any suggestions to avoid such a situation. I could probably use better documentation to make this logic clearer, so I'd be happy to merge a doc PR to improve it. In the filtering logic itself, I fail to see clear improvements, but I'm open to hear what you guys think. Maybe I could add support for a "positive_filter" and "negative_filter" property to the INI so that users could opt into a less ambiguous declaration.

[NFGman]

Thanks for explaining that. It was initially pretty bizarre but yeah, I think I had it figured out by the end. It's nice to know this is 'fixed' and not 'newly buggy'.

The way it works is pretty easy to understand, but yeah, it's not easy until you work it out and then it's just obvious.

I'd be pretty happy to try my hand at writing some filtering docs. I'll see what I can do.

In the mean time, perhaps there's a way to add the tags responsible for actions in the logs? It seems conceptually easy but thinking on it I don't know how it'd actually work from a code perspective. But a user-facing message like "Removing files because tag 'mini'" might go a long way to debugging the process.