Enhance tags manipulation

openzim / python-scraperlib

Collection of Python code to re-use across Python-based scrapers

GNU General Public License v3.0

19 stars 16 forks source link

Enhance tags manipulation #175

Closed benoit74 closed 3 months ago

benoit74 commented 3 months ago

Fix #164 Fix #156

Changes:

New utility function to compute tags from scraper defaults and user provided values
Also raise error in tags validation when duplicates are found, either in string with semi-colon separator or in

This does not deduplicate automatically tags, it is still the scraper responsibility to properly pass tags (ideally with the new utility function) since config_metadata is more a wrapper passing directly the values as-is, I did not wanted to make an exception for tags

Edit: I had to also modify URL used for yt-dlp tests because former Vimeo URL is now returning a 403 on Github runners. Hopefully the new Jeena Peertube instance will be a little bit more permissive for few years ...

benoit74 commented 3 months ago

Oups, CI is failing, I've requested review too early, sorry

benoit74 commented 3 months ago

Arf, this has nothing to do with my change, it is Vimeo returning 403 on yt_dlp tests ...

codecov[bot] commented 3 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (07cb331) to head (4402698).

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #175 +/- ## ========================================= Coverage 100.00% 100.00% ========================================= Files 32 32 Lines 1439 1448 +9 Branches 248 251 +3 ========================================= + Hits 1439 1448 +9 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

benoit74 commented 3 months ago

Moved to Jeena Peertube instance instead of Vimeo ... this will work the time it will ...

benoit74 commented 3 months ago

Yep, maybe we could support both types for both variables, but there is no use-case as far as I can tell and we've agreed to not implement something "should it become a need somewhere in the future".