openzim / zimfarm

Farm operated by bots to grow and harvest new zim files
https://farm.openzim.org
GNU General Public License v3.0
81 stars 25 forks source link

Review all input validations #783

Open rgaudin opened 1 year ago

rgaudin commented 1 year ago

In order to prevent incorrect data to be entered in the Zimfarm, we want to review all recipes inputs and update their constraints.

All constraints are in https://github.com/openzim/zimfarm/blob/main/dispatcher/backend/src/common/schemas/fields.py and https://github.com/openzim/zimfarm/tree/main/dispatcher/backend/src/common/schemas/offliners

Beside Offliner Flags, recipe inputs are:

Please list below the list of changes you'd like to constraints on those fields or Offliner flags. If a change is to be applied to all offliners (albeit using their own scaper-specific names), please say so.

My proposal:

rgaudin commented 1 year ago
kelson42 commented 8 months ago

devdocs.ios failing scenario because of this weakness https://farm.openzim.org/pipeline/cb6a074a-e33a-4552-9c45-932486a1dde9/debug

rgaudin commented 8 months ago

I believe it's a bit different: allowing arbitrary text in choice field or (more likely) a previously value choice that is not anymore and we did not update the recipe

benoit74 commented 8 months ago

I think that #910 proves that we also need to regularly check that flags are still valid.

We should check:

I've opened a distinct issue #911 since this is a bit different than just ensuring that constraints of all offliners are appropriate.

benoit74 commented 4 months ago

https://github.com/kiwix/k8s/issues/192 just shown how important it is to avoid unwanted chars in ZIM name / filename.

benoit74 commented 2 weeks ago

Removed from zimit2 project unfortunately