openfoodfacts / open-prices

An open database of prices 🧾💸💰🏷️🤑🍽️
https://prices.openfoodfacts.org
GNU Affero General Public License v3.0
42 stars 12 forks source link

Product sync: use default field values instead of None #518

Closed raphodn closed 1 month ago

raphodn commented 1 month ago

What

During the OxF daily sync, we create (or update) each product. The input data is not always consistent, for instance some fields are sometimes missing. In that case, we currently set the field value to None.

We should rather skip that field, and let the model decide its default value.

Why

raphodn commented 1 month ago

Script to fix the Product array fields default values (list instead of None)

for f in Product.ARRAY_FIELDS:
    print(f)
    print(Product.objects.filter(**{f: None}).count())
    Product.objects.filter(**{f: None}).update(**{f: list()})

Script to delete duplicate prices (and proofs)

https://github.com/openfoodfacts/open-prices/wiki/Useful-scripts#delete-duplicate-prices