search

openfoodfacts / openfoodfacts-server

Open Food Facts database, API server and web interface - 🐪🦋 Perl, CSS and JS coders welcome 😊 For helping in Python, see Robotoff or taxonomy-editor
http://openfoodfacts.github.io/openfoodfacts-server/
GNU Affero General Public License v3.0
652 stars 381 forks source link

Return numbers as numbers (and not strings) in the API, store them as numbers in MongoDB, add tests #4389

Open stephanegigandet opened 4 years ago

stephanegigandet commented 4 years ago

We have this long running issue (see #639) of numbers being sometimes stored as strings in MongoDB or in the JSON returned by the API. We have been fixing many examples of it, but they keep coming back.

It has to do with the way Perl handles scalars: they are automatically converted to strings or to numbers as needed, based on what is being done with them.

So for instance if we use a number as a string (e.g. to display it, to output it to a file, to log it etc.) it will internally become a string.

Unfortunately, the whole world does not use Perl, so it is causing issues for apps that parse the JSON and expect a defined type for each field.

So we need to:

  1. Define exactly which fields are strings and which fields are numbers
  2. Look for, investigate and fix any inconsistent types
  3. Fix all inconsistent types in the database
  4. Whenever possible, add tests to make sure we don't break things (in particular for the API where we can do some processing before returning values, and where numbers can thus become strings)
aleene commented 4 years ago

Is it possible to explicitly set the types for each and very field, instead of relying on Perl behaviour.

hangy commented 3 years ago

Is it possible to explicitly set the types for each and very field, instead of relying on Perl behaviour.

Looks like it's possible with explicit typecasting. There's also document schemas in MongoDB, but I don't know how well this would work with all the dynamic language based fields in ProductOpener.

github-actions[bot] commented 8 months ago

This issue has been open 90 days with no activity. Can you give it a little love by linking it to a parent issue, adding relevant labels and projets, creating a mockup if applicable, adding code pointers from https://github.com/openfoodfacts/openfoodfacts-server/blob/main/.github/labeler.yml, giving it a priority, editing the original issue to have a more comprehensive description… Thank you very much for your contribution to 🍊 Open Food Facts