Is the search query as minimized and as fast-looking as we can ?

teolemon commented 2 years ago

What

We're loading a shitload of products for personal ranking (and for some queries, that's not even the whole query)
Is the search query as minimized and as fast-looking as we can ?
If it's already optimized, can we manage user expectations
- progress text
- progressive sorting as we receive the response
- show unranked search results (for 25 first results of the query, with just fields=product_name,small_img,nutriscore,ecoscore, and have a spinning grey "My personal ranking" that ungreys as the client receives the whole personal results to rank

Video

https://user-images.githubusercontent.com/1689815/147953519-794f4ef9-9450-4f3a-9293-4b79ae1be4b8.mp4

Part of

533

monsieurtanuki commented 2 years ago

Part of the problem is that we retrieve 10Kb by product, because we also get all the verbose comments:

just one salmon (welcome to the matrix)

``` {code: 3019081236267, product_name: Rillettes de saumon, brands: Petit Navire, lang: en, quantity: 1 25 g, image_small_url: https://images.openfoodfacts.org/images/products/301/908/123/6267/front_fr.11 0.200.jpg, image_front_url: https://images.openfoodfacts.org/images/products/301/908/123/6267/front_ fr.110.400.jpg, image_front_small_url: https://images.openfoodfacts.org/images/products/301/908/123/ 6267/front_fr.110.200.jpg, image_ingredients_url: https://images.openfoodfacts.org/images/products/3 01/908/123/6267/ingredients_fr.40.400.jpg, image_nutrition_url: https://images.openfoodfacts.org/ima ges/products/301/908/123/6267/nutrition_fr.113.400.jpg, product_quantity: 125.0, selected_images: {f ront: {thumb: {fr: https://images.openfoodfacts.org/images/products/301/908/123/6267/front_fr.110.10 0.jpg}, small: {fr: https://images.openfoodfacts.org/images/products/301/908/123/6267/front_fr.110.2 00.jpg}, display: {fr: https://images.openfoodfacts.org/images/products/301/908/123/6267/front_fr.11 0.400.jpg}, original: {}, unknown: {}}, ingredients: {thumb: {fr: https://images.openfoodfacts.org/i mages/products/301/908/123/6267/ingredients_fr.40.100.jpg}, small: {fr: https://images.openfoodfacts .org/images/products/301/908/123/6267/ingredients_fr.40.200.jpg}, display: {fr: https://images.openf oodfacts.org/images/products/301/908/123/6267/ingredients_fr.40.400.jpg}, original: {}, unknown: {}} , nutrition: {thumb: {fr: https://images.openfoodfacts.org/images/products/301/908/123/6267/nutritio n_fr.113.100.jpg}, small: {fr: https://images.openfoodfacts.org/images/products/301/908/123/6267/nut rition_fr.113.200.jpg}, display: {fr: https://images.openfoodfacts.org/images/products/301/908/123/6 267/nutrition_fr.113.400.jpg}, original: {}, unknown: {}}, packaging: {thumb: {}, small: {}, display : {}, original: {}, unknown: {}}, other: {thumb: {}, small: {}, display: {}, original: {}, unknown: {}}}, images: {}, imagesFreshnessInLanguages: null, ingredients_analysis_tags: [en:non-vegan, en:non -vegetarian, en:palm-oil-content-unknown], nutriments: {salt_100g: 1.1, fiber_100g: 2.6, sugars_100g : 0.0, fat_100g: 22.0, saturated-fat_100g: 2.5, proteins_100g: 14.0, nova-group_100g: 4, energy_100g : 1092.0, energy-kcal: 261.0, energy-kcal_100g: 261.0, carbohydrates_100g: 0.0, fat_unit: g, protein s_unit: g, nova-group_serving: 4, carbohydrates_unit: g, energy_unit: KCAL, energy-kcal_unit: KCAL, sodium_100g: 0.44, sodium_unit: G, omega-3-fat_100g: 1.9, omega-3-fat_unit: G}, additives_tags: [en: e160c], environment_impact_level_tags: [en:low], allergens_tags: [], nutrient_levels: {sugars: low, fat: high, saturated-fat: moderate, salt: moderate}, nutrition_grade_fr: b, labels_tags: [en:produce d-in-brittany], states_tags: [en:to-be-completed, en:nutrition-facts-completed, en:ingredients-compl eted, en:expiration-date-completed, en:packaging-code-completed, en:characteristics-to-be-completed, en:origins-to-be-completed, en:categories-completed, en:brands-completed, en:packaging-completed, e n:quantity-completed, en:product-name-completed, en:photos-to-be-validated, en:packaging-photo-to-be -selected, en:nutrition-photo-selected, en:ingredients-photo-selected, en:front-photo-selected, en:p hotos-uploaded], attribute_groups: [{id: nutritional_quality, name: Qualité nutritionnelle, attribut es: [{id: nutriscore, name: Nutri-Score, title: Nutri-Score B, icon_url: https://static.openfoodfact s.org/images/attributes/nutriscore-b.svg, description: , description_short: Bonne qualité nutritionn elle, match: 61.0, status: known}, {id: low_salt, name: Sel, title: Sel en quantité modérée, icon_ur l: https://static.openfoodfacts.org/images/attributes/nutrient-level-salt-medium.svg, description_sh ort: 1.1 g / 100 g, match: 40.0, status: known}, {id: low_fat, name: Matières grasses, title: Matièr es grasses en quantité élevée, icon_url: https://static.openfoodfacts.org/images/attributes/nutrient -level-fat-high.svg, description_short: 22 g / 100 g, match: 18.0, status: known}, {id: low_sugars, name: Sucres, title: Sucres en faible quantité, icon_url: https://static.openfoodfacts.org/images/at tributes/nutrient-level-sugars-low.svg, description_short: 0 g / 100 g, match: 100.0, status: known} , {id: low_saturated_fat, name: Acides gras saturés, title: Acides gras saturés en quantité modérée, icon_url: https://static.openfoodfacts.org/images/attributes/nutrient-level-saturated-fat-medium.sv g, description_short: 2.5 g / 100 g, match: 62.8571428571429, status: known}]}, {id: allergens, name : Allergènes, warning: Il est toujours possible que les données sur les allergènes soient manquantes , incomplètes, incorrectes ou que la composition du produit ait changé. Si vous êtes allergique, vér ifiez toujours les informations sur l'emballage réel du produit., attributes: [{id: allergens_no_glu ten, name: Gluten, title: Présence inconnue : Gluten, icon_url: https://static.openfoodfacts.org/ima ges/attributes/gluten-content-unknown.svg, status: unknown}, {id: allergens_no_milk, name: Lait, tit le: Contient : Lait, icon_url: https://static.openfoodfacts.org/images/attributes/contains-milk.svg, match: 0.0, status: known}, {id: allergens_no_eggs, name: Œufs, title: Contient : Œufs, icon_url: h ttps://static.openfoodfacts.org/images/attributes/contains-eggs.svg, match: 0.0, status: known}, {id : allergens_no_nuts, name: Fruits à coque, title: Présence inconnue : Fruits à coque, icon_url: http s://static.openfoodfacts.org/images/attributes/nuts-content-unknown.svg, status: unknown}, {id: alle rgens_no_peanuts, name: Arachides, title: Présence inconnue : Arachides, icon_url: https://static.op enfoodfacts.org/images/attributes/peanuts-content-unknown.svg, status: unknown}, {id: allergens_no_s esame_seeds, name: Graines de sésame, title: Présence inconnue : Graines de sésame, icon_url: https: //static.openfoodfacts.org/images/attributes/sesame-seeds-content-unknown.svg, status: unknown}, {id : allergens_no_soybeans, name: Soja, title: Présence inconnue : Soja, icon_url: https://static.openf oodfacts.org/images/attributes/soybeans-content-unknown.svg, status: unknown}, {id: allergens_no_cel ery, name: Céleri, title: Présence inconnue : Céleri, icon_url: https://static.openfoodfacts.org/ima ges/attributes/celery-content-unknown.svg, status: unknown}, {id: allergens_no_mustard, name: Moutar de, title: Présence inconnue : Moutarde, icon_url: https://static.openfoodfacts.org/images/attribute s/mustard-content-unknown.svg, status: unknown}, {id: allergens_no_lupin, name: Lupin, title: Présen ce inconnue : Lupin, icon_url: https://static.openfoodfacts.org/images/attributes/lupin-content-unkn own.svg, status: unknown}, {id: allergens_no_fish, name: Poisson, title: Contient : Poisson, icon_ur l: https://static.openfoodfacts.org/images/attributes/contains-fish.svg, match: 0.0, status: known}, {id: allergens_no_crustaceans, name: Crustacés, title: Présence inconnue : Crustacés, icon_url: htt ps://static.openfoodfacts.org/images/attributes/crustaceans-content-unknown.svg, status: unknown}, { id: allergens_no_molluscs, name: Mollusques, title: Présence inconnue : Mollusques, icon_url: https: //static.openfoodfacts.org/images/attributes/molluscs-content-unknown.svg, status: unknown}, {id: al lergens_no_sulphur_dioxide_and_sulphites, name: Anhydride sulfureux et sulfites, title: Présence inc onnue : Anhydride sulfureux et sulfites, icon_url: https://static.openfoodfacts.org/images/attribute s/sulphur-dioxide-and-sulphites-content-unknown.svg, status: unknown}]}, {id: ingredients_analysis, name: Ingrédients, attributes: [{id: vegan, name: Végétalien, title: Non végétalien, icon_url: https ://static.openfoodfacts.org/images/attributes/non-vegan.svg, match: 0.0, status: known}, {id: vegeta rian, name: Végétarien, title: Non végétarien, icon_url: https://static.openfoodfacts.org/images/att ributes/non-vegetarian.svg, match: 0.0, status: known}, {id: palm_oil_free, name: Sans huile de palm e, title: Présence d'huile de palme inconnue, icon_url: https://static.openfoodfacts.org/images/attr ibutes/palm-oil-content-unknown.svg, status: unknown}]}, {id: processing, name: Transformation des a liments, attributes: [{id: nova, name: Groupe NOVA, title: NOVA 4, icon_url: https://static.openfood facts.org/images/attributes/nova-group-4.svg, description: , description_short: Aliments ultra-trans formés, match: 0.0, status: known}, {id: additives, name: Additifs, title: 1 additif, icon_url: http s://static.openfoodfacts.org/images/attributes/1-additives.svg, match: 80.0, status: known}]}, {id: environment, name: Environnement, attributes: [{id: ecoscore, name: Éco-Score, title: Éco-Score C, i con_url: https://static.openfoodfacts.org/images/attributes/ecoscore-c.svg, description: , descripti on_short: Impact modéré sur l'environnement, match: 42.0, status: known}, {id: forest_footprint, nam e: Empreinte forêt, title: Empreinte forêt non calculée, icon_url: https://static.openfoodfacts.org/ images/attributes/forest-footprint-not-computed.svg, description: , description_short: Pour l'instan t seulement pour les produits avec du poulet ou des oeufs, match: 0.0, status: known}]}, {id: labels , name: Labels, attributes: [{id: labels_organic, name: Agriculture biologique, title: Pas un produi t bio, icon_url: https://static.openfoodfacts.org/images/attributes/not-organic.svg, description: L' agriculture biologique vise à protéger l'environnement et à conserver la biodiversité en prohibant o u limitant l'utilisation d'engrais synthétiques, de pesticides et d'additifs alimentaires., descript ion_short: Les produits bios encouragent la durabilité écologique et la biodiversité., match: 0.0, s tatus: known}, {id: labels_fair_trade, name: Commerce équitable, title: Ne provient pas du commerce équitable, icon_url: https://static.openfoodfacts.org/images/attributes/not-fair-trade.svg, descript ion: Quand vous achetez des produits du commerce équitable, les producteurs dans les pays en dévelop pement sont payés un prix plus haut et plus équitable, ce qui les aide à atteindre des plus hauts st andards sociaux et environnementaux et à les conserver., description_short: Les produits du commerce équitable aident les producteurs des pays en voie de développement., match: 0.0, status: known}]}], ecoscore_grade: c, ecoscore_score: 43.0, ecoscore_data: {grade: c, score: 43.0, status: known, agri balyse: {score: 51.0}, adjustments: {packaging: {value: -3.0, score: 68.1}, origins_of_ingredients: {epi_score: 0.0, epi_value: -5.0, transportation_score: 0.0, transportation_value: 0.0}}, missing_da ta_warning: true}, categories_tags_fr: [Viandes, Produits de la mer, Produits à tartiner, Poissons, Plats préparés, Charcuteries, Produits à tartiner salés, Rillettes, Thons, Préparations de poisson, Plats préparés au poisson, Rillettes de poissons, Rillettes de saumon, Rillettes de thon, en:prepara tions-made-from-fish-meat], labels_tags_fr: [Produit en Bretagne]} ```

And you can note that even when we only retrieve the data from the local database, it takes time. We are too fat!

And that's only one part of the problem.

teolemon commented 2 years ago

This query should go much faster (595ms), and display unsortable results, to get the user to wait for the sorted response: https://fr.openfoodfacts.org/cgi/search.pl?search_terms=salmon&action=process&fields=brands,product_name,image_small_url,image_front_thumb_url,quantity,nutrition_grades_tags,ecoscore_tags&json=1&page_size=25
Also, should we get all knowledge panel data for the initial search ?
Shouldn't we wait for the user to open the product ?

monsieurtanuki commented 2 years ago

You ask for 25 products max in your query, and we ask for 500 (!) products: KeywordsProductQuery(keywords: value, size: 500) And you ask for 7 fields (more or less given the translated fields and so), and following are the 30 fields we ask for (cf. ProductQuery.fields:

static List<ProductField> get fields => <ProductField>[
  ProductField.NAME,
  ProductField.BRANDS,
  ProductField.BARCODE,
  ProductField.NUTRISCORE,
  ProductField.FRONT_IMAGE, // is in your query
  ProductField.IMAGE_FRONT_SMALL_URL,
  ProductField.IMAGE_FRONT_URL,
  ProductField.IMAGE_INGREDIENTS_URL,
  ProductField.IMAGE_NUTRITION_URL,
  ProductField.IMAGE_PACKAGING_URL,
  ProductField.SELECTED_IMAGE,
  ProductField.QUANTITY, // is in your query
  ProductField.SERVING_SIZE,
  ProductField.PACKAGING_QUANTITY,
  ProductField.NUTRIMENTS,
  ProductField.NUTRIENT_LEVELS,
  ProductField.NUTRIMENT_ENERGY_UNIT,
  ProductField.ADDITIVES,
  ProductField.INGREDIENTS_ANALYSIS_TAGS,
  ProductField.LABELS_TAGS,
  ProductField.LABELS_TAGS_IN_LANGUAGES,
  ProductField.ENVIRONMENT_IMPACT_LEVELS,
  ProductField.CATEGORIES_TAGS_IN_LANGUAGES,
  ProductField.LANGUAGE,
  ProductField.ATTRIBUTE_GROUPS,
  ProductField.STATES_TAGS,
  ProductField.ECOSCORE_DATA,
  ProductField.ECOSCORE_GRADE,
  ProductField.ECOSCORE_SCORE,
  ProductField.ENVIRONMENT_IMPACT_LEVELS,
];

The first step would be to lower the number of requested products from 500 to 25, wouldn't it?

teolemon commented 2 years ago

What we do on Android is infinite scroll by loading in batches of 25 or 50. So if the user starts scrolling, we load additional batches of 25.
And we could enable the personal search button as soon as we receive the 500 results (the query for the 25 first, and the full 500 would be sent in parallel)
To be clear: we wouldn't store the 25 minimized results, they are just here to buy us time

monsieurtanuki commented 2 years ago

Things would make more sense if we could send the food preferences at query time, and let the server filter / reorder the data before replying it. The end-user would get immediately the best products first.

For the moment we are not able to store paged results: the id of the list is just the keyword. If we want paged results, we need to write some code.

Running both 25 and 500 queries in parallel can be tricky.

I don't know what the personae want, but I know that downloading 500 products for one query would sound crazy. A bit like going to page 50 in google.

teolemon commented 2 years ago

Bumping to P0. This is annoying as hell.

monsieurtanuki commented 2 years ago

So what do you have in mind @teolemon?

running in parallel first 25 products (possibly minified) and all products?
running first 25 products (full) and if wanted page 2, 3, ...

In both cases the first 25 products don't mean that much, as they have nothing to do with one's personal ranking parameters. I guess it's the same problem on the website. We'd definitely be better off with the personal parameters being passed to the URL and returning the best 25 products. I don't know how feasible it would be but that would be awesome.

Another solution would be get the first 25 products, which will be good enough to know the total number of products. And say to the user: "we found 25 products that match your search, but there are 653 matches: would you like to get them all?"

teolemon commented 2 years ago

I do not believe in getting the full products for ranking (just the minimum: https://fr.openfoodfacts.org/cgi/search.pl?search_terms=salmon&action=process&fields=brands,product_name,image_small_url,image_front_thumb_url,quantity,nutrition_grades_tags,ecoscore_tags&json=1&page_size=500)
I do not believe in storing search results
Frankly, I would offer even do sorting server side on the full keyword at the potential expense of privacy ("You can get faster and more comprehensive results, but you'll send your anonymized food preferences to the server" "Use server search" "Use private search" "Make my choice default")
I would load the full product on click, and not before.

@stephanegigandet what do you think ?

VaiTon commented 2 years ago

I would load the full product on click, and not before.

This has a huge impact IMHO. Let's do this before thinking about the others

M123-dev commented 2 years ago

Another minimization idea which just came to my mind, we currently request all the fields we show on the product page but it's unlikely everyone needs all of them for ranking.

I for example don't have any big food allergies so for my ranking they can be left out completely.

This of course leads to more complexity but if there is no other way then to query around 500 products for personal search this could make a difference.

monsieurtanuki commented 2 years ago

@teolemon After several days playing with the new "next 25 products" button, how do you feel? I guess the results are much faster and the UX is acceptable, right?

openfoodfacts / smooth-app