Integrate Food Standards NUTTAB database

[Steve7654]

Similar to the FNDDS USDA database, there is an Australian Food Standards publicly available database:

https://www.foodstandards.gov.au/science/monitoringnutrients/afcd/Pages/default.aspx

This database fills in many ingredient gaps from FNDDD and is a very complete nutrient dataset as its largely compiled from scientific studies etc.

Having this dataset available to integrate, like FNDDS would be fantastic. Collectively, whole food ingredients would be a very complete list, likely better than many or most commercial app products.

Is it possible to integrate this dataset too? Is it just a matter of reviewing the terms of use and then bringing it in? I'm not a programmer so wouldn't be able to assist from a technical side, however, if of value I could certainly follow up on the permissions side if there is anything required here?

Cheers, Steve

[davidhealey]

Is there an API we can access from Waistline to search the database?

[Steve7654]

It is provided as an excel file. If that's not appropriate, is it possible to take the excel data, perhaps stored on a server or similar, and create your own API to access it? Note, ive no idea how an API works so i could be way off.

I would've thought, provided the data is available, as it is, we could massage it to whatever format is required to call in into Waistline?

Have a look at the data and you'll see its a great reference for wholefoods with broad nutrition info. Comparable with NDFSS. Collectively there would be very few gaps of food.

Cheers.

[davidhealey]

I'm reluctant to embed data within Waistline and prefer a live API that we can connect to. The database size is fairly small so it may make sense to include it as a core dataset in Waistline but I'm not sure yet. It needs some thinking doing.

[Steve7654]

I appreciate the preference to not embed within waistline, particularly as other datasets may be added in time. That was sort of where I was headed with the datafile downloaded, then remotely stored and an API set up to call it from wherever you wish to put the data (but not locally within Waistline on a device). Hope I made sense.

Would still need to read the terms of use as well. As publicly available I wouldn't expect an issue but be sure to read and understand before implementing.

[vilhelmgray]

I was about to open up a feature request for the food composition table provided by the Japanese MEXT organization, but then I saw a similar one here for Australia.

I think several countries expose their food composition data as files like this. For example, the STANDARD TABLES OF FOOD COMPOSITION IN JAPAN is exposed also as an Excel file. It'd be nice to have an API, but unfortunately these static files may be the only source of data for more obscure regional foods not available in the USDA database or similar.

Hardcoding the data into Waistline would make the app needlessly large, but a solution that might make sense is to define a database format that users can use to load their own local databases of food composition data. That way these Excel files (or really any food table) can be converted to a format that Waistline can optionally load. You could even provide these local database files as an optional download separate from the main Waistline app; or alternatively users can create it themselves if you document the format Waistline would understand.

[davidhealey]

@vilhelmgray Yes I think something like your suggestion is the way to handle this.

[davidhealey]

I've been thinking about this more and I think including additional databases or allowing the user to import them is a very complex task.

We'd have to import the data into the indexeddb which would mean creating additional DB tables. I'm not sure if there is a limit to how much data you can store in indexeddb.

We'd need to implement a database search, currently Waistline doesn't do any searches itself, that is done server side by OFF, we just send the query.

[glacambre]

Almost all countries seem to have their own food databases. A lot of them are listed here: https://www.fao.org/infoods/infoods/tables-and-databases/en/

Some of these databases can be queried online (here are the ones for canada, czechia, estonia, france, new zealand, switzerland...), so it might be possible to scrap the ones you're interested in (although this would probably be a lot of work to maintain).

As far as I can see, none of them seem to offer an official API, but some of them do seem to have pretty neat APIs underneath. For example, when searching for "egg" in the french database, the following request is sent:

{
  "from": 0,
  "size": 5,
  "query": {
    "bool": {
      "must": [
        {
          "multi_match": {
            "query": "egg",
            "fields": [
              "nomIndexEng^2",
              "nomEng"
            ]
          }
        }
      ],
      "should": [
        {
          "prefix": {
            "nomSortEng": {
              "value": "egg",
              "boost": 2
            }
          }
        }
      ]
    }
  },
  "_source": {
    "excludes": [
      "compos",
      "groupeAfficheEng",
      "nomFr",
      "nomSortEng",
      "nomSortFr",
      "nomIndexFr",
      "nomIndexEng"
    ]
  }
}

And the following response is received:

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 31,
    "max_score": 13.677863,
    "hits": [
      {
        "_index": "ciqual",
        "_type": "aliments",
        "_id": "AXeP_5FZWzZQdECwP-GQ",
        "_score": 13.677863,
        "_source": {
          "nomEng": "Egg white, cooked",
          "code": "22008",
          "groupeAfficheFr": "œufs cuits",
          "urlFr": "/aliments/22008/oeuf-blanc-(blanc-d'oeuf)-cuit",
          "grps": [
            {
              "code": "04",
              "lvl": 1
            },
            {
              "code": "0410",
              "lvl": 2
            },
            {
              "code": "041001",
              "lvl": 3
            }
          ],
          "urlEn": "/aliments/22008/egg-white-cooked"
        }
      },
      { ... },
      { ... },
      { ... },
      {
        "_index": "ciqual",
        "_type": "aliments",
        "_id": "AXeP_5F8WzZQdECwP-GT",
        "_score": 12.966907,
        "_source": {
          "nomEng": "Egg, raw",
          "code": "22000",
          "groupeAfficheFr": "œufs crus",
          "urlFr": "/aliments/22000/oeuf-cru",
          "grps": [
            {
              "code": "04",
              "lvl": 1
            },
            {
              "code": "0410",
              "lvl": 2
            },
            {
              "code": "041002",
              "lvl": 3
            }
          ],
          "urlEn": "/aliments/22000/egg-raw"
        }
      }
    ]
  }
}

You can then easily download an excel file for one of these items by using their code, e.g. downloading https://ciqual.anses.fr/exports_excel/food_item_22000_details.xlsx for raw eggs.

As far as I can see, the new zealand database doesn't make searching as nice, but it does have the advantage of offering data about single items as CSV.

Overall this sounds easier than implementing database import and search, but it's still a lot of work :smile:.

[davidhealey]

90% of my entries are quick-add so adding more databases is low on my priority list, so don't hold your breath :D

[user234683]

I've been thinking about this more and I think including additional databases or allowing the user to import them is a very complex task.

We'd have to import the data into the indexeddb which would mean creating additional DB tables. I'm not sure if there is a limit to how much data you can store in indexeddb.

As a first step, you could just read a file that the user provides in some standardized JSON or other format. The app could then add these as custom food items under a category (that way the user can filter them out to see their own actual custom items). It's not the cleanest solution, but if rearchitecting the app to support multiple databases and searching in them is too difficult, it would be better than nothing and would enable people to hack together a ton of integrations.

Personally I've been trying to find an app that can programmatically import food items, because my university puts the nutrition facts of their dining hall information online, and I've been aiming to scrape this and feed it into the app.

So I'm willing to work on a pull request for what I'm describing if you think it's a good idea.

Otherwise, worst case scenario, one strategy could be to export Waistline's database, write a program that adds food items into it, then import the database.

90% of my entries are quick-add

@davidhealey Unrelated, but where is the quick-add feature?

[davidhealey]

So I'm willing to work on a pull request for what I'm describing if you think it's a good idea.

Yes I think an import food data from JSON option is a good idea.

Unrelated, but where is the quick-add feature?

Tap and hold the + icon in the diary.