search

datafoodconsortium / connector-typescript

The connector makes things easier when integrating with DFC. This is the TypeScript version.
MIT License
2 stars 1 forks source link

Missing taxons in productTypes.json #5

Open RaggedStaff opened 7 months ago

RaggedStaff commented 7 months ago

We are trying to utilise productTypes from an update published last year (driedGoods:flakes etc...). These are unavailable through the current version of the TypeScript Connector.

Can we refresh all of the Taxonomies within the TyepScript Connector please ?

dupreesi commented 7 months ago

Hey @RaggedStaff @lecoqlibre, not sure if I'm missing something but when importing these productTypes, I'm unable to find the dried goods property within the groceryStore.

When logging the grocery store object I'm not seeing a 'DRIED_GOODS' or 'DRIED GOODS' entry.

connector.PRODUCT_TYPES['LOCAL_GROCERY-STORE']...

I can see the value being added (https://github.com/datafoodconsortium/connector-typescript/blob/c0a5a5d3a3fe8b0dd8956f6bac4203159f03db02/test/thesaurus/productTypes.json#L1429) but not sure if its linked up yet?

I also cannot seem to access :local-grocery-store:savory-groceries:flour :local-grocery-store:savory-groceries:ferment :local-grocery-store:savory-groceries:canned goods :local-grocery-store:savory-groceries:snack

Adding my log here as reference:

<ref *2> SKOSConcept {
   _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#local-grocery-store',
   _rdfDataset: DatasetExt {
     _size: null,
     _graphs: [Object: null prototype] {
       'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf': [Object]
     },
     _id: 15,
     _ids: [Object: null prototype] {
       '><': 0,
       'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#local-grocery-store': 1,
       'http://www.w3.org/1999/02/22-rdf-syntax-ns#type': 2,
       'http://www.w3.org/2004/02/skos/core#Concept': 3,
       'http://www.w3.org/2004/02/skos/core#inScheme': 4,
       'https:
 //github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf': 5,
       'http://www.w3.org/2004/02/skos/core#narrower': 6,
       'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#processed-fruit': 7,
       'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#processed-vegetable': 8,
       'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#ready-meal': 9,
       'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#savory-groceries': 10,
       'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#sweet-groceries': 11,
       'http://www.w3.org/2004/02/skos/core#prefLabel': 12,
       '"local grocery store"@en': 13,
       '"épicerie locale"@fr': 14,
       'http://www.w3.org/2004/02/skos/core#topConceptOf': 15
     },
     _entities: [Object: null prototype] {
       '1': 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#local-grocery-store',
       '2': 'http://www.w3.org/1999/02/22-rdf-syntax-ns#type',
       '3': 'http://www.w3.org/2004/02/skos/core#Concept',
       '4': 'http://www.w3.org/2004/02/skos/core#inScheme',
       '5': 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf',
       '6': 'http://www.w3.org/2004/02/skos/core#narrower',
       '7': 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#processed-fruit',
       '8': 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#processed-vegetable',
       '9': 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#ready-meal',
       '10': 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#savory-groceries',
       '11': 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#sweet-groceries',
       '12': 'http://www.w3.org/2004/02/skos/core#prefLabel',
       '13': '"local grocery store"@en',
       '14': '"épicerie locale"@fr',
       '15': 'http://www.w3.org/2004/02/skos/core#topConceptOf'
     },
     _quads: Map(10) {
       '1:2:3:https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf' => [Quad],
       '1:4:5:https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf' => [Quad],
       '1:6:7:https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf' => [Quad],
       '1:6:8:https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf' => [Quad],
       '1:6:9:https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf' => [Quad],
       '1:6:10:https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf' => [Quad],
       '1:6:11:https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf' => [Quad],
       '1:12:13:https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf' => [Quad],
       '1:12:14:https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf' => [Quad],
       '1:15:5:https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf' => [Quad]
     }
   },
   _semanticType: 'http://www.w3.org/2004/02/skos/core#Concept',
   connector: <ref *1> Connector {
     storeObject: ConnectorStoreMap { storeObject: [Map] },
     fetchFunction: [Function (anonymous)],
     factory: ConnectorFactory { connector: [Circular *1] },
     importer: ConnectorImporterJsonldStream {
       context: [Object],
       documentLoader: undefined
     },
     exporter: ConnectorExporterJsonldStream {
       context: [Object],
       outputContext: 'https://www.datafoodconsortium.org'
     },
     MEASURES: SKOSConcept {
       _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/measures.rdf#Measures',
       _rdfDataset: [DatasetExt],
       _semanticType: 'http://www.w3.org/2004/02/skos/core#ConceptScheme',
       connector: [Circular *1],
       DIMENSION: [SKOSConcept],
       UNIT: [SKOSConcept]
     },
     FACETS: SKOSConcept {
       _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/facets.rdf#DFC_ProductGlossary_Facet',
       _rdfDataset: [DatasetExt],
       _semanticType: 'http://www.w3.org/2004/02/skos/core#ConceptScheme',
       connector: [Circular *1],
       CERTIFICATION: [SKOSConcept],
       CLAIM: [SKOSConcept],
       TERRITORIALORIGIN: [SKOSConcept],
       NATUREORIGIN: [SKOSConcept],
       PARTORIGIN: [SKOSConcept]
     },
     PRODUCT_TYPES: SKOSConcept {
       _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf',
       _rdfDataset: [DatasetExt],
       _semanticType: 'http://www.w3.org/2004/02/skos/core#ConceptScheme',
       connector: [Circular *1],
       BAKERY: [SKOSConcept],
       DAIRY_PRODUCT: [SKOSConcept],
       DRINK: [SKOSConcept],
       FROZEN: [SKOSConcept],
       FRUIT: [SKOSConcept],
       INEDIBLE: [SKOSConcept],
       'LOCAL_GROCERY-STORE': [Circular *2],
       MEAT_PRODUCT: [SKOSConcept],
       VEGETABLE: [SKOSConcept]
     }
   },
   PROCESSED_FRUIT: SKOSConcept {
     _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#processed-fruit',
     _rdfDataset: DatasetExt {
       _size: null,
       _graphs: [Object: null prototype],
       _id: 15,
       _ids: [Object: null prototype],
       _entities: [Object: null prototype],
       _quads: [Map]
     },
     _semanticType: 'http://www.w3.org/2004/02/skos/core#Concept',
     connector: <ref *1> Connector {
       storeObject: [ConnectorStoreMap],
       fetchFunction: [Function (anonymous)],
       factory: [ConnectorFactory],
       importer: [ConnectorImporterJsonldStream],
       exporter: [ConnectorExporterJsonldStream],
       MEASURES: [SKOSConcept],
       FACETS: [SKOSConcept],
       PRODUCT_TYPES: [SKOSConcept]
     },
     BOTTLED_FRUIT: SKOSConcept {
       _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#bottled-fruit',
       _rdfDataset: [DatasetExt],
       _semanticType: 'http://www.w3.org/2004/02/skos/core#Concept',
       connector: [Connector]
     },
     CANNED_FRUIT: SKOSConcept {
       _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#canned-fruit',
       _rdfDataset: [DatasetExt],
       _semanticType: 'http://www.w3.org/2004/02/skos/core#Concept',
       connector: [Connector]
     },
     DRIED_FRUIT: SKOSConcept {
       _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#dried-fruit',
       _rdfDataset: [DatasetExt],
       _semanticType: 'http://www.w3.org/2004/02/skos/core#Concept',
       connector: [Connector]
     },
     'FRUIT_IN-COMPOTE': SKOSConcept {
       _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#fruit-in-compote',
       _rdfDataset: [DatasetExt],
       _semanticType: 'http://www.w3.org/2004/02/skos/core#Concept',
       connector: [Connector]
     }
   },
   PROCESSED_VEGETABLE: SKOSConcept {
     _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#processed-vegetable',
     _rdfDataset: DatasetExt {
       _size: null,
       _graphs: [Object: null prototype],
       _id: 14,
       _ids: [Object: null prototype],
       _entities: [Object: null prototype],
       _quads: [Map]
     },
     _semanticType: 'http://www.w3.org/2004/02/skos/core#Concept',
     connector: <ref *1> Connector {
       storeObject: [ConnectorStoreMap],
       fetchFunction: [Function (anonymous)],
       factory: [ConnectorFactory],
       importer: [ConnectorImporterJsonldStream],
       exporter: [ConnectorExporterJsonldStream],
       MEASURES: [SKOSConcept],
       FACETS: [SKOSConcept],
       PRODUCT_TYPES: [SKOSConcept]
     },
     BOTTLED_V
 EGETABLE: SKOSConcept {
       _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#bottled-vegetable',
       _rdfDataset: [DatasetExt],
       _semanticType: 'http://www.w3.org/2004/02/skos/core#Concept',
       connector: [Connector]
     },
     CANNED_VEGETABLE: SKOSConcept {
       _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#canned-vegetable',
       _rdfDataset: [DatasetExt],
       _semanticType: 'http://www.w3.org/2004/02/skos/core#Concept',
       connector: [Connector]
     },
     SOUP: SKOSConcept {
       _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#soup',
       _rdfDataset: [DatasetExt],
       _semanticType: 'http://www.w3.org/2004/02/skos/core#Concept',
       connector: [Connector]
     }
   },
   READY_MEAL: SKOSConcept {
     _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#ready-meal',
     _rdfDataset: DatasetExt {
       _size: null,
       _graphs: [Object: null prototype],
       _id: 10,
       _ids: [Object: null prototype],
       _entities: [Object: null prototype],
       _quads: [Map]
     },
     _semanticType: 'http://www.w3.org/2004/02/skos/core#Concept',
     connector: <ref *1> Connector {
       storeObject: [ConnectorStoreMap],
       fetchFunction: [Function (anonymous)],
       factory: [ConnectorFactory],
       importer: [ConnectorImporterJsonldStream],
       exporter: [ConnectorExporterJsonldStream],
       MEASURES: [SKOSConcept],
       FACETS: [SKOSConcept],
       PRODUCT_TYPES: [SKOSConcept]
     }
   },
   SAVORY_GROCERIES: SKOSConcept {
     _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#savory-groceries',
     _rdfDataset: DatasetExt {
       _size: null,
       _graphs: [Object: null prototype],
       _id: 18,
       _ids: [Object: null prototype],
       _entities: [Object: null prototype],
       _quads: [Map]
     },
     _semanticType: 'http://www.w3.org/2004/02/skos/core#Concept',
     connector: <ref *1> Connector {
       storeObject: [ConnectorStoreMap],
       fetchFunction: [Function (anonymous)],
       factory: [ConnectorFactory],
       importer: [ConnectorImporterJsonldStream],
       exporter: [ConnectorExporterJsonldStream],
       MEASURES: [SKOSConcept],
       FACETS: [SKOSConcept],
       PRODUCT_TYPES: [SKOSConcept]
     },
     'CREPE_AND-GALETTE': SKOSConcept {
       _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#crepe-and-galette',
       _rdfDataset: [DatasetExt],
       _semanticType: 'http://www.w3.org/2004/02/skos/core#Concept',
       connector: [Connector]
     },
     OIL: SKOSConcept {
       _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#oil',
       _rdfDataset: [DatasetExt],
       _semanticType: 'http://www.w3.org/2004/02/skos/core#Concept',
       connector: [Connector]
     },
     PASTA: SKOSConcept {
       _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#pasta',
       _rdfDataset: [DatasetExt],
       _semanticType: 'http://www.w3.org/2004/02/skos/core#Concept',
       connector: [Connector]
     },
     PIE_PASTRY: SKOSConcept {
       _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#pie-pastry',
       _rdfDataset: [DatasetExt],
       _semanticType: 'http://www.w3.org/2004/02/skos/core#Concept',
       connector: [Connector]
     },
     RICE: SKOSConcept {
       _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#rice',
       _rdfDataset: [DatasetExt],
       _semanticType: 'http://www.w3.org/2004/02/skos/core#Concept',
       connector: [Connector]
     },
     SALT: SKOSConcept {
       _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#salt',
       _rdfDataset: [DatasetExt],
       _semanticType: 'http://www.w3.org/2004/02/skos/core#Concept',
       connector: [Connector]
     },
     SEMOLINA: SKOSConcept {
       _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#semolina',
       _rdfDataset: [DatasetExt],
       _semanticType: 'http://www.w3.org/2004/02/skos/core#Concept',
       connector: [Connector]
     }
   },
   SWEET_GROCERIES: SKOSConcept {
     _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#sweet-groceries',
     _rdfDataset: DatasetExt {
       _size: null,
       _graphs: [Object: null prototype],
       _id: 16,
       _ids: [Object: null prototype],
       _entities: [Object: null prototype],
       _quads: [Map]
     },
     _semanticType: 'http://www.w3.org/2004/02/skos/core#Concept',
     connector: <ref *1> Connector {
       storeObject: [ConnectorStoreMap],
       fetchFunction: [Function (anonymous)],
       factory: [ConnectorFactory],
       importer: [ConnectorImporterJsonldStream],
       exporter: [ConnectorExporterJsonldStream],
       MEASURES: [SKOSConcept],
       FACETS: [SKOSConcept],
       PRODUCT_TYPES: [SKOSConcept]
     },
     BISCUIT: SKOSConcept {
       _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#biscuit',
       _rdfDataset: [DatasetExt],
       _semanticType: 'http://www.w3.org/2004/02/skos/core#Concept',
       connector: [Connector]
     },
     CONFECTIONERY: SKOSConcept {
       _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#confectionery',
       _rdfDataset: [DatasetExt],
       _semanticType: 'http://www.w3.org/2004/02/skos/core#Concept',
       connector: [Connector]
     },
     HONEY: SKOSConcept {
       _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#honey',
       _rdfDataset: [DatasetExt],
       _semanticType: 'http://www.w3.org/2004/02/skos/core#Concept',
       connector: [Connector]
     },
     JAM: SKOSConcept {
       _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#jam',
       _rdfDataset: [DatasetExt],
       _semanticType: 'http://www.w3.org/2004/02/skos/core#Concept',
       connector: [Connector]
     },
     PASTRY: SKOSConcept {
       _semanticId: 'https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#pastry',
       _rdfDataset: [DatasetExt],
       _semanticType: 'http://www.w3.org/2004/02/skos/core#Concept',
       connector: [Connector]
     }
   }
 }
RaggedStaff commented 7 months ago

Thanks @dupreesi

The Concepts you mention (flour, ferment etc) are not currently present in productTypes.json. I think they were added to the list after I updated the productTypes last year. I've added them on VOCBench & need to get them reviewed/approved & we'll release asap.

I don't understand why the driedGoods concept isn't available... @lecoqlibre do you have any ideas?

lecoqlibre commented 6 months ago

@RaggedStaff It seems there is an issue within VocBench, the dried_goods is not a narrower of local-grocery-store:

image

The connector is using the narrower property to set the parent/child relationship.

@RaggedStaff Can you make a new PR with a fix please?

lecoqlibre commented 6 months ago

I also noticed that LOCAL_GROCERY-STORE should be LOCAL_GROCERY_STORE with a "_" instead of a "-". There are several other terms which are not correctly spelled.

I should fix that in the connector and I wanted to know if I can release a new version without deprecating the wrong terms? So @dupreesi you will probably have to replace the some existing terms in your code if you already used them.

@RaggedStaff Do we have other users of the connector who might be concerned?

RaggedStaff commented 6 months ago

I also noticed that LOCAL_GROCERY-STORE should be LOCAL_GROCERY_STORE with a "_" instead of a "-". There are several other terms which are not correctly spelled.

:thinking: I'm confused - I thought we agreed to use camelCase for Taxon URI's, this category predates the agreement of that standard. I thought we also agreed standardisation would be part of v2.0, as it's a breaking change.

I should fix that in the connector and I wanted to know if I can release a new version without deprecating the wrong terms? So

@RaggedStaff Do we have other users of the connector who might be concerned?

@mkllnk, @rioug - heads up: will this impact the OFN Product Links work? Hoping the SKOS-Concepts are not hard-coded. :crossed_fingers:

RaggedStaff commented 6 months ago

@RaggedStaff It seems there is an issue within VocBench, the dried_goods is not a narrower of local-grocery-store:

The connector is using the narrower property to set the parent/child relationship.

@RaggedStaff Can you make a new PR with a fix please?

I'm reviewing all the narrower/broader relationships across the Taxonomy & will submit a PR shortly.

RaggedStaff commented 6 months ago

@lecoqlibre I'm noticing a few newer concepts that are defined by SKOS:broader only. I suspect that's how VOCbench works. Is there a way we can either:

  1. Fix the connector to use broader, rather than narrower ?

  2. Force VOCbench to generate the inferred relationships ?

lecoqlibre commented 6 months ago

🤔 I'm confused - I thought we agreed to use camelCase for Taxon URI's, this category predates the agreement of that standard.

I'm talking about TypeScript code and we can't use hyphens in variable names. We provide developers an API to easily use our taxons when programming. So instead of writing the full URI of the taxons, developers could use for instance PRODUCT_TYPES.TOMATO.ROUND_TOMATO (which is automatically replaced by the full URI https://github.com/datafoodconsortium/taxonomies/releases/latest/download/productTypes.rdf#round-tomato). To generate the name of the variable, the code is looking at the last part of the URI and transforms it to uppercase with "-" replaced by "_", so ROUND_TOMATO.

To avoid the need for this transformation, maybe we should change our URI naming policy. The primary need was to stop having names starting with a number as said in the discussion https://github.com/datafoodconsortium/taxonomies/discussions/13. I will make a new proposal on this discussion.

But at the end, it's probably up to the developers of the connector to choose the case they want. Because some programming languages prefer a case instead of an other, right?

I thought we also agreed standardisation would be part of v2.0, as it's a breaking change.

If you remember, we said that we could avoid breaking changes using the "deprecated" tag, right?

But I just realize that it could lead to problem if a platform is exchanging data having a deprecated URI with a platform using newest URI. For instance, two products will not have the same type except if we add a owl:sameAs predicate to reconcile the URI. Doing so would need the platforms to support inference. This would be possible to achieve within the connector.

Fix the connector to use broader, rather than narrower ?

Maybe we could, we should open an issue for that. Or even maybe a discussion about how to make the taxons API.

Force VOCbench to generate the inferred relationships ?

Could you look at it @RaggedStaff ?

lecoqlibre commented 6 months ago

@dupreesi We fixed the taxonomy file but it's not officially released yet. In the meantime, you can use the file on the dedicated branch: https://github.com/datafoodconsortium/taxonomies/blob/fix/issue27/productTypes.json.

With this file you will be able to use the following:

connector.PRODUCT_TYPES['LOCAL_GROCERY-STORE'].DRIED_GOODS;
connector.PRODUCT_TYPES['LOCAL_GROCERY-STORE'].DRIED_GOODS.FLOUR;
connector.PRODUCT_TYPES['LOCAL_GROCERY-STORE'].SAVORY_GROCERIES.FERMENT;
connector.PRODUCT_TYPES['LOCAL_GROCERY-STORE'].SAVORY_GROCERIES.CANNEDGOODS;
connector.PRODUCT_TYPES['LOCAL_GROCERY-STORE'].SAVORY_GROCERIES.SNACK;

@dupreesi @RaggedStaff Is it OK to fix the name of categories by replacing all "-" with "_"? So LOCAL_GROCERY-STORE would become LOCAL_GROCERY_STORE. This way you will be able to access it with the dot notation instead of the brackets like connector.PRODUCT_TYPES.LOCAL_GROCERY_STORE. This will impact all the categories that had a wrong name so you might need to change some code.

mkllnk commented 6 months ago

will this impact the OFN Product Links work? Hoping the SKOS-Concepts are not hard-coded.

We don't have them hard-coded yet. We introduced the UI to add DFC URIs to OFN product types but it hasn't been used yet. The long-term goal is also to do that mapping in code and then we could use the connector for this. But first we need users to create the mapping. :wink:

The sooner you change this the less impact it will have on us.