search

openfoodfoundation / openfoodnetwork

Connect suppliers, distributors and consumers to trade local produce.
https://www.openfoodnetwork.org
GNU Affero General Public License v3.0
1.12k stars 724 forks source link

OFN DFC Products #9170

Closed lin-d-hop closed 11 months ago

lin-d-hop commented 2 years ago

Description

I want to be able use the DFC standard to interface with OFN.

The OFN v0 Products endpoint gives enough functionality for a simple interface with the DFC. By linking OFN v0 product endpoint to the DFC we can build integrations from the DFC standard and then change OFN under the hood without impacting the integrations. As the v0 products endpoint has everything that is needed, this will mean that we can deliver on DFC requirements and AU API requirements and build integrations that will be supported while we make major changes to the OFN product model.

Another advantage is that we'll minimise some of the risk of using the OFN v0 endpoint. For example some fields if updated via the endpoint may break existing products -> updating unit_type of a variant can break other products. Due to the split between product and variant models in OFN updating fields on the product can have adverse effects. This caused a lot of headache when writing product import, so any update functionality via the API will run the same risks. We'll be looking at this in #9069.

This is a proposal, and currently a placeholder. If we have @mkllnk's buy in to the approach we'll set up a kick off with Garethe and Maikel and others to get this work started.

The exact data fields and API functions we'll be serving.... coming soon! Basic CRUD but with a more limited set of fields.

TODO - Understand how we are going to map OFN producer price to the DFC model.

Acceptance Criteria & Tests

  1. A DFC Products compliant endpoint that serves OFN data.
  2. The DFC connector should be independent of OFN code, and should interface with the OFN v0 API
lin-d-hop commented 2 years ago

Relevant conversation on Slack: https://openfoodnetwork.slack.com/archives/C018S101R4J/p1652184870004909

RaggedStaff commented 2 years ago

As per comments on Slack... I suggest we create a default DFC:Offer & DFC:customerCategory to hold a "Producer Price" or "default price" for a Product. Since OFN isn't utilising these classes we can create default values for the DFC output.

At this stage DFC isn't read to capture multiple offers/prices per product so we should be able to modify the Prototype to display a price, if that's helpful/necessary.

I don't think there should be any issues with this approach but very happy to be corrected! :upside_down_face:

RaggedStaff commented 2 years ago

Required JSON-LD output format is available here: https://datafoodconsortium.gitbook.io/dfc-standard-documentation/appendixes/practical-examples/version-1.7

Context data has been abstracted to here: http://static.datafoodconsortium.org/ontologies/context.json

This includes a number of reference data ontologies (for look up fields). When looking through the JSON-LD schema, you'll need to reference the prefixes (dfc-b, dfc-pt etc..) against this list: 

"dfc-b": "http://static.datafoodconsortium.org/ontologies/DFC_BusinessOntology.owl#",
"dfc-p": "http://static.datafoodconsortium.org/ontologies/DFC_ProductOntology.owl#",
"dfc-t": "http://static.datafoodconsortium.org/ontologies/DFC_TechnicalOntology.owl#",
"dfc-u": "http://static.datafoodconsortium.org/data/units.rdf#",
"dfc-pt": "http://static.datafoodconsortium.org/data/productTypes.rdf#",
"dfc-a": "http://static.datafoodconsortium.org/data/claims.rdf#",
"dfc-d": "http://static.datafoodconsortium.org/data/dimensions.rdf#",
"dfc-c": "http://static.datafoodconsortium.org/data/certifications.rdf#",
"dfc-g": "http://static.datafoodconsortium.org/data/geoOrigin.rdf#",

All files in data/ (that end *.rdf#) are reference data and will have constrained values that need to be mapped to.

I will post a full breakdown of the JSON-LD tomorrow.

mkllnk commented 2 years ago

Thanks for that. I have to admit that it's still overwhelming for me as I haven't worked with JSON-LD before. But I'm sure we can work it out.

Off topic: I couldn't find a way to suggest a change in the documentation but this example doesn't seem right defining weight in cm (line 101):

        {
          "@type":"dfc-b:PhysicalCharacteristic",
          "dfc-b:hasUnit": "dfc-u:Centimetre",
          "dfc-b:hasPhysicalDimension": "dfc-d:Weight",
          "dfc-b:value": "300"
        }
RaggedStaff commented 2 years ago

Thanks for that. I have to admit that it's still overwhelming for me as I haven't worked with JSON-LD before. But I'm sure we can work it out.

Hearing this seems overwhelming. If it helps JSON-LD has eactly the same syntax as JSON, it just uses externally defined schemas (those @context files I posted above)

Off topic: I couldn't find a way to suggest a change in the documentation but this example doesn't seem right defining weight in cm (line 101):

Yep, definitely not the finest sample code! :wink: I'll suggest a correction to the FR team, thanks.

lin-d-hop commented 2 years ago

The important nuance here is that at the OFN Product level this first 'Offer' will be for sale only to hubs. Once we move toward implementing shopfronts each shopfront will have it's own Offer(s). And if the hub is a Producer Shop then the hub will need to be able to have:

So perhaps we need to include a DFC:CustomerCategory that actually is not at all General. So as Garethe suggests 'ProducerPrice' might be a good fit. In the first iteration the 'ProducerPrice' should probably be visible to a CustomerCategory that includes only the Producer themselves. This will prevent any other users being able to access this price while we are using it in this way. The important thing is that we ensure it is clear that this Offer is distinct from other Offers because cost price and sale price will be handled in the same way unlike within OFN.

As an aside, this is a good thing (in case that's not clear). Cost Price and Sale Price aren't single iteration things. One enterprises cost price is another's sale price in a recursive way. This fundamental problem with the OFN Product model is handled by the DFC and is where we are going with the Network feature (and why I think the best approach pre-network is to serve OFN Product v0.5 API over the DFC).

As a first round we are probably looking at input fields akin to Product Import:

@kirstenalarsen do those fields meet your needs?

kirstenalarsen commented 2 years ago

Hi all.

I have done some update of the API Use Cases doc, moving the product related ones upfront into 'up next' and clarifying the use cases I have, which boil down to CRUD products and variants from external sources, including on_hand and price.

I would love to extend the scope of this to scoping by Shop and/or Order Cycles to that we can get and watch particular products / stock levels etc, but as this will go far into Offers I suspect it will be well out of scope for this first bit.

I have also mapped the existing bulk_products endpoint to identify the particular fields that are needed for creating a product and variant, as per the bulk import file. Some of these map directly and others are needed to calculate or retrieve the correct values - particularly relating to 'unit fields' above. Until we have major refactoring in place we need these fields at a minimum to still be available somehow

Just to have all the info linked, I also updated the Products Endpoint google doc from previous discussions to highlight the fields that I think we need here as well.

Depending on timing of refactoring / Network 2.0 . . it would be really handy to get array of 'visible' inventory overrides - would only need Enterprise ID, on hand and price. This could be a first step towards 'offer' as is basically an enterprise owner and their stock/price level. We would then need to point it at a different product down the track

RaggedStaff commented 2 years ago

Off topic: I couldn't find a way to suggest a change in the documentation but this example doesn't seem right defining weight in cm (line 101):

        {
          "@type":"dfc-b:PhysicalCharacteristic",
          "dfc-b:hasUnit": "dfc-u:Centimetre",
          "dfc-b:hasPhysicalDimension": "dfc-d:Weight",
          "dfc-b:value": "300"
        }

This has been corrected now: JSON-LD, v1.7 ! :tada:

RaggedStaff commented 2 years ago

As promised (sorry for the delay). Here's a breakdown of my understanding of the JSON-LD 1.7 (current interface for prototype). Hope it all makes sense, do ask me to clarify anything that is still gibberish! :roll_eyes:

@base the URI (base URL) for the platform. This will be used to prefix any URI’s supplied later on (e.g. SuppliedProduct.id)

All objects are identified by @id: the URI (Uniform Resource Identifier, can be a URL, must be a unique way of identifying a resource, brief overview here), in this case it’s the route (probably the API endpoint as a relative URL) the platform used to locate the resource on OFN. It doesn’t matter if some of the actual endpoints don’t exist at this stage (the Prototype only uses product endpoints at present) as long as the id will be unique within OFN.

Person (affiliates to an Enterprise) and is the user making the request (admin user linked to Les Communs OIDC account authorising the API call)

Enterprise: (supplies suppliedProduct, manages catalogItem, owns Brand and defines customerCategory) the business that user is affiliated to (will that be more than 1 potentially?)

Customer Category links catalogItem to offer (which holds price)

Brand attribute of suppliedProduct. Free text

SuppliedProduct contains majority of product data including:

catalogItem This is the listing of an item (typically a product) in a Sales Catalog (shop front?) so has a few attributes not on SuppliedProduct, differentiated by URI (will have same OFN Product Id), can include:

offer The offers that define prices for a catalogItem, by customerCategory, can include:

kirstenalarsen commented 2 years ago

I am quite confused about how the offers and Customer Categories are going to interplay with OFN fees. If I understand what you're saying above @lin-d-hop , it sounds like we will only be able to change the 'final sale price' on a product for a Hub shop i.e. the price including fees. This is NOT the price that most of our users are manipulating. If they have fees these go on top of the underlying 'cost' price, and that is the price that changes / is managed. So I would think that any Enterprise that has manage product permissions needs the ability to change the 'cost price' at the producer level, whether that is for a Producer Shop or a Hub

lin-d-hop commented 2 years ago

@kirstenalarsen Actually what I'm saying is that we'll only be able to change the Producer Price. We won't be touching anything to do with fees or final sales price etc at this stage.

lin-d-hop commented 2 years ago

Thanks @RaggedStaff @mkllnk and @kirstenalarsen for the work you've done finding the relevant fields on the APIs and our requirements.

I've taken an extra step in this spreadsheet (on the Mapping sheet) to understand how we can create something in which:

@RaggedStaff There are a few questions for you, in particular:

RachL commented 2 years ago

we need to lose when going via the DFC

Just a note to add that the DFC can also evolve. We can make proposals if you think they are needed :+1:

RaggedStaff commented 2 years ago

Thanks @lin-d-hop, tried to answer as best I can.

@RaggedStaff There are a few questions for you, in particular:

* How is Tax information conveyed within the DFC? In OFN Products have tax categories which influence the type of tax that will be applied to them when sold in a shopfront. So for now we want to ignore the actual application of tax (which depends on the location of the buyer) but we do want to know how to track the taxability of the product itself.

As far as I can tell... it isn't... yet! I'll double check this with the Ontology team and we can get it added.

I would expect it to be captured as a product attribute, but I can't see it anywhere. Actual tax probably has to hang off transaction but we should be able to propose an additional attribute around Sales Tax Category perhaps with a Tax % attribute, maybe a Geographic Jurisdiction attribute as well. It would probably sit on Supplied Product.

Does that make sense for OFN? Could you provide some examples from OFN for us to work with?

* Within OFN products can have a totally flexible 'unit type'. For example we don't limit to 'items' and instead allow products to be 'bunches' or 'dozen' or 'packs'. Does DFC allow this flexibility at all or will this be something we need to lose when going via the DFC?

Hmm, yes: units are constrained in DFC. units.rdf is the reference list for all units, we can add any missing standard units in there (package & piece are already there, bunch & dozen aren't). I don't think we can make it a free text field, is that what it is in OFN? If so how do you handle weight variant relationships? :confused:

lin-d-hop commented 2 years ago

In OFN there is: unit type = eg grams, litres, items unit value = eg 500, 1, if unit_type = items there is an additional field to describe eg bunches, loaves.

OFN does some very clever stuff around units.

It would be great to pitch both tax and units to the DFC. Tax we can't live without. Units we can if necessary.

RaggedStaff commented 2 years ago

Ooo, shiny! :sunglasses:

We might be able to work that out - we could include an optional text for a unit description that can be populated for generic types. Also noticing units don't currently carry any indication of a datatype within them, I think they probably should. :thinking:

RaggedStaff commented 2 years ago

Ok. a field (has)Tax, is being added to the Price component, as part of some refactoring/mapping to other standards. Details are here.

lin-d-hop commented 2 years ago

A boolean? Most places have multiple possible tax rates....

RaggedStaff commented 2 years ago

Ah, no sorry... I think text (it's xml, so all text really :stuck_out_tongue_winking_eye:), the "has" refers to the relationship: Price (can) has Tax.

lin-d-hop commented 2 years ago

Ok so for the UK collab we need to define our fields here. Maybe we can define a Tax object?

RaggedStaff commented 2 years ago

There will be a Tax object, linked to Price on an Offer. Not clear it will have anything beyond some text saying what the Tax is called. Would make sense to standardise it - Taxes are usually standardised! :wink: I've asked for clarification from the ontology team.

lin-d-hop commented 2 years ago

locally standardised. Globally wildly diverse. Fortunately others have solved this problem (global taxes work reasonable well in OFN) so we can just copy best practice.

RaggedStaff commented 2 years ago

Ok, update on Tax. Currently envisaged as just capturing Tax Amount, linked to Price. Assuming we can calculate that fairly easily? Not clear if that will be sufficient for OFN requirements.

Ping @lin-d-hop : Can you confirm whether we need some way to identify the Tax rate/category applied at the Product level? I think that was what you were asking for, if so I can go back and confirm that with the Ontology team.

drummer83 commented 1 year ago

@lin-d-hop This one has been reopened and is now in Test Ready. What should we test here?

lin-d-hop commented 1 year ago

\this was moved out of test ready two weeks ago. label removed. Its the holding epic so nothing to test in this

mkllnk commented 1 year ago

Maybe a quick summary of where we are at:

kirstenalarsen commented 1 year ago

Thanks @mkllnk that's incredibly helpful! Just to that last point, another option would be to add our own UI to exchange data 'generally' right? And/or to N8N, Shopify or whatever we choose? Definitely design needed, but exciting that we're at the point of thinking about this. Ping @mariocarabotta

mkllnk commented 1 year ago

another option would be to add our own UI to exchange data 'generally' right? And/or to N8N, Shopify or whatever we choose?

One thought @dacook and I had was to use webhooks for this. We don't have a standard format for webhooks yet and we could just say now that all webhooks will deliver the payload as DFC data. That will mean that the user can use the webhooks UI to say which event will be sent to which platform to trigger any actions there. That could be product creation or update could be synced to another platform, order placed could be sent to another platform or order cycle close could go to N8N which can then retrieve a supplier report via the API and put it into a spreadsheet.

I haven't looked into creating N8N sources yet but it would probably a good feature to integrate DFC into N8N as input source besides databases, emails, spreadsheets etc.

lin-d-hop commented 1 year ago

@RaggedStaff @mkllnk Would either of you object to me closing this issue as done? Once we are testing the DFC with real products then no doubt we will identify the missing attributes that are needed. Other than that is there anything here we need to keep this open for?

EDIT: Okay sorry this is obvs the epic for other bits.... I'd just love to tidy up some of these DFC threads.... but maybe not just yet

kirstenalarsen commented 1 year ago

TBC in product circle re. funded feature management

RachL commented 11 months ago

Closing as most of the issues are tracked under this epic now: https://github.com/openfoodfoundation/openfoodnetwork/issues/10289