SiB Directory Components to Display OFN Canada data

[David-OFN-CA]

Our next step is to to populate the Canadian deployment of the SiB directory components with Canadian data.

@karlafalk, myself, and @acsikos to work on connecting SiB directory components with Canadian n8n workflows and webhooks

Task list:

• [ ] changing all the URLs from api.regenerative.org.au to Canadian webhook URLs
• [ ] confirm OFN Canadian test data displaying correctly on Waterloo Digital Food Commons wordpress site

[karlafalk]

I have upload all the n8n workflows that point to API endpoints from this post into n8n Canada's account and pointed to them our Discover Regen airtable base.

I think that the URL needs to replace the text 'api.regenerative.org.au' with 'n8n.openfoodnetwork.org' to have the SiB components pointing to our n8n webhooks.

Not sure on this though as there are 9 SiB workflow in n8n, some with multiple webhooks (19 webhooks in total)...so should there also be 19 urls to change to 'n8n.openfoodnetwork.org'?

[karlafalk]

@mkllnk @rioug So far we have been unable to get the wordpress site to display OFN Canada data.

I have upload the 9 workflow templates for the SiB webhooks from this post into our n8n account, and changed all the Airtable nodes to our credentials. I also adjusted the webhook urls to read 'regen-CAN/' instead of 'regen/' so they aren't confused with your webhook urls. I then activated all the webhooks.

We then pasted the code from this comment into our wordpress site that worked fine at displaying the AUS data, but when we changed the urls to our webhook urls we get nothing on the wordpress site.

We changed the text of the urls from 'api.regenerative.org.au' to 'n8n.openfoodnetwork.org' to correspond with the urls of our n8n webhooks. So for example the url for producers-src was changed from "https://api.regenerative.org.au/webhook/regen/producer-bundle" to "https://**n8n.openfoodnetwork.org**/webhook/**regen-CAN**/producer-bundle, and likewise for all the other urls.
Are we correct in changing 'api.regenerative.org.au' to 'n8n.openfoodnetwork.org' or should we be changing it to something else?

One thing that I am unclear about is that there are n8n webhooks that correspond to the urls we are pasting for: /produce_categories, /assurance_partners, /supporting_evidence_classifications, and /certifications, BUT there is no /product-bundle or /producer-bundle webhooks, there is just /producer and /products in the n8n webhooks. I am wondering if things aren't working because we don't have the '-bundle' part in those first two urls? Could the process be getting hung up at the first 2 urls? Should there be corresponding /producer-bundle and /product-bundle webhooks?

Also, most of the 9 workflows are executing in n8n, but some are getting stuck at the OFN DFC API node with the following error:

When we are adding the code to wordpress site we are getting this error: Mixed Content: The page at '' was loaded over HTTPS, but requested an insecure resource ''. This request has been blocked; the content must be served over HTTPS.

We are also getting this output:

Any advice? Next steps we should take to troubleshoot this? Thanks for your help!

[mkllnk]

I can see a few things that I hadn't told you yet. :wink:

• You need to use producers instead of producer-bundle and products instead of product-bundle. The bundles are for performance only and we just added them in the last weeks. We can add them to your stack when everything is working.
• You are right to use the n8n server in the URL, perfect.
• The OFN DFC calls have to be authenticated. I don't see that in your workflow. We use "Header Auth" and in the credentials the header name is X-Api-Token and the value comes from Airtable as an expression like {{ $json.fields['User API key'] }}.
• You need to use https://n8n..., not http. That's why you get the mixed content message.

[karlafalk]

Thanks @mkllnk for that info. For the OFN DFC calls to be authenticated, I have created this Header Auth credential:

And when I run the workflow I am still getting this error:

Is there something wrong in my code? Why is the expression {{ $json.fields['User API key'] }} and {{ $json.fields.uid }}undefined? I can see those fields in the input.

[mkllnk]

I just looked at the failing run of the producer webhook. It looks like you just called the test URL of the webhook without adding the id of the enterprise. Then Airtable seems to work but because it's missing the id, it gives you a list of all enterprises. Then the DFC API node gets a list instead of a single record and the expression doesn't point to the right variable because the data structure is different for lists.

So when calling that endpoint, you need to add the id to the URL like /producer/?id=rec.....

[karlafalk]

Thanks @mkllnk I can see how that makes sense in that were getting a list from Airtable, so the DFC doesn't know which record to call on.
I made the change you suggested:

But then I am getting this error:

Also, I don't really understand how it should work as I can imagine we wouldn't want to have to go in and manually add the '?id=rec....' for each enterprise we are calling?

[karlafalk]

Also we are still getting this error after we changed to the wordpress code as suggested above.

https://n8n.openfoodnetwork.org/webhook/regen_CAN/producers' from origin 'http://nrw.ocl.mybluehost.me/' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: The value of the 'Access-Control-Allow-Credentials' header in the response is '' which must be 'true' when the request's credentials mode is 'include'.

Maybe this is because that webhook - https://n8n.openfoodnetwork.org/webhook/regen_CAN/producers is one of the workflows that still isn't working because of the DFC node?

This is the results we get on the wordpress site:

[David-OFN-CA]

Thanks @mkllnk I can see how that makes sense in that were getting a list from Airtable, so the DFC doesn't know which record to call on. I made the change you suggested: But then I am getting this error:

Also, I don't really understand how it should work as I can imagine we wouldn't want to have to go in and manually add the '?id=rec....' for each enterprise we are calling?

Hi @karlafalk just flagging that those screenshot links don't appear to be working

[mkllnk]

I made the change you suggested

I'm sorry. I didn't mean to add the id to the URL of the webhook node configuration. The id is added by the component when it requests the data of an enterprise. And when you test your webhook, you need to add it to the URL in the browser.

[mkllnk]

CORS policy

When the workflow fails, it doesn't set the headers correctly. Once the workflow succeeds, the CORS errors should go away.

[karlafalk]

@mkllnk I am not sure I follow?

Should the webhook looks like this then:

Or do I remove the '?id=' part completely?

And when I test I manually add '?id=recK..... to the url in the browser?

[mkllnk]

Or do I remove the '?id=' part completely?

Yes, remove it from the webhook node and add it manually when testing.

[karlafalk]

Thanks @mkllnk! That worked and now all the n8n workflows are working, except for the last one for social medias. I get an error if I leave the code as /{{ $('Webhook').item.json.params.social_media_id }} but if I change it to /facebook it works fine. So not sure why the /{{ $('Webhook').item.json.params.social_media_id }} expression isn't working?

However, we are still not able to push the OFN canada data to the wordpress site. This is the code we are using:

<script type="module" src="https://cdn.skypack.dev/@startinblox/core@0.19"></script>
<script type="module" src="https://cdn.skypack.dev/@startinblox/router@0.12"></script>
<script type="module" src="https://cdn.jsdelivr.net/npm/@startinblox/component-ofn-rpp"></script>

<solid-ofn-rpp
  producers-src="https://n8n.openfoodnetwork.org/webhook/regen_CAN/producers"
  products-src="https://n8n.openfoodnetwork.org/webhook/regen_CAN/products"
  range-product-categories="https://n8n.openfoodnetwork.org/webhook/regen_CAN/produce_categories"
  range-producer-assurances="https://n8n.openfoodnetwork.org/webhook/regen_CAN/assurance_partners"
  range-producer-supporting-evidence="https://n8n.openfoodnetwork.org/webhook/regen_CAN/supporting_evidence_classifications"
  range-producer-certifications="https://n8n.openfoodnetwork.org/webhook/regen_CAN/certifications"
  uniq="directory"
  route-prefix="directory"
  ></solid-ofn-rpp>

And this is still the result we are getting at https://nrw.ocl.mybluehost.me/website_a64d4da6/component-test/:

With these errors:

[mkllnk]

except for the last one for social medias

Did you test it with a full URL like this?

• https://n8n.openfoodnetwork.org/webhook-test/b3c587b7-1fce-47c5-9f49-9371bf79bcee/enterprises_CAN/1904/social_medias/facebook?id=recF8olr2eXtsRvuZ

I just tried that one and it worked. I noticed that your producer workflow lists different URLs though.

- https://openfoodnetwork.ca/api/dfc/enterprises/1904/social_medias/instagram?id=recF8olr2eXtsRvuZ
+ https://n8n.openfoodnetwork.org/webhook-test/b3c587b7-1fce-47c5-9f49-9371bf79bcee/enterprises_CAN/1904/social_medias/instagram?id=recF8olr2eXtsRvuZ

[mkllnk]

we are still not able to push the OFN canada data to the wordpress site.

I see, the CORS errors are persisting. I was hoping that n8n provides the right headers but its settings are slightly different to what we need. We have two options:

• Go into every Webhook Response node and add a response header: access-control-allow-credentials: true
• Set up a proxy server like api.regenerative.org.au and let that one add the headers.

I now also realise that the component needs access to the context.json file which we host on our API server. So maybe it's quickest if I set that up for you now.

[mkllnk]

Oh, actually, I don't have to change anything. You can simply use our API server like this:

• https://api.regenerative.org.au/webhook/regen_CAN/producers

In that response I also noticed that all the links are missing the _CAN in the URLs. You need to go into every workflow and update the URLs.

For testing, I would go to that endpoint and click through to all the URLs to test if they work. The endpoint is cached though. If you need the most recent version, you need to add ?nocache=1 at the end:

• https://api.regenerative.org.au/webhook/regen_CAN/producers?nocache=1

And for other endpoints, where there is a parameter already, you add &nocache=1:

• https://api.regenerative.org.au/webhook/regen_CAN/producer?id=recF8olr2eXtsRvuZ?nocache=1

[karlafalk]

Oh, actually, I don't have to change anything. You can simply use our API server like this:

* https://api.regenerative.org.au/webhook/regen_CAN/producers

@mkllnk Ok thanks! We have now updated the code to this:

<script type="module" src="https://cdn.skypack.dev/@startinblox/core@0.19"></script>
<script type="module" src="https://cdn.skypack.dev/@startinblox/router@0.12"></script>
<script type="module" src="https://cdn.jsdelivr.net/npm/@startinblox/component-ofn-rpp"></script>

<solid-ofn-rpp
 producers-src="https://api.regenerative.org.au/webhook/regen_CAN/producers"
 products-src="https://api.regenerative.org.au/webhook/regen_CAN/products"
 range-product-categories="https://api.regenerative.org.au/webhook/regen_CAN/produce_categories"
 range-producer-assurances="https://api.regenerative.org.au/webhook/regen_CAN/assurance_partners"
 range-producer-supporting-evidence="https://api.regenerative.org.au/webhook/regen_CAN/supporting_evidence_classifications"
 range-producer-certifications="https://api.regenerative.org.au/webhook/regen_CAN/certifications"
 uniq="directory"
 route-prefix="directory"
 ></solid-ofn-rpp>

And I have added _CAN to the webhook response URLs where I could. Many don't have anywhere to updated the url, and look like this:

I tested all the 6 urls above with the ?nocache=1 at the end and they all executed successfully.

I then pasted the updated code into the wordpress site, we are now getting a different error that we are unable to interpret:

Not sure how to trouble shoot the 'TypeError'? Thanks for you help! I feel like we are getting closer!

Also, it looks like the SiB components are fetching the airtable data as these executions appeared in our n8n account (and I didn't trigger them):

[karlafalk]

@mkllnk Good News is that we got the 'Connected Apps' to work in OFN and now when I click 'Allow Data Sharing' it pushes data to our Airtable base.

The only issue I am running into with this that is relevant to the Food Justice Project, is that I get an error in the workflow if my Enterprise belongs to more than one OFN Group. For my Demo Hub, that is part of one group, the n8n workflow that pushes the data from OFN to Airtable works fine, but for my Waterloo Enterprise, that belongs to 2 groups, I get the following error:

The error seems to be in this Filter formula: {uid}={{ $('Merge').item.json ['dfc-b:affiliates'].split("/") [61}}

The item 'dfc-b:affiliates' comes through from the DFC node:

Not sure how to troubleshoot this? Is this another case of the a list being returned and the expression not being able to pick from the list?

Thanks!

[David-OFN-CA]

cheers @karlafalk - thanks for all your follow through on this, and this @mkllnk for your help to date - much appreciated

@mkllnk @kirstenalarsen just flagging that although our first priority for now, is just to get the SiB directory components displaying OFN Canada data - the most recent issue that @karlafalk flags in here last is a key one:

The only issue I am running into with this that is relevant to the Food Justice Project, is that I get an error in the workflow if my Enterprise belongs to more than one OFN Group. For my Demo Hub, that is part of one group, the n8n workflow that pushes the data from OFN to Airtable works fine, but for my Waterloo Enterprise, that belongs to 2 groups, I get [an] error

Our MVP is a directory in which listest enterprises are filterable by the three 'eat local' 'buy local' 'volunteer local' categories. So, the use case that Karla is exploring here is key - if the SiB components aren't able to handle enterprises that belong to 2 or more groups, we'd have to rethink our strategy of avoiding the use of airtable and miniextentions ...

For now, I think the priority remains getting the SiB directory components displaying OFN Canada data, but I justed want to flag that the ability of SiB components to work withenterprises that belong multiple OFN 'groups' is foundational to our current deployment strategy

[kirstenalarsen]

@mkllnk to have a look at it, potentially fix some or all of the n8n issues, and put detailed notes here for @karlafalk to complete or understandf the

[mkllnk]

@karlafalk, I think that the JS error messages come up due to inconsistent data. The producers list is pointing to Australian endpoints, like https://api.regenerative.org.au/webhook/regen/producer?id=recHkUWv8G0lfZFs1.

That raises an error, the component can't resolve the data and fails.

I find it really difficult to find the right workflow for an endpoint. My naming was something like "List producers" and "List producer" so that I know where to look. Now you just numbered them. May I rename them?

Anyway, the producer list is in _SIB workflow1Karlaworking. The URL is defined in the Set node. I changed it to regen_CAN now and the producer list has the right URLs.

Can you go through the workflows and look for URLs that need updating? Every workflow defines URLs somewhere, but it may be in Set or in Code.

[mkllnk]

I also went into the workflow _SIB workflow 3Karlaworking - Show producer and update the Code node to reference the webhook id of social media. And didn't find a workflow for the address. I wonder if I overlooked that one or we don't need it. Let's see.

[David-OFN-CA]

Hi @mkllnk please do rename them! sorry for the trouble

[David-OFN-CA]

@karlafalk let me know if you want to cooridnate around making the URL updates that Maikel requests

[karlafalk]

Can you go through the workflows and look for URLs that need updating? Every workflow defines URLs somewhere, but it may be in Set or in Code.

Ok @mkllnk thanks for renaming everything. That makes sense about the inconsistent data, I didn't realize the urls were at other points throughout the workflows. I have went through and changed all the urls from "regen" to "regen_CAN". I was also wondering if these urls that are also in various workflows may be an issue: https://discover.openfoodnetwork.org.au/context.json (This looks like it is coming from AUS?) _https://n8n.openfoodnetwork.org.uk/webhook/regen_CAN/producers_ (This looks like it is going to the old server? which is now just n8n.openfoodnetwork.org. Should that be changed?)

It looks like maybe not, since now when I check our wordpress site http://nrw.ocl.mybluehost.me/website_a64d4da6/component-test/, the CAN data is coming through. And I can sort by our groups (Demo Hub is part of Volunteer Waterloo) Thanks for looking at things and helping us get to this point!

[mkllnk]

I was also wondering if these urls that are also in various workflows may be an issue

Well spotted. Those URLs get re-written by our proxy server so that it's no problem.

[mkllnk]

While the component displays some data, I also saw some requests failing. I think it's due to invalid data in Airtable.

• Producer recdt6FPIt2jk8KBR has an enterprise id of 0000 which is invalid and it can't find it on the OFN side.

That record needs to be fixed or deleted. In Australia, we also have a view for "public enterprises only" which is referenced in the Airtable node to list enterprises. That allows us to incomplete or work-in-progress data in Airtable.

[mkllnk]

I also noticed that the Show Address node is missing. I now copied it to the forum:

• https://community.openfoodnetwork.org/t/n8n-workflows-for-the-discover-regenerative-api/2954/6

It needs importing into n8n and then its webhook id needs to be copied to the Show Producer workflow in the Code node next to hasAddress.

[karlafalk]

It needs importing into n8n and then its webhook id needs to be copied to the Show Producer workflow in the Code node next to hasAddress.

@mkllnk I have added the Show Address workflow to n8n and added it's webhook url to the Code node of the Show Producer workflow:

Not sure if I did this correctly?

Otherwise, I can't get that new workflow - Show Address to work when testing. I tried to follow the logic with the Show social media workflow when testing -

Did you test it with a full URL like this?

* https://n8n.openfoodnetwork.org/webhook-test/b3c587b7-1fce-47c5-9f49-9371bf79bcee/enterprises_CAN/1904/social_medias/facebook?id=recF8olr2eXtsRvuZ

I just tried that one and it worked. I noticed that your producer workflow lists different URLs though.

But I couldn't get it to work. I see that for testing the ":address_id" needs to be replaced by something (like Facebook for ":social_media_id"....I tried Street or City, followed by the ?id=rec... but couldn't get that to work?

[mkllnk]

I see, the code didn't need the whole URL of the address endpoint, only the id like this:

replaceURL(enterprise, "dfc-b:hasAddress", "888be4b3-5b54-4e98-8808-c6aabc0e375c", record_id);

Oh, n8n has a clash on the webhook id and even after I create a new webhook with a new id it doesn't accept the same name for it. I need to work around that. :thinking:

[mkllnk]

I solved the address problem:

• I changed the address workflow to the webhook: regen_CAN/addresses/:address_id.
• I updated the enterprise workflow to use the webhook id f88de9d2-e059-424b-b90a-c261232297a6/regen_CAN.
• I updated the address workflow to query the Canadian instance in the HTTP Request.

[karlafalk]

@mkllnk Ok that is great! Thank you!

[David-OFN-CA]

@karlafalk is this issue resolved now?

[karlafalk]

@David-OFN-CA Yes! The issue is resolved. So I guess we can close this issue?