Aggregate Culture Mauricie events (Shawinigan, Cogecoa and Tourisme Mauricie) and Caligram events using APIs

[saumier]

Our next CMS client is Culture Mauricie.

They were using Caligram CMS, and are now switching to Footlight CMS because Caligram is closing.

In the past, they were loading events from these 3 APIs. Our challenge is to design and build a tool to load events from APIs in a generic way so that we can repeat with other customers without releasing new code.

This can be split into separate issues to track work, but this issue should be kept as the high-level issue for the design of the generic solution.

Shawinigan

Culture Shawinigan is a city arts council for the town of Shawinigan

API to a local ticketing platform https://api.ticketacces.net/v1/showtimes?KeyID=CULTURESHAWI&KeySecret=wMalrXUtnFEMIK7PDENGbPxRfiGY&Domains=shawinigan.ticketacces.net&Language=fr

Important: this API must not be called more than 2 or 3 times a day.

Cogeco

https://www.amphitheatrecogeco.com/ l’Amphithéâtre Cogeco is a theatre in a city called Trois-Rivieres, one of the oldest cities in Quebec.

API http://amphitheatrecogeco.com/json?module=spectacle&method=getspectacles

FYI: This site is already being crawled by Footlight Console as an experiment. https://console.footlight.io/events?seedurl=amphitheatrecogeco-com&view=list

Tourisme

https://www.tourismemauricie.com/ For a short while we will load event from this site, but a few months later they will also switch to Footlight CMS as a fourth client (we will then have to figure out how to share events between calendars in the CMS.)

API https://www.tourismemauricie.com/wp-admin/admin-ajax.php?action=caligram

[sahalali]

@saumier Can we discuss this in detail after the standup?

[sahalali]

@tammy-culture Firewall is blocking to access the URL for Tourisme Mauricie https://www.tourismemauricie.com/wp-admin/admin-ajax.php?action=caligram.

Can you please see if someone can help us to get access to the URL?

Screen shot:

[tammy-culture]

Hi Suhail, I will look into this and get back to you. Thanks.

[saumier]

@sahalali @dev-aravind I suggested in the standup this morning that you work together on this one. Something that could be done in the analysis is to document the mapping between each external API source and the CMS. The analysis document could have a table for each API and explain how it would be mapped to CMS.

A question that we should ask ourselves in the analysis is whether we should use the 3 APIs or instead crawl the websites with Console.

FYI @tammy-culture had already asked Kim in the past to crawl Cogeco has an experiment https://console.footlight.io/events?seedurl=amphitheatrecogeco-com&view=list so we can compare the Cogeco API versus crawling Cogeco website to decide which is better. If the data in the Cogeco API is very similar to that obtained by the Console, then we should choose the API for reliability and maintainability. But if the API has many fewer fields then we should consider the Console approach. So to answer this question we will need to know what are the differences in the data as part of the analysis.

[tammy-culture]

@sahalali @saumier I spoke with with Tourism Mauricie this morning and they will ask their tech team to provide access to theire API. I will keep you all posted.

[dev-aravind]

@saumier First Draft of the mapping between fields

[saumier]

@dev-aravind Here are the links to the websites.

https://www.amphitheatrecogeco.com

https://www.tourismemauricie.com

[dev-aravind]

@saumier second draft of the mapping between fields.

[tammy-culture]

@sahalali, @saumier

Hello, I just heard back from Tourism Mauricie and they said to try the the api again. Let me know how it goes.

Thanks, Tammy

[sahalali]

@tammy-culture Thanks for the update. It works now.

[dev-aravind]

@sahalali I've added Tourism Mouricie data too to our mapping.

[tammy-culture]

@sahalali

Hello everyone, I received an important note in regards to the Culture Shawinigan API. The API should not be called more than 2 or 3 times a day. (Very important).

Thanks.

[saumier]

@dev-aravind @sahalali To start the project with Culture Mauricie @tammy-culture would like us to start with the Amphithéâtre Cogeco API and load it into the CMS as soon as possible so she can work with them to setup their calendar. I propose we use this API to continue learning how best to do this, and create a separate issue to track.

[tammy-culture]

@sahalali @saumier

Please let me know if this is the right place to bring this up.

It was brought to my attention by our new client, that the API used to load events for Culture Shawinigan does not include all the events listed on their site: https://cultureshawinigan.ca.

That is because the API used is from a ticketing platform, so only ticketed paid events are covered. It does not include any of the free events that are listed on the Culture Shawinigan site. The client would like to know what potential solutions and options could be available to aggregate data (ie. scraping the site or a using an excel or google sheet to upload? - just a thought)

[sahalali]

@saumier We have the option to import events and related entities using an Excel sheet in a specific format. We can share the footlight-cms supported format. I would suggest the development team do the import to CMS because lets validate the data before importing.

[sahalali]

@tammy-culture I've encountered the same issue with accessing the API https://www.tourismemauricie.com/wp-admin/admin-ajax.php?action=caligram.

The error message is "Access denied". I noticed this issue today, not sure if it is happening due to the change in IP and location.

[tammy-culture]

@sahalali Can you provide a screen shot of the error message and I will send it along. Thanks.

[sahalali]

@tammy-culture Please find the screenshot of the error page.

[tammy-culture]

@sahalali Thank you.

[tammy-culture]

@sahalali I have another api to analyze and integrate to the Culture Mauricie calendar. It is an API from Culture Trois Rivières. Please let me know if it is usable. Thanks.

https://api.tuxedoticket.ca/documentation

[sahalali]

This is the API documentation we prepared for cms-api and admin-api. To get data using this URL, we will need to authenticate. So if the client could provide us a login credential, then we could start analyzing the data.

[tammy-culture]

@sahalali The Tourism Mauricie api should work now.

[sahalali]

@tammy-culture The issue still exists at my end. I am unable to access the API endpoint. The screenshot is atatched.

[tammy-culture]

Thanks @sahalali I will look into it.

[tammy-culture]

@sahalali Hi, the Tourism Mauricie API should be working now. Thanks.

[sahalali]

Estimation: 7 days

1. Add support to upload RDF using footlight-aggregator (RDF data following Schema.org).
2. Prepare GithubAction: a. Run Ontotext/refine b. Prepare and verify mapping files c. Convert data to RDF using REFINE. d. Call Footlight-aggregator API to update events.

The confidence of estimation is Medium, the estimation may vary a little during development.

[sahalali]

This is ready to schedule.

[saumier]

@sahalali Let me know as soon as you have the github workflow that runs ontotext/refine with the command transform passing in the csv filename and mapping filename. I will use the same workflow for importing Ville de Laval. Also related to https://github.com/culturecreates/artsdata-orion/issues/3

[tammy-culture]

@sahalali @troughc Is there a way for me to get a google sheet copy of the data available in each of these 2 APIs? (Shawinigan and Cogeco). Many thanks.

[sahalali]

@tammy-culture The Shawningan response in XML format and Cogeco in JSON. As we have not imported both sources into CMS, we will need to convert the data to CSV.

[tammy-culture]

@sahalali Is this another way that I can view the data that is easier for you before we load it into the CMS? I would like to see what type of mappings that Culture Mauricie will have to do (example mapping between event types)

[saumier]

@tammy-culture Have you seen this mapping done by Dev? It has a single event as example. https://docs.google.com/spreadsheets/d/1gORhHpQrpKw4D9bJquil6GbNfpRM-F2R-3rSH1LCKb8/edit#gid=0

[tammy-culture]

@tammy-culture Have you seen this mapping done by Dev? It has a single event as example. https://docs.google.com/spreadsheets/d/1gORhHpQrpKw4D9bJquil6GbNfpRM-F2R-3rSH1LCKb8/edit#gid=0

Yes I have thank you. @sahalali What I actually need rather than all the data are just the concepts from each API mapping to event types (additionalType in Footlight CMS):

• Tourisme Mauricie (categorie)
• Cogeco (category.name)
• Shawinigan (eventgenrename, eventratingname, eventsubgenrename, showtimemaincategoryname)

Can you provide this in a spreadsheet. please.

Many thanks, Tammy

[sahalali]

@troughc Please find the list of event types under each API in this Google sheet.

The Tourisme Mauricie is NOT accessible to me today, So I've added event types from a saved old response.

[sahalali]

@tammy-culture The Tourisme API is not accessible to me. Can you please help me out to get it accessible? Attached is the screenshot of the error message.

[sahalali]

@tammy-culture To bypass the issue with repeated IP address changes. We can use our staging-server-3 to call the Tourisme Mauricie API always. We use a static IP address for the server, so we need not have to worry about the IP address change.

Can you please work with Tourisme Mauricie to whitelist the IP address: 3.97.145.197. Please let us know once done.

[tammy-culture]

@sahalali The IP address is whitelisted, done.

[tammy-culture]

Hi @sahalali,

Caligram does not have the ability to provide a CSV file for Culture Mauricie, but they do have 2 APIs (which I never thought to ask for).

Here are the two API access points they provided:

(1) Get a list of all upcoming events: https://culturemauricie.caligram.com/api/events

(2) Obtain detailed information about an event (necessary to obtain all the dates of an event): https://culturemauricie.caligram.com/api/events/[id]

For more information, here's the API documentation: http://api.caligram.com

Can you use the above APIs? Or do we scrape/crawl the site?

Please let me know when you can.

Thanks.

[sahalali]

@tammy-culture We can use the above APIs. We will be able to upload the data using the footlight aggregator with a new API with minimum development effort.

[sahalali]

@tammy-culture is Caligram API the 4th source for the Culture Mauricie calendar?

[tammy-culture]

@sahalali The Caligram API is for the current Culture Mauricie cultural calendar called (D'ici.ca). When we decide to officially do a LAST transfer of data from Caligram API to the Culture Mauricie Calendar, this API will no longer be needed as a source. It is being used only to transfer events from the current calendar to Footlight CMS.

Does this make sense?

[sahalali]

Yes. That make sense.

[sahalali]

@saumier How can I add videoUrl and contactPoint to Events in RDF? I could not find a property from Schema.org for the same. I know there are classes for VideoObject and ContactPoint to represent the data however I am unsure what property to use under the event.

cc @tammy-culture

[saumier]

@sahalali In the Footlight console I use the following.

For the video URL you can use:

"http://schema.org/recordedIn":  "https://www.youtube.com/embed/-LtoFSrT3Bk"

And for the contact point you can use:

"http://schema.org/contactPoint" : 
 {
    "@type": "http://schema.org/ContactPoint",
    "http://schema.org/email": "infoloisirsbma@gatineau.ca",
    "http://schema.org/telephone": "8199312902",
    "http://schema.org/url": "http://thewebsite.com",
    "http://schema.org/name": [
      {
        "@language": "fr",
        "@value": "Soutien et relations avec les communautés"
      }
    ]
 }

[tammy-culture]

Hi @sahalali, I noticed for single date occurrences in acaligram, the dates appear as a date range rather than a single date. Is there a way to fix this?

Here is a link to the event in the staging calendar: https://staging.cms.footlight.io/dashboard/657199fea5ac3d006314ea92/events/add-event/65bb635465190e00a04af136

[sahalali]

I will look into this @tammy-culture

[tammy-culture]

@sahalali When I added the Taxonomy for event types in production, I took the time to put the event types in alphabetical order. I see now that it is no longer the case in both production and staging. Is there a way of making sure the order is respected? Or can we provide alphabetical sorting? In the interim, I am going to resort the event types in alphabetical order again.

Thanks.

[sahalali]

I cleaned up taxonomy last day and removed a few concepts that I added and no longer required. I think the reordering might have happened then.

The order should persist always until the user makes any change.

[sahalali]

As per the discussion during today's standup

1. Import events to the Culture Mauricie Calendar in the Production.
2. The unpublished events from Caligram should not be imported to CMS.

[tammy-culture]

@sahalali I noticed that the CMS production is missing quite a few Event Types from acaligram, is this normal? Thanks.