Automatic content creations through webscrapping/connectors from external sources

[daemuu]

Proposed by @Tonfa during our last Work Residency (i let him develop further if needed)

[Tonfa]

Je pense que c'est un dev au niveau SemApps @srosset81 ?

Expected behaviour:

• a “website” field is offered at the top of each entry form (event, place, organization) to crawl the site (or just that page? fo ran event it usually has all info) and retrieve the relevant information to complete the form.

A small description is above this field. After clicking on the "Go" button, a progress indicator (percentage) or execution indicator (clock running, process running) is displayed to assure the user that something is happening

In case of timeout an error message is displayed "Please try again, we had difficulties. You can also change the website link".

FR: un champ "site web" est offert en haut de chaque formulaire d'entrée (event, place, organization) pour crawler le site et récupérer les informations pertinentes pour remplir le formulaire.

Une petite description est au dessus de ce champ. Après un clic sur le bouton "Go" s'affiche un indicateur d'avancement (pourcentage) ou d'éxecution (horloge qui tourne, processus qui tourne) pour garantir à l'utilisateur qu'il se passe quelque chose

En cas de time out un message d'erreur s'affiche "Veuillez réessayer, nous avons eu des difficultés. Vous pouvez aussi changer le lien du site web".

[srosset81]

As discussed with @GuillaumeAV by phone, this is certainly a lot of work and it will give very little results because websites are too different to have relevant informations. At the best we could get the site title, the site icon, maybe a small description, and even these informations could be useless.

[srosset81]

It could be more interesting to connect with other relevant sources, like Transiscope, and import the few data they may have (if CDLT actors are on Transiscope ?). The functionning could be similar to what we do with Wikipedia (you type a few words, you select the Wikipedia entry and it prefills the title/name). We would only need to add a fetchGogocarto util here: https://github.com/assemblee-virtuelle/semapps/tree/master/src/frontend/packages/interop-components/src/fetch

[GuillaumeAV]

An alternative could be a nicely designed chatbot (or a mailer) that asks regularly some questions in order to fulfill proegessively the profile ?

[srosset81]

It could be more interesting to connect with other relevant sources, like Transiscope, and import the few data they may have (if CDLT actors are on Transiscope ?). The functionning could be similar to what we do with Wikipedia (you type a few words, you select the Wikipedia entry and it prefills the title/name). We would only need to add a fetchGogocarto util here: https://github.com/assemblee-virtuelle/semapps/tree/master/src/frontend/packages/interop-components/src/fetch

I've created an issue to remember of this https://github.com/assemblee-virtuelle/semapps/issues/1190