Import list of data portals from data.gouv.fr

okfn / dataportals.org

Open Data Portals and Sites around the world

http://dataportals.org/

133 stars 80 forks source link

Import list of data portals from data.gouv.fr #280

Open augusto-herrmann opened 2 years ago

augusto-herrmann commented 2 years ago

The French national government open data portal, data.gouv.fr, has a list of harvest data sources in a CSV resource named export-harvest-YYYYMMDD-HHMMSS.csv at this dataset. If we filter by the validation field with the value of "Accepted", we may have a list of open data portals that could be added here.

Thanks @agarrone for the tip on the Team Open Data forum.

agarrone commented 2 years ago

Regarding French portals you might want to take a look at : https://airtable.com/shrWxHPi2XjLu9xtM 🙂

augusto-herrmann commented 2 years ago

Thanks, @agarrone! Does it make sense to import both these datasets, or is one of them (which one?) enough?

augusto-herrmann commented 2 years ago

Note for whoever is going to implement this: since the table has public access, when importing data from Airtable with Python you can use this package in order to avoid have to set up credentials to get a token to use the Airtable API.

agarrone commented 2 years ago

The first one are the harvested portals on the national portal (data.gouv.fr). It represents local and national portals but it is not exhaustive.

The second one is crowd sourced and, I believe, maintained by the non profit https://www.opendatafrance.net/ and https://twitter.com/LoicHay. It represents only locals portals but some are not harvested (yet) by data.gouv.fr

So I'm afraid you'll need both to have a better view on the issue...

Don't hesitate to ask questions on the forum :)

augusto-herrmann commented 2 years ago

Thanks, @agarrone!

Do you have any idea on how we could discover geographical coordinates to display those portals on a map? I shoud probably ask about that on the forum too, once I'm caught up with the thread.

agarrone commented 2 years ago

Hi @augusto-herrmann

I've gathered in a file most local open data portal in France with geographical coordinates to make your life easier.

Some context :

France has a very complex administration system with lots of different authority and lots of open data portals.
There are various sources to identify data portals but not an exhaustive one ( Some example : the catalog of harvested portal on data.gouv.fr, the crowd sourced one on local portals, the inventory maintained by Open Data France, the inventory made by "Open Collectivités".)

I used the latter to produce this file. Note that some organizations have more than one portal. Sometimes because of thematic data (special portal on geo data for example). I missed some geo coordinates because the API I used if not entirely exhaustive yet.

french_open_data_portals (1).csv

Thanks to @geoffreyaldebert and @restuccia for the help ;)

Let me know if you have any questions :)

augusto-herrmann commented 2 years ago

Wow, thanks @agarrone for all the work, explanation and references! :trophy:

I'm sorry but it will take me a while before I'm able to work on this. If anyone else reading this is able and willing to do it sooner, feel free to take it from here.

agarrone commented 2 years ago

No problem thanks for the interest :)