search

okfn / ckanext-lacounts

CKAN extension for the LA Counts project
GNU Affero General Public License v3.0
8 stars 5 forks source link

ESRI harvester(s) #9

Closed amercader closed 6 years ago

amercader commented 6 years ago

"ESRI" is not really a source, it could be ArcGIS Server REST, ESRI geoportal, ... Clarify which ones we need.

IIRC Data.gov has some ESRI based harvesters that we might be able to use (source)

Map the fields to our schema (#6)

amercader commented 6 years ago

Example sources compiled by LA Counts:

type Actual type URL Title
ESRI ESRI Geoportal Server http://geohub.lacity.org City of Los Angeles
ESRI ArcGIS REST Services https://services5.arcgis.com/7nsPwEMP38bSkCjy/ArcGIS/rest/services City of Los Angeles
ESRI ? http://navigatela.lacity.org/NavigateLA/ City of Los Angeles
ESRI ? http://bss.lacity.org/NeighborhoodCouncils/Street_Assessment_Map/map.html City of Los Angeles
ESRI ArcGIS REST Services https://services1.arcgis.com/tp9wqSVX1AitKgjd/ArcGIS/rest/services City of Los Angeles
ESRI ArcGIS REST Services https://services2.arcgis.com/zNjnZafDYCAJAbN0/ArcGIS/rest/services City of Pasadena
ESRI ? http://pasgis.maps.arcgis.com/apps/MapAndAppGallery/index.html?appid=2053f2c744304bf8a787eeee8f768b05&group=6db0dd784c344d05a28200515ad7b4ba City of Pasadena
ESRI ArcGIS REST Services http://public.gis.lacounty.gov/public/rest/services County of Los Angeles
ESRI ? http://www.californiaschoolcampusdatabase.org GreenInfo Network
ESRI ArcGIS REST Services https://services5.arcgis.com/YzWImMY4GtDcMxOx/ArcGIS/rest/services Southern California Association of Governments (SCAG)
ESRI ESRI Geoportal Server http://gisdata-scag.opendata.arcgis.com/ Southern California Association of Governments (SCAG)
ESRI ESRI Geoportal Server https://hifld-geoplatform.opendata.arcgis.com/ U.S. Department of Homeland Security

Edit: added the actual types. More info on them:

amercader commented 6 years ago

The ESRI Geoportal Server instances expose a /data.json endpoint with the metadata serialized as JSON-LD. This can be harvested with the RDF DCAT harvester on ckanext-dcat. This harvester does already a good job of translating the JSON-LD into a CKAN dataset dict but has also plenty of entry points where we can hook into our mapping / common logic.

For the ArcGIS REST API endpoints, we will reuse the one from data.gov, but this one needs to be reviewed as it might have custom modifications.

brew commented 6 years ago

Closed prematurely. Still need to implement the ArcGIS harvester.

amercader commented 6 years ago

Closing for now. If the ArcGIS REST harvester is needed we'll open a separate issue