Create a new DOC harvester

avdata99 commented 5 years ago

Part of EPIC at #91. Migrate the geodatagov CKAN extension.

Create a new DOC harvester based on the previous harvester: DocHarvester

Tasks

[ ] Analyse existing harvester code.
[ ] Write notes describin the process
[ ] Write some tests
[ ] Port code to new setup
[ ] Find "real-life" files we can use for testing
[ ] Test in live environment (?) -- this might come later

adborden commented 5 years ago

Can you clarify what the DOC harvester is? The link points to the arcgis harvester.

avdata99 commented 5 years ago

I fixed the link. Is about the DocHarvester.

avdata99 commented 5 years ago

This is the list of harvest sources with type: single-doc:

Aid Dashboard: https://www.dropbox.com/s/4hri6qr3k6ehttb/CCAPS-Aid-Dashboard.xml
Building the Climate Security Vulnerability Model: https://utexas.box.com/s/gs3xaf8fuha1haa1xvsk
NGDA NAIP HVS: https://gis.apfo.usda.gov/geoharvest/NAIP17/Current_Year_NAIP_Imagery_NGDA.xml
NAIP NGDA 2017 Planned Marketplace HVS: https://gis.apfo.usda.gov/geoharvest/NAIP17/Planned_NAIP_Marketplace_NGDA_2017.xml
identifier test for csdgm: http://rpm.tigbox.com/del/coops1/coops/CO-OPS_Water_Level_Remote_Sensing_Profile.xml
Total Rainfall USAF 557th WW: http://165.221.100.74/arcgis1/rest/services/G_Precipitation/AFWA_WEEK/ImageServer/info/metadata
GDELT Dataset: http://gdeltproject.org/data-gov.metadata.v2.xml
AidData Malawi Geocoded and Climate Aid Dataset: https://www.dropbox.com/s/xshm3ksa2jxpkz5/CCAPS---AidData-Malawi-Geocoded-and-Climate-Aid-Dataset%20%281%29.xml
2017Cart_aiannhkml: https://meta.geo.census.gov/data/existing/decennial/GEO/CPMB/boundary/2017Cartographic/aiannhkml/
Park facilities: http://carto.gis.gatech.edu/Metadata/ParkFacilities.xml
Coa Parks: http://geospatial.gatech.edu/Metadata/ParkFacilities.xml
Test Harvest Source: https://raw.githubusercontent.com/kvuppala/data.gov/a84a4e0272d2c4b99d6b3d2fb3724e3baebfc99a/roots-nextdatagov/assets/js/VRSYCW.B2017180.095514.txt
Aid Dashboard User Guide: https://utexas.box.com/s/xupzwzy30umy2lgi3pp5
NFHL Harvest Source: https://content.femadata.com/FIMA/Risk_MAP/NFHL/metadata/NFHL_metadata.xml
ISO 19115 2003: http://arcticlcc.org/assets/products/ALCC2012-10/spatial/NoAK_LandscapePermafrost_19115-2003.xml
tstusaf: http://fasgis.maps.arcgis.com/sharing/rest/content/items/9644aa43e73f4e18b6efd0a0d44d2d78/info/metadata/metadata.xml?token=pWJQbGx_U-_cu8irH5wbeDxM30xNO4zHb7tUtpcvqrgEJcBJ6uhndFhnCUm5cjQPVlbGvXe1rCDoN3l36i7rBkzZ-WJJyNR5o4Kg-jw72K9T8ip96LhpSim7kmkGsKuV6r5B5SaIqM5VCjPuLyYJsuO6dSA96_a3HeS73S8AvHkSaqdSSasM_eZDeDzx5oYt

GSA / datagov-ckan-multi

Create a new DOC harvester #106

Tasks