Harvest Datasets in Production

thejuliekramer commented 4 years ago

User story

As the data.gov team, I want to see all of the datasets from the imported harvest sources on the production site

Implementation details

Acceptance criteria

[ ] When I go to the production site
[ ] And I go to https://catalog-next.data.gov/datasets
[ ] Then I see the harvest sources listed in the source type tab in this document: https://docs.google.com/spreadsheets/d/1LgVvgiUC5Vc-OQaSmtvQNlTFLhs3Mho65ekggYEiYh4/edit#gid=264788117
[ ] And the number of datasets imported per harvest source matches the spreadsheet.

thejuliekramer commented 3 years ago

@hkdctol These are looking good - one that I see that is broken from the client side is New Mexico Resource Geographic Information System (NM RGIS) - is it possible to reach out to them?

thejuliekramer commented 3 years ago

Seeing the same error in catalog as in catalog-next for source national-elevation-dataset-ned-1-arc-second-collection https://catalog.data.gov/api/3/action/harvest_source_show?name_or_id=national-elevation-dataset-ned-1-arc-second-collection

Error: "Could not harvest WAF link http://localhost: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1067816350>: Failed to establish a new connection: [Errno 111] Connection refused',))",

hkdctol commented 3 years ago

@thejuliekramer I've reached out to a contact for New Mexico source -- I think we don't have current harvest source. On the National elevation dataset - is that a USGS harvest source?

hkdctol commented 3 years ago

New Mexico contact is unavailable but we should hear back by Wednesday.

thejuliekramer commented 3 years ago

@hkdctol New Mexico has been updated and harvesting properly - I updated the spreadsheeting. Thanks!

thejuliekramer commented 3 years ago

@hkdctol other sources we could reach out to are

Federal Laboratory Consortium Data.json (federal-laboratory-consortium-data-json) Error loading JSON content - no objects to fetch
NSGIC GIS State HS (nsgic-gis-state-hs) Unable to get content from URL
Census TIGER 2012 Other Identifiers (census-tiger-2012-other-identifiers) 400 Error from source

hkdctol commented 3 years ago

Checking with agency POCs on those.

thejuliekramer commented 3 years ago

@hkdctol I ran all of the manual sources and updated with error messages on the spreadsheet - I think this is ready to review

thejuliekramer commented 3 years ago

Other big collections

Collection	Children WEB	Children DB	Harvest Source
Lidar Point Cloud - USGS National Map 3DEP Downloadable Data Collection	1,139,371	1,126,190	USGS Lidar Point Cloud LAS Harvest Source
USGS public distribution of FSA 10:1 NAIP Imagery Downloadable Data Collection from The National Map	218,702	368,310	FSA 10:1 NAIP Imagery Collection
PKG ID 949b7c71-6913-45f1-85b9-2a4a3dd6e2bc (maybe deleted or not published)	?	352,778
USGS Historical Topographic Map Collection	84	341,816	USGS Historical Topographic Maps

thejuliekramer commented 3 years ago

@hkdctol Adding these large sources here to also check on

USGS Lidar Point Cloud LAS Harvest Source

FSA 10:1 NAIP Imagery Collection

USGS Historical Topographic Maps

thejuliekramer commented 3 years ago

The last two harvest sources I am working on are here

USGS Lidar Point Cloud LAS Harvest Source https://admin-catalog-next.data.gov/harvest/admin/usgs-lidar-point-cloud-las-harvest-source

and

Original Product Resolution (OPR) Source Digital Elevation Models (DEMs) - USGS National Map https://admin-catalog-next.data.gov/harvest/admin/usgs-ned-original-product-resolution-opr-downloadable-data-collection

GSA / datagov-ckan-multi