pulibrary / pulmap

Princeton University Library Geoblacklight application.
Apache License 2.0
8 stars 2 forks source link

Remove pulmap:geoblacklight_harvester:index[BIG10] cron job until our schema is updated #1009

Closed eliotjordan closed 2 years ago

eliotjordan commented 2 years ago

[Pulmap/production] RSolr::Error::Http: RSolr::Error::Http - 400 Bad RequestError: { "responseHeader":{ "status":400, "QTime":22}, "error":{ "metadata":[ "error-class","org.apache.solr.common.SolrException", "root-error-class","org.apache.solr.common.SolrException"], "msg":"Document is missing mandatory uniqueKey field: layer_slug_s", "code":400}}

Backtrace

line 57 of [PROJECT_ROOT]/lib/tasks/pulmap.rake: block (3 levels) in <top (required)>

View full backtrace and more info at honeybadger.io

Request Data: "[{\"id\":\"VAC3073-M-00794\",\"model_name_ssi\":\"Document\",\"model_pk_ssi\":\"8c0b9dd5-1dc0-45fb-8d22-4f767cf9c46c\",\"gbl_mdVersion_s\":\"Aardvark\",\"dct_title_s\":\"1929 road map Indiana\",\"dct_description_sm\":[\"Issued by the State Automobile Insurance Association. Includes place name index on recto. Advertisements and 'Section of transcontinental highway map' showing the United States south of a line from Bangor, Maine to Wahpeton, North Dakota and east of a line from Limon, Colorado to Sanderson, Texas on verso. Scale 1:860,000. Engravers and publishers, the George F. Cram Company.\"],\"dct_language_sm\":[\"eng\"],\"dct_publisher_sm\":[\"Cram\"],\"gbl_resourceClass_sm\":[\"Maps\"],\"dcat_keyword_sm\":[\"Indiana Historic Maps\"],\"dct_issued_s\":\"1929\",\"dct_temporal_sm\":[\"1929\"],\"gbl_dateRange_drsim\":[\"[1929 TO 1929]\"],\"gbl_indexYear_im\":[1929],\"dct_spatial_sm\":[\"Indiana\",\"Indiana\"],\"locn_geometry\":\"ENVELOPE(-88.11,-84.75,41.75,37.86)\",\"dcat_centroid_ss\":\"39.805,-86.43\",\"pcdm_memberOf_sm\":[\"01d-01\"],\"gbl_resourceType_sm\":[\"Road maps\"],\"dct_format_s\":\"JPEG\",\"dct_references_s\":\"{\\"http://schema.org/downloadUrl\\\":\\\"https://purl.dlib.indiana.edu/iudl/images/VAC3073/full/VAC3073-M-00794\\\",\\\"http://schema.org/url\\\":\\\"https://purl.dlib.indiana.edu/iudl/images/VAC3073/VAC3073-M-00794\\\"}\",\"b1g_image_ss\":\"https://purl.dlib.indiana.edu/iudl/images/VAC3073/full/VAC3073-M-00794\",\"b1g_access_s\":\"{}\",\"dct_identifier_sm\":[\"VAC3073-M-00794\"],\"geomg_id_s\":\"VAC3073-M-00794\",\"schema_provider_s\":\"Indiana University\",\"b1g_dateAccessioned_sm\":[\"2019-01-16\"],\"b1g_publication_state_s\":\"published\",\"dct_accessRights_s\":\"Public\",\"gbl_suppressed_b\":false,\"gbl_mdModified_dt\":\"2021-05-07T21:52:25Z\",\"date_created_dtsi\":\"2021-04-27T16:30:42Z\",\"date_modified_dtsi\":\"2021-05-07T21:52:25Z\",\"b1g_geom_import_id_ssi\":\"251\"},{\"id\":\"0dd569b5-cb64-4a9e-8ae1-e95a3382ec42\",\"model_name_ssi\":\"Document\",\"model_pk_ssi\":\"a2d9984a-76c0-421a-9de8-e534a474fdf0\",\"gbl_mdVersion_s\":\"Aardvark\",\"dct_title_s\":\"1936-37 map and business directory of Angola\",\"dct_description_sm\":[\"Includes street index and business directory of Angola, Ind. around the maps. Scale approximately 1:8,800.\"],\"dct_language_sm\":[\"eng\"],\"dct_publisher_sm\":[\"C.W. Chadwick\"],\"gbl_resourceClass_sm\":[\"Maps\"],\"dcat_keyword_sm\":[\"Central business districts\",\"Indiana Historic Maps\"],\"dct_temporal_sm\":[\"1936\"],\"gbl_dateRange_drsim\":[\"[1936 TO 1936]\"],\"gbl_indexYear_im\":[1936],\"dct_spatial_sm\":[\"Indiana\",\"Angola, Indiana\"],\"locn_geometry\":\"ENVELOPE(-85.03,-84.96,41.6601,41.61)\",\"dcat_centroid_ss\":\"41.63505,-84.995\",\"pcdm_memberOf_sm\":[\"01d-01\"],\"gbl_resourceType_sm\":[\"Thematic maps\"],\"dct_format_s\":\"JPEG\",\"dct_references_s\":\"{\\"http://schema.org/downloadUrl\\\":\\\"https://purl.dlib.indiana.edu/iudl/images/VAC3073/full/VAC3073-M-00593\\\",\\\"http://schema.org/url\\\":\\\"http://purl.dlib.indiana.edu/iudl/images/VAC3073/VAC3073-M-00593\\\"}\",\"b1g_image_ss\":\"https://purl.dlib.indiana.edu/iudl/images/VAC3073/thumbnail/VAC3073-M-00593\",\"b1g_access_s\":\"{}\",\"dct_identifier_sm\":[\"0dd569b5-cb64-4a9e-8ae1-e95a3382ec42\"],\"geomg_id_s\":\"0dd569b5-cb64-4a9e-8ae1-e95a3382ec42\",\"schema_provider_s\":\"Indiana University\",\"b1g_dateAccessioned_sm\":[\"2019-01-16\"],\"b1g_publication_state_s\":\"published\",\"dct_accessRights_s\":\"Public\",\"gbl_suppressed_b\":false,\"gbl_mdModified_dt\":\"2021-05-07T21:52:29Z\",\"date_created_dtsi\":\"2021-04-27T16:31:09Z\",\"date_modified_dtsi\":\"2021-05-07T21:52:29Z\",\"b1g_geom_import_id_ssi\":\"251\"},{\"id\":\"dc6cafc7-403f-4358-92ab-b509739aa9b6\",\"model_name_ssi\":\"Document\",\"model_pk_ssi\":\"d6aeac99-203a-4d75-a39f-85a9e684c274\",\"gbl_mdVersion_s\":\"Aardvark\",\"dct_title_s\":\"1937 map of La Porte County, Indiana\",\"dct_description_sm\":[\"Complements of John A. Kelley, Democratic candidate for Surveyor. Scale approximately 1:120,000.\"],\"dct_language_sm\":[\"eng\"],\"dct_creator_sm\":[\"Kelley, John A., C.E., cartographer.\"],\"dct_publisher_sm\":[\"John A. Kelley?\"],\"gbl_resourceClass_sm\":[\"Maps\"],\"dcat_keyword_sm\":[\"Indiana Historic Maps\"],\"dct_temporal_sm\":[\"1936\"],\"gbl_dateRange_drsim\":[\"[1936 TO 1936]\"],\"gbl_indexYear_im\":[1936],\"dct_spatial_sm\":[\"Indiana\",\"LaPorte County, Indiana\"],\"locn_geometry\":\"ENVELOPE(-86.93,-86.48,41.7501,41.23)\",\"dcat_centroid_ss\":\"41.49005,-86.70500000000001\",\"pcdm_memberOf_sm\":[\"01d-01\"],\"dct_format_s\":\"JPEG\",\"dct_references_s\":\"{\\"http://schema.org/downloadUrl\\\":\\\"http://fedora.dlib.indiana.edu:8080/fedora/get/iudl:1074580/LARGE\\\",\\\"http://schema.org/url\\\":\\\"http://purl.dlib.indiana.edu/iudl/images/VAC3073/VAC3073-M-00566\\\"}\",\"b1g_image_ss\":\"http://fedora.dlib.indiana.edu:8080/fedora/get/iudl:1074580/THUMBNAIL\",\"b1g_access_s\":\"{}\",\"dct_identifier_sm\":[\"dc6cafc7-403f-4358-92ab-b509739aa9b6\"],\"geomg_id_s\":\"dc6cafc7-403f-4358-92ab-b509739aa9b6\",\"schema_provider_s\":\"Indiana University\",\"b1g_dateAccessioned_sm\":[\"2019-01-16\"],\"b1g_publication_state_s\":\"published\",\"dct_accessRights_s\":\"Public\",\"gbl_suppressed_b\":false,\"gbl_mdModified_dt\":\"2021-05-07T21:52:47Z\",\"date_created_dtsi\":\"2021-04-27T16:30:57Z\",\"date_modified_dtsi\":\"2021-05-07T21:52:47Z\",\"b1g_geom_import_id_ssi\":\"251\"},{\"id\":\"VAC3073-M-00754\",\"model_name_ssi\":\"Document\",\"model_pk_ssi\":\"1e612fed-1401-4047-88ba-a535c47e1667\",\"gbl_mdVersion_s\":\"Aardvark\",\"dct_title_s\":\"1969 traffic map, state of Indiana: 24 hour annual average total traffic\",\"dct_description_sm\":[\"Insets: Indianapolis -- [New Albany area]. Scale approximately 1:650,000. Prepared by the Division of Planning, Indiana State Highway Commission in cooperation with the U.S. Department of Transportation, Federal Highway Administration.\"],\"dct_language_sm\":[\"eng\"],\"dct_publisher_sm\":[\"State Highway Commission of Indiana\"],\"gbl_resourceClass_sm\":[\"Maps\"],\"dcat_keyword_sm\":[\"Indiana Historic Maps\"],\"dct_issued_s\":\"1969\",\"dct_temporal_sm\":[\"1969\"],\"gbl_dateRange_drsim\":[\"[1969 TO 1969]\"],\"gbl_indexYear_im\":[1969],\"dct_spatial_sm\":[\"Indiana\"],\"locn_geometry\":\"ENVELOPE(-88.08,-84.78,41.75,37.76)\",\"dcat_centroid_ss\":\"39.754999999999995,-86.43\",\"pcdm_memberOf_sm\":[\"01d-01\"],\"gbl_resourceType_sm\":[\"Thematic maps\"],\"dct_format_s\":\"JPEG\",\"dct_references_s\":\"{\\"http://schema.org/downloadUrl\\\":\\\"https://purl.dlib.indiana.edu/iudl/images/VAC3073/full/VAC3073-M-00754\\\",\\\"http://schema.org/url\\\":\\\"https://purl.dlib.indiana.edu/iudl/images/VAC3073/VAC3073-M-00754\\\"}\",\"b1g_image_ss\":\"https://purl.dlib.indiana.edu/iudl/images/VAC3073/full/VAC3073-M-00754\",\"b1g_access_s\":\"{}\",\"dct_identifier_sm\":[\"VAC3073-M-00754\"],\"geomg_id_s\":\"VAC3073-M-00754\",\"schema_provider_s\":\"Indiana University\",\"b1g_dateAccessioned_sm\":[\"2019-01-16\"],\"b1g_publication_state_s\":\"published\",\"dct_accessRights_s\":\"Public\",\"gbl_suppressed_b\":false,\"gbl_mdModified_dt\":\"2021-05-07T21:52:29Z\",\"date_created_dtsi\":\"2021-04-27T16:30:49Z\",\"date_modified_dtsi\":\"2021-05-07T21:52:29Z\",\"b1g_geom_import_id_ssi\":\"251\"},{\"id\":\"2a872c04-f83b-47ac-a3ac-298968ad6c4a\",\"model_name_ssi\":\"Document\",\"model_pk_ssi\":\"027ba809-9a66-4a9a-b519-027053303c21\",\"gbl_mdVersion_s\":\"Aardvark\",\"dct_title_s\":\"[1970 census enumeration district and census tract map of Monroe County, Indiana].\",\"dct_description_sm\":[\"Title supplied by cataloger.; Blue line print.; Base map: General Highway and transportation map, Monroe County, Indiana / prepared by the Indiana State Highway Commission, Division of Planning, 1963.; Shows census enumeration district and census tract boundaries and numbers for the 1970 census. Scale approximately 1:126,720.\"],\"dct_language_sm\":[\"eng\"],\"dct_publisher_sm\":[\"United States Bureau of the Census, Geography Division\"],\"gbl_resourceClass_sm\":[\"Maps\"],\"dcat_keyword_sm\":[\"Census data\",\"Census\",\"Indiana Historic Maps\"],\"dct_temporal_sm\":[\"1970?\"],\"gbl_dateRange_drsim\":[\"[1970 TO 1970]\"],\"gbl_indexYear_im\":[1970],\"dct_spatial_sm\":[\"Indiana\",\"Monroe County, Indiana\"],\"locn_geometry\":\"ENVELOPE(-86.68,-86.31,39.35,39)\",\"dcat_centroid_ss\":\"39.175,-86.495\",\"pcdm_memberOf_sm\":[\"01d-01\"],\"dct_format_s\":\"JPEG\",\"dct_references_s\":\"{\\"http://schema.org/downloadUrl\\\":\\\"https://purl.dlib.indiana.edu/iudl/images/VAC3073/full/VAC3073-M-00254\\\",\\\"http://schema.org/url\\\":\\\"http://purl.dlib.indiana.edu/iudl/images/VAC3073/VAC3073-M-00254\\\"}\",\"b1g_image_ss\":\"https://purl.dlib.indiana.edu/iudl/images/VAC3073/thumbnail/VAC3073-M-00254\",\"b1g_access_s\":\"{}\",\"dct_identifier_sm\":[\"2a872c04-f83b-47ac-a3ac-298968ad6c4a\"],\"geomg_id_s\":\"2a872c04-f83b-47ac-a3ac-298968ad6c4a\",\"schema_provider_s\":\"Indiana University\",\"b1g_dateAccessioned_sm\":[\"2019-01-16\"],\"b1g_publication_state_s\":\"published\",\"dct_accessRights_s\":\"Public\",\"gbl_suppressed_b\":false,\"gbl_mdModified_dt\":\"2021-05-07T21:52:47Z\",\"date_created_dtsi\":\"2021-04-27T16:30:58Z\",\"date_modified_dtsi\":\"2021-05-07T21:52:47Z\",\"b1g_geom_import_id_ssi\":\"251\"},{\"id\":\"aaa94ac9-5ad4-4bc2-951d-5f2061170838\",\"model_name_ssi\":\"Document\",\"model_pk_ssi\":\"c09f1e19-2da5-476a-b67e-49aa2f5f5705\",\"gbl_mdVersion_s\":\"Aardvark\",\"dct_title_s\":\"1990 to 2000 Census Tract Correspondence and Combined Tracts: Twin Cities Metropolitan Area, Minnesota, 1990-2000\",\"dct_alternative_sm\":[\"1990 to 2000 Census Tract Correspondence and Combined Tracts\"],\"dct_description_sm\":[\"This dataset consists of a correspondence table and shape file of the combined 1990 and 2000 Census tracts.Note: Census tracts from 1990 and 2000 do not match in all areas. This dataset is an attempt to match them. Some combining and splitting of tracts was necessary. See the Lineage in Section 2 of the metadata for more information.\"],\"dct_language_sm\":[\"eng\"],\"dct_creator_sm\":[\"Metropolitan Council\"],\"gbl_resourceClass_sm\":[\"Datasets\",\"Web services\"],\"dct_subject_sm\":[\"Census\"],\"dcat_keyword_sm\":[\"census\",\"census tracts\",\"demographics\",\"tract correspondence\"],\"dcat_theme_sm\":[\"Society\"],\"dct_issued_s\":\"2015-09-03\",\"dct_temporal_sm\":[\"1990-2000\"],\"gbl_dateRange_drsim\":[\"[1990 TO 2000]\"],\"gbl_indexYear_im\":[1990],\"dct_spatial_sm\":[\"Anoka County, Minnesota\",\"Carver County, Minnesota\",\"Dakota County, Minnesota\",\"Hennepin County, Minnesota\",\"Ramsey County, Minnesota\",\"Scott County, Minnesota\",\"Washington County, Minnesota\",\"Twin Cities Metropolitan Area, Minnesota\",\"Minnesota\"],\"b1g_geonames_sm\":[\"http://sws.geonames.org/5037779\"],\"locn_geometry\":\"ENVELOPE(-94.012,-92.732,45.415,44.471)\",\"dcat_centroid_ss\":\"44.943,-93.372\",\"pcdm_memberOf_sm\":[\"05a-01\"],\"gbl_resourceType_sm\":[\"Vector data\"],\"dct_format_s\":\"Shapefile\",\"gbl_georeferenced_b\":false,\"dct_references_s\":\"{\\"http://schema.org/downloadUrl\\\":\\\"https://resources.gisdata.mn.gov/pub/gdrs/data/pub/us_mn_state_metc/society_census_trt_comb1990_2000/shp_society_census_trt_comb1990_2000.zip\\\",\\\"http://www.w3.org/1999/xhtml\\\":\\\"https://resources.gisdata.mn.gov/pub/gdrs/data/pub/us_mn_state_metc/society_census_trt_comb1990_2000/metadata/metadata.html\\\",\\\"http://schema.org/url\\\":\\\"https://gisdata.mn.gov/dataset/us-mn-state-metc-society-census-trt-comb1990-2000\\\",\\\"urn:x-esri:serviceType:ArcGIS#DynamicMapLayer\\\":\\\"https://gis2.metc.state.mn.us/arcgis/rest/services/MetroGIS/Demographics/MapServer\\\"}\",\"b1g_image_ss\":\"https://resources.gisdata.mn.gov/pub/gdrs/data/pub/us_mn_state_metc/society_census_trt_comb1990_2000/metadata/preview.jpg\",\"b1g_access_s\":\"{}\",\"dct_identifier_sm\":[\"https://gisdata.mn.gov/dataset/us-mn-state-metc-society-census-trt-comb1990-2000\"],\"geomg_id_s\":\"aaa94ac9-5ad4-4bc2-951d-5f2061170838\",\"schema_provider_s\":\"Minnesota Geospatial Commons\",\"b1g_dct_accrualMethod_s\":\"CKAN\",\"b1g_dateAccessioned_sm\":[\"2020-08-31\"],\"b1g_publication_state_s\":\"published\",\"dct_rights_sm\":[\"This dataset is public domain under the Minnesota Government Data Practices Act (Minnesota Statutes Chapter 13).\"],\"dct_accessRights_s\":\"Public\",\"gbl_suppressed_b\":false,\"gbl_mdModified_dt\":\"2021-09-15T19:17:56Z\",\"date_created_dtsi\":\"2021-04-26T17:03:49Z\",\"date_modified_dtsi\":\"2021-09-15T19:17:56Z\",\"b1g_geom_import_id_ssi\":\"349\"},{\"id\":\"CB2D6C12-BFFF-4DC8-853A-EB512E93DC47\",\"model_name_ssi\":\"Document\",\"model_pk_ssi\":\"9557a181-cbb9-4a20-bb67-9e878d9ca7c6\",\"gbl_mdVersion_s\":\"Aardvark\",\"dct_title_s\":\"1990-2000 Election Data (with 2020 Wards) Wisconsin, 2020\",\"dct_alternative_sm\":[\"1990-2000 Election Data (with 2020 Wards) Wisconsin, 2020\"],\"dct_language_sm\":[\"eng\"],\"dct_creator_sm\":[\"Wisconsin Legislative Technology Services Bureau\"],\"gbl_resourceClass_sm\":[\"Datasets\"],\"dcat_keyword_sm\":[\"Statewide\",\"Wisconsin State Agencies\"],\"dcat_theme_sm\":[\"Boundaries\"],\"dct_temporal_sm\":[\"2020\"],\"gbl_dateRange_drsim\":[\"[2020 TO 2020]\"],\"gbl_indexYear_im\":[2020],\"dct_spatial_sm\":[\"Wisconsin\"],\"locn_geometry\":\"ENVELOPE(-92.889352,-86.763994,47.080728,42.491931)\",\"dcat_centroid_ss\":\"44.7863295,-89.826673\",\"pcdm_memberOf_sm\":[\"10d-03\",\"D-academic-institutions-geospatial-data\"],\"gbl_resourceType_sm\":[\"Polygon data\"],\"dct_format_s\":\"Shapefile\",\"dct_references_s\":\"{\\"http://schema.org/downloadUrl\\\":\\\"https://gisdata.wisc.edu/public/WI_19902000_Election_Data_Wards_2020.zip\\\",\\\"http://schema.org/url\\\":\\\"https://data-ltsb.opendata.arcgis.com/\\\",\\\"http://www.isotc211.org/schemas/2005/gmd/\\\":\\\"https://gisdata.wisc.edu/public/metadata/WI_19902000_Election_Data_Wards_2020.xml\\\"}\",\"b1g_access_s\":\"{}\",\"dct_identifier_sm\":[\"https://geodata.wisc.edu/catalog/CB2D6C12-BFFF-4DC8-853A-EB512E93DC47\"],\"geomg_id_s\":\"CB2D6C12-BFFF-4DC8-853A-EB512E93DC47\",\"schema_provider_s\":\"UW-Madison Robinson Map Library\",\"b1g_dct_accrualMethod_s\":\"GeoBlacklight\",\"b1g_dateAccessioned_sm\":[\"2021-08-18\"],\"b1g_publication_state_s\":\"published\",\"dct_accessRights_s\":\"Public\",\"gbl_mdModified_dt\":\"2021-11-30T22:28:07Z\",\"date_created_dtsi\":\"2021-08-18T18:00:34Z\",\"date_modified_dtsi\":\"2021-11-30T22:28:07Z\",\"b1g_geom_import_id_ssi\":\"436\",\"dct_description_sm\":[\"This data represents election results for 1990-2000 in Wisconsin, mapped with 2020 ward boundaries. [Ward Data Overview: July 2020 municipal wards were collected by LTSB through the WISE-Decade system. Current statutes require each county clerk, or board of election commissioners, no later than January 15 and July 15 of each year, to transmit to the LTSB, in an electronic format (approved by LTSB), a report confirming the boundaries of each municipality, ward and supervisory district within the county as of the preceding “snapshot” date of January 1 or July 1 respectively. Population totals for 2011 wards are carried over to the 2018 dataset for existing wards. New wards created since 2011 due to annexations, detachments, and incorporation are allocated population from Census 2010 collection blocks. LTSB has topologically integrated the data, but there may still be errors.Election Data Overview:The 1990-2000 Wisconsin election data that is included in this file was collected by LTSB from the Wisconsin Elections Commission (WEC) after each general election. A disaggregation process was performed on this election data based on the municipal ward layer that was avai...

eliotjordan commented 2 years ago

Big10 changed their solr schema. We should stop this rake task until we update our schema in the future or figure out a crosswalk.