This probably happens because the API fails to return collections or something. The best thing to do would be to add a runtime check on the length of the slice, and log if there is some problem with the API response if that's not already being captured.
Sample log message:
cchc-crawler | time="2021-08-30T08:05:38Z" level=info msg="Starting a crawl of all collections"
cchc-crawler | time="2021-08-30T08:05:38Z" level=debug msg="Fetching all digital collections" url="https://www.loc.gov/collections/?at%21=aka%2Cbreadcrumbs%2Cbrowse%2Ccategories%2Ccontent%2Ccontent_is_post%2Cexpert_resources%2Cfacet_trail%2Cfacet_views%2Cfacets%2Cfeatured_items%2Cform_facets%2Clegacy-url%2Cnext%2Cnext_sibling%2Coptions%2Coriginal_formats%2Cpages%2Cpartof%2Cprevious%2Cprevious_sibling%2Cresearch-centers%2Cshards%2Csite_type%2Csubjects%2Ctimeline_1852_1880%2Ctimeline_1881_1900%2Ctimeline_1901_1925%2Ctimestamp%2Ctopics%2Cviews&c=1000&fa=subject_topic%3Aamerican+history&fo=json"
cchc-crawler | panic: runtime error: index out of range [0] with length 0
cchc-crawler |
cchc-crawler | goroutine 18 [running]:
cchc-crawler | main.Collection.Save(0x0, 0x0, 0x0, 0x17fe1, 0x0, 0x0, 0x0, 0x1, 0x2719c40, 0xed8bd183b, ...)
cchc-crawler | /app/collections.go:80 +0x559
cchc-crawler | main.StartFetchingCollections(0xc0001164e0)
cchc-crawler | /app/entry-all-collections.go:28 +0x98
cchc-crawler | created by main.main
cchc-crawler | /app/main.go:56 +0x125
cchc-crawler | time="2021-08-30T08:06:39Z" level=info msg="Starting the LOC.gov API crawler"
This probably happens because the API fails to return collections or something. The best thing to do would be to add a runtime check on the length of the slice, and log if there is some problem with the API response if that's not already being captured.
Sample log message: