OCW: NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not ex...

[sentry-io[bot]]

Sentry Issue: OPEN-7QV

NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.
  File "course_catalog/etl/ocw_next.py", line 139, in transform_resource
    settings.OCW_NEXT_AWS_STORAGE_BUCKET_NAME, file_s3_path
  File "boto3/resources/factory.py", line 520, in do_action
    response = action(self, *args, **kwargs)
  File "boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "botocore/client.py", line 635, in _make_api_call
    raise error_class(parsed_response, operation_name)

NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.
(1 additional frame(s) were not displayed)
...
  File "course_catalog/etl/ocw_next.py", line 143, in transform_resource
    settings.OCW_NEXT_LIVE_BUCKET, file_s3_path
  File "boto3/resources/factory.py", line 520, in do_action
    response = action(self, *args, **kwargs)
  File "boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "botocore/client.py", line 635, in _make_api_call
    raise error_class(parsed_response, operation_name)

ERROR syncing course file courses/21m-873-theater-arts-topics-fall-2004-january-iap-2005/resources/flrpnos1-1/data.json for course courses/21m-873-theater-arts-topics-fall-2004-january-iap-2005/

[mbertrand]

https://ocw.mit.edu/courses/21m-873-theater-arts-topics-fall-2004-january-iap-2005/resources/flrpnos1-1/ - Somehow this got assigned a file path of /courses/af92afc52570f43bb01cbfdf161ef3fa_flrpnOS1.pdf - It should be courses/21m-873-theater-arts-topics-fall-2004-january-iap-2005/af92afc52570f43bb01cbfdf161ef3fa_flrpnOS1.pdf

Ditto for https://ocw.mit.edu/courses/sts-471j-engineering-apollo-the-moon-project-as-a-complex-system-spring-2007/resources/4_9_mit_role/ :

Current file path: courses/3354a1dbeecaa22cf9c0e7e1cbfe4727_4_9_mit_role.pdf Should be: courses/sts-471j-engineering-apollo-the-moon-project-as-a-complex-system-spring-2007/3354a1dbeecaa22cf9c0e7e1cbfe4727_4_9_mit_role.pdf

It gets weirder: these files actually are where they're supposed to be in the ocw-studio bucket, so it's just the file paths that are wrong:

https://ol-ocw-studio-app-production.s3.amazonaws.com/courses/21m-873-theater-arts-topics-fall-2004-january-iap-2005/af92afc52570f43bb01cbfdf161ef3fa_flrpnOS1.pdf

https://ol-ocw-studio-app-production.s3.amazonaws.com/courses/sts-471j-engineering-apollo-the-moon-project-as-a-complex-system-spring-2007/3354a1dbeecaa22cf9c0e7e1cbfe4727_4_9_mit_role.pdf

[mbertrand]

At some point there must have been a bug in ocw-to-hugo that generated the wrong output for some files (that wrong output still exists in the ocw-to-hugo-output-qa bucket for sts-471j-engineering-apollo-the-moon-project-as-a-complex-system-spring-2007). But when I ran the latest ocw-to-hugo code locally on the course, the output file path was correct.

The easiest way to fix this is probably with a management command or just a shell script to update the affected files, which are:

1-061-transport-processes-in-the-environment-fall-2008:  /courses/9adb12cdd6735b981f540f516e4cad31_walk2.avi
1-061-transport-processes-in-the-environment-fall-2008:  /courses/4d8b195a7270bb063d6ad4d0cc397dd4_WALK2.AVI
1-103-civil-engineering-materials-laboratory-spring-2004:  /courses/9178192f249159b4d00fc909275ea234_GroupB1.zip
1-103-civil-engineering-materials-laboratory-spring-2004:  /courses/f27f27ef3fce63a63d8d447ed4eec2e9_groupA1.zip
1-103-civil-engineering-materials-laboratory-spring-2004:  /courses/8f214b5602418ff3667312d86ec243f7_groupB2.zip
1-103-civil-engineering-materials-laboratory-spring-2004:  /courses/3dfaac447053cb1e6b7e21c76f492f0d_groupA2.zip
11-943j-urban-transportation-land-use-and-the-environment-spring-2002:  /courses/710e8227f836dae4a2fa46295cfb6477_onursal_and_gautum.pdf
11-s942-wanderings-in-psychogeography-exploring-landscapes-of-history-biography-memory-culture-nature-poetry-surreality-fantasy-and-madness-fall-2020:  /courses/595e7bd1d41261c3fb6378715656f450_MIT11_s942f20_shao_slides.pptx
12-453-crosby-lectures-in-geology-history-of-africa-fall-2005:  /courses/a887c2c3484ab6c16b37d5b006c6248b_1_four_aces.zip
15-763j-manufacturing-system-and-supply-chain-design-spring-2005:  /courses/930f81b6e63e42a933850176e972925e_TNG_data.xls
15-763j-manufacturing-system-and-supply-chain-design-spring-2005:  /courses/91cde358e30e89eb3b5896f8acc9434c_TNG_SCG_BASE.xls
15-763j-manufacturing-system-and-supply-chain-design-spring-2005:  /courses/ee3da0b89bb95daded3e4c7beed42177_tng_data.xls
15-763j-manufacturing-system-and-supply-chain-design-spring-2005:  /courses/ecdf001ac8c09d52830f132ac77f690a_tng_scg_base.xls
16-01-unified-engineering-i-ii-iii-iv-fall-2005-spring-2006:  /courses/1c94f906aad786bc224e02c9554767e1_q3.pdf
16-885j-aircraft-systems-engineering-fall-2004:  /courses/3c8d442ed82b164aa983396fb6c954dd_citationx_case.pdf
16-885j-aircraft-systems-engineering-fall-2004:  /courses/b6c4ee0bdf2ed17372175ccac27de02a_final_report_v6.pdf
16-891j-space-policy-seminar-spring-2003:  /courses/69188114f06202115a9765f8807bd177_aerocommissionFinalReport.pdf
17-951-special-graduate-topic-in-political-science-political-behavior-fall-2005:  /courses/174d020a2bad77b0c96d33f8a23ddd03_4_party_ID.pdf
17-951-special-graduate-topic-in-political-science-political-behavior-fall-2005:  /courses/70beee822b7ca01d929440770dd09afe_4_party_id.pdf
18-01-single-variable-calculus-fall-2005:  /courses/d0287a0ba830d5489158eaa67ae22e26_OCW01f05final.pdf
18-01-single-variable-calculus-fall-2005:  /courses/67082304b3b10f4726fa4955853ac3b0_ocw01f05final.pdf
18-996-random-matrix-theory-and-its-applications-spring-2004:  /courses/68e62894d5f99a72473b2027f220949e_DIABL.pdf
18-996-random-matrix-theory-and-its-applications-spring-2004:  /courses/02f4dca660917f68799c3438b7ec97b2_diabl.pdf
21m-873-theater-arts-topics-fall-2004-january-iap-2005:  /courses/af92afc52570f43bb01cbfdf161ef3fa_flrpnOS1.pdf
2-611-marine-power-and-propulsion-fall-2006:  /courses/ef9780ba832dd6c46e5a8d2e58fb30ab_16pa6B_engine.pdf
2-611-marine-power-and-propulsion-fall-2006:  /courses/0d73eeef34debc57d43b2cd339ff0a96_16pa6b_engine.pdf
4-580-inquiry-into-computation-and-design-fall-2006:  /courses/c8a00739fdd4c7f3a9ecf188b03b5c2f_l7b_reasoningvis.pdf
4-580-inquiry-into-computation-and-design-fall-2006:  /courses/e7b775ec7c7966bd13aa2e05d6173bd1_l7acal_shapes.pdf
4-580-inquiry-into-computation-and-design-fall-2006:  /courses/e575cb62a33a33531d5497c6ed998bdb_l7Acal_shapes.pdf
4-580-inquiry-into-computation-and-design-fall-2006:  /courses/2b1ebaf77bc93612d9275a2ef0cf2985_l7B_reasoningvis.pdf
6-047-computational-biology-fall-2015:  /courses/f07cce82d55025d75ebccd81b502f37c_ps4docs.zip
6-542j-laboratory-on-the-physiology-acoustics-and-perception-of-speech-fall-2005:  /courses/07b9ceebc4e5279e17b9733003dffd0a_lab09_B.tar
8-01x-physics-i-classical-mechanics-with-an-experimental-focus-fall-2002:  /courses/e02b7805c1696c7f281bf1dba5f0358c_problem_LVPS.pdf
8-022-physics-ii-electricity-and-magnetism-fall-2004:  /courses/617f537e5307bb5f6876a5cf133c39fa_Q3F2003.pdf
8-022-physics-ii-electricity-and-magnetism-fall-2004:  /courses/9b6a40bfbb20a95826bec1e2921ef8b7_q3f2003.pdf
9-07-statistics-for-brain-and-cognitive-science-fall-2016:  /courses/e4e347a8343d000398f851c2a53ec73d_9.07_F2016.zip
hst-583-functional-magnetic-resonance-imaging-data-acquisition-and-analysis-fall-2008:  /courses/8040ec8241970bee245564727d199cdb_Lab1_fMRIAcquisition.zip
hst-583-functional-magnetic-resonance-imaging-data-acquisition-and-analysis-fall-2008:  /courses/aeba84daa68485f6084dfce810d19968_subject7session78913.zip
hst-583-functional-magnetic-resonance-imaging-data-acquisition-and-analysis-fall-2008:  /courses/4ba1c9f51f0524b10ef46a0cd39a1371_Tutorial1.zip
res-18-001-calculus-online-textbook-spring-2005:  /courses/e7f7f60d3efd6635e436c49469cec0e4_MITRES_18_001_Calculus.pdf
sts-471j-engineering-apollo-the-moon-project-as-a-complex-system-spring-2007:  /courses/3354a1dbeecaa22cf9c0e7e1cbfe4727_4_9_mit_role.pdf

based on the query

WebsiteContent.objects.filter(type="resource").filter(file__regex=r'^/courses\/[A-Za-z0-9\-\.\_]+(\.).*')

[pdpinch]

What end-user impact is there from this issue?

[mbertrand]

Those particular resources won't be in the search index