Bad content files

[mbertrand]

Steps to Reproduce

These files all contain some course page HTML instead of the data expected for their extension (related to mitodl/ocw-studio#1255, they originally were assigned the wrong path during import).

https://ocw.mit.edu/courses/1-061-transport-processes-in-the-environment-fall-2008/4d8b195a7270bb063d6ad4d0cc397dd4_WALK2.AVI https://ocw.mit.edu/courses/1-061-transport-processes-in-the-environment-fall-2008/9adb12cdd6735b981f540f516e4cad31_walk2.avi https://ocw.mit.edu/courses/1-103-civil-engineering-materials-laboratory-spring-2004/3dfaac447053cb1e6b7e21c76f492f0d_groupA2.zip https://ocw.mit.edu/courses/1-103-civil-engineering-materials-laboratory-spring-2004/8f214b5602418ff3667312d86ec243f7_groupB2.zip https://ocw.mit.edu/courses/1-103-civil-engineering-materials-laboratory-spring-2004/9178192f249159b4d00fc909275ea234_GroupB1.zip https://ocw.mit.edu/courses/1-103-civil-engineering-materials-laboratory-spring-2004/f27f27ef3fce63a63d8d447ed4eec2e9_groupA1.zip https://ocw.mit.edu/courses/11-943j-urban-transportation-land-use-and-the-environment-spring-2002/710e8227f836dae4a2fa46295cfb6477_onursal_and_gautum.pdf https://ocw.mit.edu/courses/11-s942-wanderings-in-psychogeography-exploring-landscapes-of-history-biography-memory-culture-nature-poetry-surreality-fantasy-and-madness-fall-2020/595e7bd1d41261c3fb6378715656f450_MIT11_s942f20_shao_slides.pptx https://ocw.mit.edu/courses/12-453-crosby-lectures-in-geology-history-of-africa-fall-2005/a887c2c3484ab6c16b37d5b006c6248b_1_four_aces.zip https://ocw.mit.edu/courses/15-763j-manufacturing-system-and-supply-chain-design-spring-2005/91cde358e30e89eb3b5896f8acc9434c_TNG_SCG_BASE.xls https://ocw.mit.edu/courses/15-763j-manufacturing-system-and-supply-chain-design-spring-2005/930f81b6e63e42a933850176e972925e_TNG_data.xls https://ocw.mit.edu/courses/15-763j-manufacturing-system-and-supply-chain-design-spring-2005/ecdf001ac8c09d52830f132ac77f690a_tng_scg_base.xls https://ocw.mit.edu/courses/15-763j-manufacturing-system-and-supply-chain-design-spring-2005/ee3da0b89bb95daded3e4c7beed42177_tng_data.xls https://ocw.mit.edu/courses/16-01-unified-engineering-i-ii-iii-iv-fall-2005-spring-2006/1c94f906aad786bc224e02c9554767e1_q3.pdf https://ocw.mit.edu/courses/16-885j-aircraft-systems-engineering-fall-2004/3c8d442ed82b164aa983396fb6c954dd_citationx_case.pdf https://ocw.mit.edu/courses/16-885j-aircraft-systems-engineering-fall-2004/b6c4ee0bdf2ed17372175ccac27de02a_final_report_v6.pdf https://ocw.mit.edu/courses/16-891j-space-policy-seminar-spring-2003/69188114f06202115a9765f8807bd177_aerocommissionFinalReport.pdf https://ocw.mit.edu/courses/17-951-special-graduate-topic-in-political-science-political-behavior-fall-2005/174d020a2bad77b0c96d33f8a23ddd03_4_party_ID.pdf https://ocw.mit.edu/courses/17-951-special-graduate-topic-in-political-science-political-behavior-fall-2005/70beee822b7ca01d929440770dd09afe_4_party_id.pdf https://ocw.mit.edu/courses/18-01-single-variable-calculus-fall-2005/67082304b3b10f4726fa4955853ac3b0_ocw01f05final.pdf https://ocw.mit.edu/courses/18-01-single-variable-calculus-fall-2005/d0287a0ba830d5489158eaa67ae22e26_OCW01f05final.pdf https://ocw.mit.edu/courses/18-996-random-matrix-theory-and-its-applications-spring-2004/02f4dca660917f68799c3438b7ec97b2_diabl.pdf https://ocw.mit.edu/courses/18-996-random-matrix-theory-and-its-applications-spring-2004/68e62894d5f99a72473b2027f220949e_DIABL.pdf https://ocw.mit.edu/courses/2-611-marine-power-and-propulsion-fall-2006/0d73eeef34debc57d43b2cd339ff0a96_16pa6b_engine.pdf https://ocw.mit.edu/courses/2-611-marine-power-and-propulsion-fall-2006/ef9780ba832dd6c46e5a8d2e58fb30ab_16pa6B_engine.pdf https://ocw.mit.edu/courses/21m-873-theater-arts-topics-fall-2004-january-iap-2005/af92afc52570f43bb01cbfdf161ef3fa_flrpnOS1.pdf https://ocw.mit.edu/courses/4-580-inquiry-into-computation-and-design-fall-2006/2b1ebaf77bc93612d9275a2ef0cf2985_l7B_reasoningvis.pdf https://ocw.mit.edu/courses/4-580-inquiry-into-computation-and-design-fall-2006/c8a00739fdd4c7f3a9ecf188b03b5c2f_l7b_reasoningvis.pdf https://ocw.mit.edu/courses/4-580-inquiry-into-computation-and-design-fall-2006/e575cb62a33a33531d5497c6ed998bdb_l7Acal_shapes.pdf https://ocw.mit.edu/courses/4-580-inquiry-into-computation-and-design-fall-2006/e7b775ec7c7966bd13aa2e05d6173bd1_l7acal_shapes.pdf https://ocw.mit.edu/courses/6-047-computational-biology-fall-2015/f07cce82d55025d75ebccd81b502f37c_ps4docs.zip https://ocw.mit.edu/courses/6-542j-laboratory-on-the-physiology-acoustics-and-perception-of-speech-fall-2005/07b9ceebc4e5279e17b9733003dffd0a_lab09_B.tar https://ocw.mit.edu/courses/8-01x-physics-i-classical-mechanics-with-an-experimental-focus-fall-2002/e02b7805c1696c7f281bf1dba5f0358c_problem_LVPS.pdf https://ocw.mit.edu/courses/8-022-physics-ii-electricity-and-magnetism-fall-2004/617f537e5307bb5f6876a5cf133c39fa_Q3F2003.pdf https://ocw.mit.edu/courses/8-022-physics-ii-electricity-and-magnetism-fall-2004/9b6a40bfbb20a95826bec1e2921ef8b7_q3f2003.pdf https://ocw.mit.edu/courses/9-07-statistics-for-brain-and-cognitive-science-fall-2016/e4e347a8343d000398f851c2a53ec73d_9.07_F2016.zip https://ocw.mit.edu/courses/hst-583-functional-magnetic-resonance-imaging-data-acquisition-and-analysis-fall-2008/4ba1c9f51f0524b10ef46a0cd39a1371_Tutorial1.zip https://ocw.mit.edu/courses/hst-583-functional-magnetic-resonance-imaging-data-acquisition-and-analysis-fall-2008/8040ec8241970bee245564727d199cdb_Lab1_fMRIAcquisition.zip https://ocw.mit.edu/courses/hst-583-functional-magnetic-resonance-imaging-data-acquisition-and-analysis-fall-2008/aeba84daa68485f6084dfce810d19968_subject7session78913.zip https://ocw.mit.edu/courses/res-18-001-calculus-online-textbook-spring-2005/e7f7f60d3efd6635e436c49469cec0e4_MITRES_18_001_Calculus.pdf https://ocw.mit.edu/courses/sts-471j-engineering-apollo-the-moon-project-as-a-complex-system-spring-2007/3354a1dbeecaa22cf9c0e7e1cbfe4727_4_9_mit_role.pdf

[mbertrand]

Turns out that none of these currently seem to be linked to from any site pages (other than the resource's own page). Maybe they should be deleted?

However while looking into these, I accidentally came across a couple others with the same problem that are linked to from https://ocw.mit.edu/courses/1-103-civil-engineering-materials-laboratory-spring-2004/pages/labs/:

https://ocw.mit.edu/courses/1-103-civil-engineering-materials-laboratory-spring-2004/fa0d164f2ae68b698289cb6158654d81_GroupA2.zip (link: "Subgroup A2 Data" for Lab 1)

https://ocw.mit.edu/courses/1-103-civil-engineering-materials-laboratory-spring-2004/924ee79bd35f612f6fcbb09255d93654_GroupA1.zip (link: "Subgroup A1 Data" for Lab 7)

So the issue might be more widespread.

[mbertrand]

List of files on production S3 that are < 100K, last modified after 3/29/2022, and contain HTML:

courses/1-061-transport-processes-in-the-environment-fall-2008/4d8b195a7270bb063d6ad4d0cc397dd4_WALK2.AVI
courses/1-061-transport-processes-in-the-environment-fall-2008/9adb12cdd6735b981f540f516e4cad31_walk2.avi
courses/1-103-civil-engineering-materials-laboratory-spring-2004/3dfaac447053cb1e6b7e21c76f492f0d_groupA2.zip
courses/1-103-civil-engineering-materials-laboratory-spring-2004/8f214b5602418ff3667312d86ec243f7_groupB2.zip
courses/1-103-civil-engineering-materials-laboratory-spring-2004/9178192f249159b4d00fc909275ea234_GroupB1.zip
courses/1-103-civil-engineering-materials-laboratory-spring-2004/924ee79bd35f612f6fcbb09255d93654_GroupA1.zip
courses/1-103-civil-engineering-materials-laboratory-spring-2004/f27f27ef3fce63a63d8d447ed4eec2e9_groupA1.zip
courses/1-103-civil-engineering-materials-laboratory-spring-2004/fa0d164f2ae68b698289cb6158654d81_GroupA2.zip
courses/1-978-from-nano-to-macro-introduction-to-atomistic-modeling-techniques-january-iap-2007/3539024b09107b5093ba423ff45100f5_xrunshear.xyz
courses/1-978-from-nano-to-macro-introduction-to-atomistic-modeling-techniques-january-iap-2007/37e81cc2d4da7ca21086dbb875a0da01_all.xyz
courses/1-978-from-nano-to-macro-introduction-to-atomistic-modeling-techniques-january-iap-2007/a3975ff03fdbb16e5bc9d0d1f4526203_out_small_bending_load.dcd
courses/11-943j-urban-transportation-land-use-and-the-environment-spring-2002/710e8227f836dae4a2fa46295cfb6477_onursal_and_gautum.pdf
courses/11-s942-wanderings-in-psychogeography-exploring-landscapes-of-history-biography-memory-culture-nature-poetry-surreality-fantasy-and-madness-fall-2020/595e7bd1d41261c3fb6378715656f450_MIT11_s942f20_shao_slides.pptx
courses/12-103-science-and-policy-of-natural-hazards-spring-2010/04f6fc874d39346b2fc17af9e26b3835_animation_2.mov
courses/12-453-crosby-lectures-in-geology-history-of-africa-fall-2005/a887c2c3484ab6c16b37d5b006c6248b_1_four_aces.zip
courses/12-808-introduction-to-observational-physical-oceanography-fall-2004/106fb6e488b75d0b45921eb6d1fb8fa6_ghc_Oxkg.mat
courses/12-808-introduction-to-observational-physical-oceanography-fall-2004/34f19a555ff342880c79dd415469e4f6_ghc_pr.mat
courses/12-808-introduction-to-observational-physical-oceanography-fall-2004/6562a9dce59835ad939b7bd1d917ee82_ghc_Sil.mat
courses/12-808-introduction-to-observational-physical-oceanography-fall-2004/df2be8b5af69456df411da8168119d8e_ghc_Ph.mat
courses/12-808-introduction-to-observational-physical-oceanography-fall-2004/fce06f0a9a2e8e038a9b1fafced0ba50_ghc_Ni.mat
courses/14-32-econometrics-spring-2007/5be0b0cf3e50340bc1cea9d9396fe707_oct89.sas7bdat
courses/14-32-econometrics-spring-2007/b043654b03fecb4dfeed99bf9a5572d1_oct84.sas7bdat
courses/15-763j-manufacturing-system-and-supply-chain-design-spring-2005/91cde358e30e89eb3b5896f8acc9434c_TNG_SCG_BASE.xls
courses/15-763j-manufacturing-system-and-supply-chain-design-spring-2005/930f81b6e63e42a933850176e972925e_TNG_data.xls
courses/15-763j-manufacturing-system-and-supply-chain-design-spring-2005/ecdf001ac8c09d52830f132ac77f690a_tng_scg_base.xls
courses/15-763j-manufacturing-system-and-supply-chain-design-spring-2005/ee3da0b89bb95daded3e4c7beed42177_tng_data.xls
courses/16-01-unified-engineering-i-ii-iii-iv-fall-2005-spring-2006/1c94f906aad786bc224e02c9554767e1_q3.pdf
courses/16-01-unified-engineering-i-ii-iii-iv-fall-2005-spring-2006/4cefff5084bcc9d83880bcef5eb14130_gnat_315p_nt.exe
courses/16-885j-aircraft-systems-engineering-fall-2004/3c8d442ed82b164aa983396fb6c954dd_citationx_case.pdf
courses/16-885j-aircraft-systems-engineering-fall-2004/b6c4ee0bdf2ed17372175ccac27de02a_final_report_v6.pdf
courses/16-891j-space-policy-seminar-spring-2003/69188114f06202115a9765f8807bd177_aerocommissionFinalReport.pdf
courses/17-951-special-graduate-topic-in-political-science-political-behavior-fall-2005/174d020a2bad77b0c96d33f8a23ddd03_4_party_ID.pdf
courses/17-951-special-graduate-topic-in-political-science-political-behavior-fall-2005/70beee822b7ca01d929440770dd09afe_4_party_id.pdf
courses/18-01-single-variable-calculus-fall-2005/67082304b3b10f4726fa4955853ac3b0_ocw01f05final.pdf
courses/18-01-single-variable-calculus-fall-2005/d0287a0ba830d5489158eaa67ae22e26_OCW01f05final.pdf
courses/18-024-multivariable-calculus-with-theory-spring-2011/0afae3f5056e3c9fb9b56a9b0925df6f_MIT18_024s11_ChA_Fnotes.pdf
courses/18-996-random-matrix-theory-and-its-applications-spring-2004/02f4dca660917f68799c3438b7ec97b2_diabl.pdf
courses/18-996-random-matrix-theory-and-its-applications-spring-2004/68e62894d5f99a72473b2027f220949e_DIABL.pdf
courses/2-22-design-principles-for-ocean-vehicles-13-42-spring-2005/6a53eb0eaa874c008c7b570460789e84_data.zip
courses/2-611-marine-power-and-propulsion-fall-2006/0d73eeef34debc57d43b2cd339ff0a96_16pa6b_engine.pdf
courses/2-611-marine-power-and-propulsion-fall-2006/aac1729b9b17feeb93b9935e1a801e71_26stirling.avi
courses/2-611-marine-power-and-propulsion-fall-2006/ef9780ba832dd6c46e5a8d2e58fb30ab_16pa6B_engine.pdf
courses/2-72-elements-of-mechanical-design-spring-2009/ac70cf04da63e97f33464758c8029efc_lathe09.zip
courses/21m-380-music-and-technology-sound-design-spring-2016/73bb2f51aa991effbdb315e9d49f8d08_MIT21M_380S16_lec_slides.pdf
courses/21m-873-theater-arts-topics-fall-2004-january-iap-2005/af92afc52570f43bb01cbfdf161ef3fa_flrpnOS1.pdf
courses/3-016-mathematics-for-materials-scientists-and-engineers-fall-2005/638f9ae65a92ce94e59ed54811c2779a_Lecture11.zip
courses/3-016-mathematics-for-materials-scientists-and-engineers-fall-2005/d43d16aa2aa00e5ac170ac148986d1e0_Lecture18.zip
courses/4-580-inquiry-into-computation-and-design-fall-2006/2b1ebaf77bc93612d9275a2ef0cf2985_l7B_reasoningvis.pdf
courses/4-580-inquiry-into-computation-and-design-fall-2006/c8a00739fdd4c7f3a9ecf188b03b5c2f_l7b_reasoningvis.pdf
courses/4-580-inquiry-into-computation-and-design-fall-2006/e575cb62a33a33531d5497c6ed998bdb_l7Acal_shapes.pdf
courses/4-580-inquiry-into-computation-and-design-fall-2006/e7b775ec7c7966bd13aa2e05d6173bd1_l7acal_shapes.pdf
courses/6-047-computational-biology-fall-2015/f07cce82d55025d75ebccd81b502f37c_ps4docs.zip
courses/6-542j-laboratory-on-the-physiology-acoustics-and-perception-of-speech-fall-2005/07b9ceebc4e5279e17b9733003dffd0a_lab09_B.tar
courses/6-837-computer-graphics-fall-2012/874c023973c2b49404559ab6454d3c64_distrib1.tar.gz
courses/8-01x-physics-i-classical-mechanics-with-an-experimental-focus-fall-2002/e02b7805c1696c7f281bf1dba5f0358c_problem_LVPS.pdf
courses/8-022-physics-ii-electricity-and-magnetism-fall-2004/617f537e5307bb5f6876a5cf133c39fa_Q3F2003.pdf
courses/8-022-physics-ii-electricity-and-magnetism-fall-2004/9b6a40bfbb20a95826bec1e2921ef8b7_q3f2003.pdf
courses/9-07-statistics-for-brain-and-cognitive-science-fall-2016/e4e347a8343d000398f851c2a53ec73d_9.07_F2016.zip
courses/9-17-systems-neuroscience-lab-spring-2013/022c94d940937461fbe6dfbf556b459b_Project_3.zip
courses/hst-582j-biomedical-signal-and-image-processing-spring-2007/0afce1c2b26169919d4d6df520a20bef_lab3files.zip
courses/hst-583-functional-magnetic-resonance-imaging-data-acquisition-and-analysis-fall-2008/4ba1c9f51f0524b10ef46a0cd39a1371_Tutorial1.zip
courses/hst-583-functional-magnetic-resonance-imaging-data-acquisition-and-analysis-fall-2008/8040ec8241970bee245564727d199cdb_Lab1_fMRIAcquisition.zip
courses/hst-583-functional-magnetic-resonance-imaging-data-acquisition-and-analysis-fall-2008/a20aec1d9ac495f8120d71d6368aeecd_Tutorial_with_dicom.zip
courses/hst-583-functional-magnetic-resonance-imaging-data-acquisition-and-analysis-fall-2008/aeba84daa68485f6084dfce810d19968_subject7session78913.zip
courses/ids-410j-modeling-and-assessment-for-policy-spring-2013/9ec10ae94d67aab8b8acf1f125d238b2_MITESD_864S13_lecture15.pdf
courses/mas-110-fundamentals-of-computational-media-design-fall-2008/49d393b928aa936eb8f3c00a692bb9bc_csoundxo.tar
courses/sts-003-the-rise-of-modern-science-fall-2010/bc75b43ba105e1e8532dbb450374cd40_MITSTS_003F10_lec13.pdf
courses/sts-471j-engineering-apollo-the-moon-project-as-a-complex-system-spring-2007/3354a1dbeecaa22cf9c0e7e1cbfe4727_4_9_mit_role.pdf
courses/sts-471j-engineering-apollo-the-moon-project-as-a-complex-system-spring-2007/353d2b2b1f4012aa4a7ca8b692cc1020_ap15_descent.mpg
courses/sts-471j-engineering-apollo-the-moon-project-as-a-complex-system-spring-2007/bbcaeb2be7f6cabf385d5b3721659f4c_a14land24fps_DivX.avi

After checking a handful of these manually, some exist on the legacy OCW site but have 0 bytes: courses/1-103-civil-engineering-materials-laboratory-spring-2004/924ee79bd35f612f6fcbb09255d93654_GroupA1.zip courses/1-103-civil-engineering-materials-laboratory-spring-2004/fa0d164f2ae68b698289cb6158654d81_GroupA2.zip

Some don't exist there either: courses/1-103-civil-engineering-materials-laboratory-spring-2004/8f214b5602418ff3667312d86ec243f7_groupB2.zip courses/21m-873-theater-arts-topics-fall-2004-january-iap-2005/af92afc52570f43bb01cbfdf161ef3fa_flrpnOS1.pdf

And some are valid files: courses/3-016-mathematics-for-materials-scientists-and-engineers-fall-2005/638f9ae65a92ce94e59ed54811c2779a_Lecture11.zip courses/3-016-mathematics-for-materials-scientists-and-engineers-fall-2005/d43d16aa2aa00e5ac170ac148986d1e0_Lecture18.zip

[gumaerc]

So I've been looking into this and it's pretty puzzling. I looked into one of the zip files to use as an example, and tried:

• https://ocw.mit.edu/courses/hst-583-functional-magnetic-resonance-imaging-data-acquisition-and-analysis-fall-2008/aeba84daa68485f6084dfce810d19968_subject7session78913.zip - Returns a file with a zip extension that contains HTML generated by Hugo (Labs page)
• https://open-learning-course-data-production.s3.amazonaws.com/hst-583-functional-magnetic-resonance-imaging-data-acquisition-and-analysis-fall-2008/aeba84daa68485f6084dfce810d19968_subject7session78913.zip - Returns the same thing

This is all pretty confusing and I'm still not 100% sure what's going on but I did notice that for this file at least, in open-learning-course-data the extracted asset is missing the UUID prefix. A number of other files in that course are also missing the UUID. If we look at the entry for this file in the parsed JSON, it looks like this:

{
    "order_index": null,
    "uid": "aeba84daa68485f6084dfce810d19968",
    "id": "subject7session78913.zip",
    "parent_uid": "6344a97bcc1105ad4911175f735ecf18",
    "title": "Data for Parts I-III: subject 7",
    "caption": null,
    "file_type": "application/zip",
    "alt_text": null,
    "credit": null,
    "platform_requirements": "File decompression software, such as Winzip  or StuffIt, is required to open the .zip files found on the course site. .dcm is an image format created by the National Electrical Manufacturers Association (NEMA) as a standard for distributing and viewing medical images, such as MRIs, CT scans, and ultrasound images.",
    "description": "Data files for a lab assignment on statistical analysis of fMRI data.",
    "type": "OCWFile",
    "file_location": "https://open-learning-course-data-production.s3.amazonaws.com/hst-583-functional-magnetic-resonance-imaging-data-acquisition-and-analysis-fall-2008/aeba84daa68485f6084dfce810d19968_subject7session78913.zip"
  }

Notice how the id property of the file matches the filename we saw in the S3 bucket that doesn't contain the UUID prefix, but the file_location property does have the prefix. Since file_location is what's used in ocw-to-hugo to write the file property of the resulting resource, that accounts for why it would be pointing to a file that doesn't exist. What's still a mystery to me is how the Hugo generated HTML for the Labs page is what ended up there instead. Very strange!