gumaerc
commented
2 years ago So I looked into this, and this particular course seems to have had a bad import back in Jan. 28, 2021 and for some reason it was not updated by subsequent imports. The short_id property was not an actual short ID, but something we did early on which is simply truncate the slug to 100 characters. I manually corrected this particular course, and it will publish now. I did some further investigation into other courses that may have bad data hanging around. One thing I noticed about this course was that it had no menu or metadata imported, so I ran a query for the total amount of Website objects in production vs the total about WebsiteContent objects that are of the types sitemetadata and navmenu:
In [2]: from websites.models import *
In [3]: course_starter = WebsiteStarter.objects.get(name="ocw-course")
In [4]: Website.objects.filter(starter=course_starter).count()
Out[4]: 2673
In [5]: WebsiteContent.objects.filter(type="sitemetadata").count()
Out[5]: 2651
In [6]: WebsiteContent.objects.filter(type="navmenu").count()
Out[6]: 2649What this shows us is that there are 22 course sites without sitemetadata content, and 24 sites without a navmenu:
Sites without sitemetadata:
In [11]: for site in Website.objects.filter(starter=course_starter):
...: count = WebsiteContent.objects.filter(type="sitemetadata", website=site).count()
...: if count == 0:
...: print(site.name)
...:
8-701-introduction-to-nuclear-and-particle-physics-spring-2004
6-803-the-human-intelligence-enterprise-spring-2006
21h-244-imperial-and-revolutionary-russia-culture-and-politics-1700-1917-fall-2012
6-172-performance-engineering-of-software-systems-fall-2010
21w-780-communicating-in-technical-organizations-fall-2001
6-094-introduction-to-matlab-january-iap-2010
5-73-introductory-quantum-mechanics-i-fall-2002
18-s34-problem-solving-seminar-fall-2007
6-803-the-human-intelligence-enterprise-spring-2002
17-561-european-politics-fall-2011
17-565-israel-history-politics-culture-and-identity-spring-2011
21a-265-food-and-culture-spring-2011
1-258j-public-transportation-systems-spring-2010
18-312-algebraic-combinatorics-spring-2009
14-381-statistical-method-in-economics-fall-2013
18-435j-quantum-computation-fall-2018
21a-01-how-culture-works-fall-2012
22-01-introduction-to-nuclear-engineering-and-ionizing-radiation-fall-2015
24-118-paradox-infinity-spring-2013
7-016-introductory-biology-fall-2014
5-310-laboratory-chemistry-fall-2017
6-436j-fundamentals-of-probability-fall-2008Sites without navmenu:
In [12]: for site in Website.objects.filter(starter=course_starter):
...: count = WebsiteContent.objects.filter(type="navmenu", website=site).count()
...: if count == 0:
...: print(site.name)
...:
8-701-introduction-to-nuclear-and-particle-physics-spring-2004
6-803-the-human-intelligence-enterprise-spring-2006
introduction-to-nuclear-engineering-and-ionizing-radiation
21h-244-imperial-and-revolutionary-russia-culture-and-politics-1700-1917-fall-2012
18s097-introduction-to-metric-spaces-iap-2021
6-172-performance-engineering-of-software-systems-fall-2010
21w-780-communicating-in-technical-organizations-fall-2001
6-094-introduction-to-matlab-january-iap-2010
5-73-introductory-quantum-mechanics-i-fall-2002
18-s34-problem-solving-seminar-fall-2007
6-803-the-human-intelligence-enterprise-spring-2002
17-561-european-politics-fall-2011
17-565-israel-history-politics-culture-and-identity-spring-2011
21a-265-food-and-culture-spring-2011
1-258j-public-transportation-systems-spring-2010
18-312-algebraic-combinatorics-spring-2009
14-381-statistical-method-in-economics-fall-2013
18-435j-quantum-computation-fall-2018
21a-01-how-culture-works-fall-2012
22-01-introduction-to-nuclear-engineering-and-ionizing-radiation-fall-2015
24-118-paradox-infinity-spring-2013
7-016-introductory-biology-fall-2014
5-310-laboratory-chemistry-fall-2017
6-436j-fundamentals-of-probability-fall-2008Printing the short_id property of the courses in the above list reveals:
In [13]: for site in Website.objects.filter(starter=course_starter):
...: count = WebsiteContent.objects.filter(type="navmenu", website=site).count()
...: if count == 0:
...: print(site.short_id)
...:
8-701-introduction-to-nuclear-and-particle-physics-spring-2004
6-803-the-human-intelligence-enterprise-spring-2006
22.01-fall-2021
21h-244-imperial-and-revolutionary-russia-culture-and-politics-1700-1917-fall-2012
18.s097-iap-2021
6-172-performance-engineering-of-software-systems-fall-2010
21w-780-communicating-in-technical-organizations-fall-2001
6-094-introduction-to-matlab-january-iap-2010
5-73-introductory-quantum-mechanics-i-fall-2002
18-s34-problem-solving-seminar-fall-2007
6-803-the-human-intelligence-enterprise-spring-2002
17-561-european-politics-fall-2011
17-565-israel-history-politics-culture-and-identity-spring-2011
21a-265-food-and-culture-spring-2011
1-258j-public-transportation-systems-spring-2010
18-312-algebraic-combinatorics-spring-2009
14-381-statistical-method-in-economics-fall-2013
18-435j-quantum-computation-fall-2018
21a-01-how-culture-works-fall-2012
22-01-introduction-to-nuclear-engineering-and-ionizing-radiation-fall-2015
24-118-paradox-infinity-spring-2013
7-016-introductory-biology-fall-2014
5-310-laboratory-chemistry-fall-2017
6-436j-fundamentals-of-probability-fall-2008
2 of these courses seem to have a correct short_id property, which is the difference between the amount of courses missing sitemetadata and navmenu's. So, it seems like missing sitemetadata is a good way to see courses that have an old, bad import.
pdpinch
MAbdurrehman12
Steps to Reproduce
Expected Behavior
it publishes
Actual Behavior
it fails to publish
Stacktrace
https://cicd.odl.mit.edu/teams/ocw/pipelines/draft/jobs/build-ocw-site/builds/3?vars.site=%2221w-732-5-introduction-to-technical-communication-explorations-in-scientific-and-technical-writing-fall-2006%22