EricWay1024
commented
2 weeks ago I considered that option. The situation is like this: the URL for a module (taking AMCS2033 American Radicalism as an example) is like this
https://campus.nottingham.ac.uk/psc/csprd_pub/EMPLOYEE/HRMS/c/UN_PROG_AND_MOD_EXTRACT.UN_PLN_EXTRT_FL_CP.GBL?PAGE=UN_CRS_EXT4_FPG&CAMPUS=U&TYPE=Module&YEAR=2024&TITLE=American%20Radicalism&MODULE=AMCS2033&CRSEID=016489&LINKA=&LINKB=&LINKC=UDD-ACSwhere there is this parameter CRSEID=016489 that we need to obtain. Unfortunately this does not seem trivial. This CRSEID is not fetched when the list of modules (like here) is loaded; rather, it is fetched after you click on that module in the list, and then a POST request is sent with the following data (converted from CURL command thanks to https://curlconverter.com/):
import requests
cookies = {} # Omitted
headers = {} # Omitted
data = {
'ICAJAX': '1',
'ICNAVTYPEDROPDOWN': '0',
'ICType': 'Panel',
'ICElementNum': '0',
'ICStateNum': '18',
'ICAction': 'ADDRESS_LINK$0',
'ICModelCancel': '0',
'ICXPos': '0',
'ICYPos': '0',
'ResponsetoDiffFrame': '-1',
'TargetFrameName': 'None',
'FacetPath': 'None',
'ICFocus': '',
'ICSaveWarningFilter': '0',
'ICChanged': '0',
'ICSkipPending': '0',
'ICAutoSave': '0',
'ICResubmit': '0',
'ICSID': 'LSvvCKSCxe6C7js/87aD51fizvCE3qaSdjvS3zDAogo=',
'ICActionPrompt': 'false',
'ICTypeAheadID': '',
'ICBcDomData': 'C~UnknownValue~EMPLOYEE~HRMS~UN_PROG_AND_MOD_EXTRACT.UN_PLN_EXTRT_FL_CP.GBL~UN_CRS_EXT2_FPG~Curriculum Catalogue~UnknownValue~UnknownValue~https://campus.nottingham.ac.uk/psc/csprd_pub/EMPLOYEE/HRMS/c/UN_PROG_AND_MOD_EXTRACT.UN_PLN_EXTRT_FL_CP.GBL?CAMPUS=U&TYPE=Module&YEAR=2024&TITLE=&Module=&SCHOOL=UDD-ACS&LINKA=~UnknownValue',
'ICDNDSrc': '',
'ICPanelHelpUrl': '',
'ICPanelName': '',
'ICPanelControlStyle': 'pst_side2-hidden pst_panel-mode ',
'ICFind': '',
'ICAddCount': '',
'ICAppClsData': '',
'win0hdrdivPT_SYSACT_RETLST': 'psc_hidden',
'win0hdrdivPT_SYSACT_HELP': 'psc_hidden',
}
response = requests.post(
'https://campus.nottingham.ac.uk/psc/csprd_pub/EMPLOYEE/HRMS/c/UN_PROG_AND_MOD_EXTRACT.UN_PLN_EXTRT_FL_CP.GBL',
cookies=cookies,
headers=headers,
data=data,
)where data is pretty complicated. Now it is actually possible to obtain the CRSEID of the modules belonging to the same school by modifying the data fields ICStateNum and ICAction: for the former you need to increment it by one every time, and for the latter the number after ADDRESS_LINK$ corresponds to the index of the module of interest in the module list. So far so good. However, it is not clear yet to me how to obtain the CRSEID for modules of other schools. Naturally, you'd think that you should change ICBcDomData, which seems to record all the pages you have visited (by observing a few such POST requests). Further, you'd think you should change the SCHOOL= field there so as to trick the server into thinking that your last visited page is the module list of the school of interest. And being very careful, you'd also modify the SCHOOL parameters of the Referer field in the headers. But after all these, the request wouldn't give you the right data; the data for the old school is returned, if anything. I doubt that the server remembers the pages you visited through Cookies or other intricate means that I have no intention to figure out, and thus it appears to me that selenium is the quickest way to get things done.
lucienshawls
I wonder if
module/fetch_modules.pycould userequestsrather thanselenium, because the latter significantly reduces efficiency, and requires complicated configurations.