Request Endpoint/Response Body for Search Bar Functionality

[dillondesilva]

• [x] Show the units as user is typing accordingly (as shown in the prototype below)
• [ ] For MVP: Go to the unit of study page when unit is clicked (e.g: clicking COMP2017 will go to https://www.sydney.edu.au/units/COMP2017)

On the route /units, I am unable to utilise the search bar functionality feature. Would love to see this implemented!

I did some analysis (Via Dev Tools -> Network) on https://www.sydney.edu.au/units page to see if I could obtain the REST API endpoint and response body for querying information about a unit.

When I query "COMP" into the search bar, I notice a request is made to the following resource: https://www.sydney.edu.au/s/search.html?query=COMP&collection=Sydney-Curriculum_UOS&profile=_default_preview&form=custom-json&start_rank=1.

The corresponding response body is of the form:

{
"resultsSummary" : {
"fullyMatching" : 108,
"totalMatching" : 108,
"numRanks" : 10,
"currStart" : 1,
"currEnd" : 10
},
"results":[
{
"rank" : 1,
"uosCode" : "COMP3608",
"description" : "An advanced alternative to COMP3308; covers material at an advanced and challenging level.",
"title" : "Introduction to Artificial Intelligence (Adv)",
"uosApprovalFlag" : "YES",
"UoSURL" : "https://sydney.edu.au/students/units/unit.COMP3608.html"
}
,
{
"rank" : 2,
"uosCode" : "COMP2022",
"description" : "This unit provides an introduction to the foundations of computing. The main aims are to introduce and compare different models of computation based on state-machines, grammars and algebra, and logic.",
"title" : "Models of Computation",
"uosApprovalFlag" : "YES",
"UoSURL" : "https://sydney.edu.au/students/units/unit.COMP2022.html"
}
,
{
"rank" : 3,
"uosCode" : "COMP3109",
"description" : "This unit provides an introduction to the foundations of programming languages and their implementation. The main aims are to teach what are: semantics, programming paradigms and implementation of programming languages.",
"title" : "Programming Languages and Paradigms",
"uosApprovalFlag" : "YES",
"UoSURL" : "https://sydney.edu.au/students/units/unit.COMP3109.html"
}
,
{
"rank" : 4,
"uosCode" : "COMP2922",
"description" : "This unit provides an introduction to the foundations of computing. The main aims are to introduce and compare different models of computation based on state-machines, grammars and algebra, and logic.",
"title" : "Models of Computation (Adv)",
"uosApprovalFlag" : "YES",
"UoSURL" : "https://sydney.edu.au/students/units/unit.COMP2922.html"
}
,
{
"rank" : 5,
"uosCode" : "COMP4103",
"description" : "Students enrolled in the Honours programs study various advanced aspects of Computer Science. The program may include lectures, tutorials, seminars and practicals. They will undertake a research project. Assessment will include the project and may include examinations and classwork.",
"title" : "Computer Science Honours Project A",
"uosApprovalFlag" : "YES",
"UoSURL" : "https://sydney.edu.au/students/units/unit.COMP4103.html"
}
,
{
"rank" : 6,
"uosCode" : "COMP4104",
"description" : "Students enrolled in the Honours programs study various advanced aspects of Computer Science. The program may include lectures, tutorials, seminars and practicals. They will undertake a research project. Assessment will include the project and may include examinations and classwork.",
"title" : "Computer Science Honours Project B",
"uosApprovalFlag" : "YES",
"UoSURL" : "https://sydney.edu.au/students/units/unit.COMP4104.html"
}
,
{
"rank" : 7,
"description" : "Students enrolled in the Honours programs study various advanced aspects of Computer Science. The program may include lectures, tutorials, seminars and practicals. They will undertake a research project. Assessment will include the project and may include examinations and classwork.",
"uosCode" : "COMP4105",
"title" : "Computer Science Honours Project C",
"uosApprovalFlag" : "YES",
"UoSURL" : "https://sydney.edu.au/students/units/unit.COMP4105.html"
}
,
{
"rank" : 8,
"uosCode" : "COMP4106",
"description" : "Students enrolled in the Honours programs study various advanced aspects of Computer Science. The program may include lectures, tutorials, seminars and practicals. They will undertake a research project. Assessment will include the project and may include examinations and classwork.",
"title" : "Computer Science Honours Project D",
"uosApprovalFlag" : "YES",
"UoSURL" : "https://sydney.edu.au/students/units/unit.COMP4106.html"
}
,
{
"rank" : 9,
"uosCode" : "COMP4447",
"description" : "This is an advanced course on Pervasive Computing, with a focus on the Internet of Things (IoT). It introduces the key aspects of the IoT and explores these in terms of the new research towards creating user interfaces that disappear into the environment and are available pervasively, for example in homes, workplaces, cars and carried.",
"title" : "Pervasive Computing",
"uosApprovalFlag" : "YES",
"UoSURL" : "https://sydney.edu.au/students/units/unit.COMP4447.html"
}
,
{
"rank" : 10,
"uosCode" : "COMP5047",
"description" : "This is an advanced course on Pervasive Computing, with a focus on the Internet of Things (IoT). It introduces the key aspects of the IoT and explores these in terms of the new research towards creating user interfaces that disappear into the environment and are available pervasively, for example in homes, workplaces, cars and carried.",
"title" : "Pervasive Computing",
"uosApprovalFlag" : "YES",
"UoSURL" : "https://sydney.edu.au/students/units/unit.COMP5047.html"
}
]
}

I've omitted the request/response headers from this post as they are quite lengthy but happy to include them in comments if necessary.

This endpoint seems fairly simple to call and it is publicly accessible. A client-side fetch request should suffice for completing the unit search feature but there may be other approaches as well.

Hopefully this information gives enough of a jump start for working on the feature!

[jennifermtan]

Thanks for this! Maybe @GoodGameRuler can comment on this too?

[GoodGameRuler]

I have a few comments on this @dillondesilva. I am quite impressed with the alternative solution.

Firstly, we have to remember that first and foremost UniTrack will be a degree planning service. That means the requirement of keeping track of exact unit information isn't necessary. This means that we can have outdated information.

Secondly, I would prefer not to do client-side requests as that may cause issues (especially when making mistakes with react hooks). I am sure USYD handles this well, but it also isn't too hard to fetch all this data once and store it. It would also make it easier down the line when we do data scraping.

With the above two points, I think the most optimal solution would be to write a script that runs through this endpoint on COMP, INFO, SOFT, DATA, and ISYS as the input query, and convert them into a series of SQL insert queries, that we can just run once. We are currently working on integrating Vercel Postgres which also introduces various optimisation benefits I believe. What are your thoughts @dillondesilva?

I will point out that I had a look at the network queries for an individual unit page, and as far as I can see there is no endpoint to be exploited there. So to get detailed unit information like pre-reqs, we will have to data scrape. However, this is a very viable solution until we think about scraping further down the line.

[dillondesilva]

Hey @GoodGameRuler

Yeah all of the above sounds like a great idea! I had to go back and quickly refresh my knowledge on fetching requests, CORS, etc and I can see why client-side request isn't a great idea. I agree having our own endpoint to handle this stuff seems like the go - what do you think about using NextJS API routes?

For the long term, using Vercel Postgres sounds like a good idea (especially for something like a pre-req tree). I also noticed that actual UoS pages such as this one are server-side rendered so web scraping is something that needs to happen eventually and it wouldn't hurt to start it now.

EDIT: Just checkout the major-postgresSetup branch which is using API routes :P I'll close this issue for now as it sounds more like the next step is API development related

[GoodGameRuler]

Some updates. Have accomplished a basic sketch of what you suggested.

While this is completely functional at the moment and with minimal work I could display the Unit Titles, and redirect to the unit page, this is a select type input not a text type, which means I'll need to change a bit.

Usage

[jennifermtan]

Some updates. Have accomplished a basic sketch of what you suggested.

While this is completely functional at the moment and with minimal work I could display the Unit Titles, and redirect to the unit page, this is a select type input not a text type, which means I'll need to change a bit.

Usage

That works! Could you reuse the styling we already have and just convert it to a select option?

[GoodGameRuler]

That works! Could you reuse the styling we already have and just convert it to a select option?

Unfortunately we want to be able to search for multiple units at the same time. So pressing enter should display all matching units. The select doesn't allow for that at the moment.

With selects you can't search you have to select.

[GoodGameRuler]

Found alternatives

Alternative 1: mui

Alternative 2: shadcn

[dillondesilva]

Some updates. Have accomplished a basic sketch of what you suggested.

While this is completely functional at the moment and with minimal work I could display the Unit Titles, and redirect to the unit page, this is a select type input not a text type, which means I'll need to change a bit.

Usage

Looks great!

[jennifermtan]

Found alternatives

Alternative 1: mui

Alternative 2: shadcn

I like the second one better but we could probably be consistent with MUI if we need it for other components, plus it's more developed. What do you think? @dillondesilva @GoodGameRuler

[github-actions[bot]]

Stale issue message - 7 days till automatic close