issues
search
sandboxnu
/
major-scraper
Scraping Northeastern's Academic Catalog for use in GraduateNU.
GNU General Public License v3.0
3
stars
0
forks
source link
Address nested array for AND in OR course
#12
Closed
rael346
closed
8 months ago
rael346
commented
8 months ago
This ticket addressed #10
What was done
Changing the test runner from
jest
to
vitest
for better compatibility with ESM
Generate the grammar in
ts
instead of
js
to allow the test to run parse stage separately
Remove the Classify stage tests (since the scraper's url fetch is unreliable)
Add tests for tokenize stage in the form of snapshots
The tests use locally stored html since the fetch step is unreliable (as mentioned above with the Classify stage test)
Add tests for parse stage
10 can be boiled down to the "AND course followed by an OR course" test case
Fix #10 by adding a check whether the
reqs
is an array or not
This is mainly because
nearley
will always wrap the parsed results inside an array
when using a post-processor
Future works
Add more tests for XOMs and any other cases that weren't covered here
Review the grammar and the post-processor to make sure the fix is robust enough/there is a better way to do it through the grammar
Address the failing test for unbounded range courses (similar nested array problem)
This ticket addressed #10
What was done
jest
tovitest
for better compatibility with ESMts
instead ofjs
to allow the test to run parse stage separately10 can be boiled down to the "AND course followed by an OR course" test case
reqs
is an array or notnearley
will always wrap the parsed results inside an array when using a post-processorFuture works