Kiro47 / MTU-Transfer-Course-Gatherer

I got tired of looking for classes to transfer in by hand, so here we are.
Mozilla Public License 2.0
4 stars 2 forks source link

Refactor/banweb scrape #26

Closed Kiro47 closed 4 years ago

Kiro47 commented 4 years ago

First attempts at refactoring the Banweb scraper into a more modular fashion as we'll quite possibly looking into gathering other data as well soon.

Improvements:

Kiro47 commented 4 years ago

Example of new logging with the format [TIME][LOG_LEVEL][CLASS:FUNCTION] MSG

[2020-04-30T03:06:22][INFO][Data_Gathering:info] Acquiring state map [2020-04-30T03:06:23][INFO][Data_Gathering:info] Mapping colleges to states [2020-04-30T03:06:41][INFO][Data_Gathering:info] Getting course data from colleges

Note the FUNCTION parameter does seem to have some issues due to the wrapper, might have to figure out some magic to get this to work as expected. If you use that formatting with the normal logger directly you'd get the following:

[2020-04-30T03:06:22][INFO][Data_Gathering:get_course_object_list] Acquiring state map [2020-04-30T03:06:23][INFO][Data_Gathering:get_course_object_list] Mapping colleges to states [2020-04-30T03:06:41][INFO][Data_Gathering:get_course_object_list] Getting course data from colleges

codetheweb commented 4 years ago

Would it be worth it to add something like halo or yaspin for some nice feedback while scrapping?

Kiro47 commented 4 years ago

Would it be worth it to add something like halo or yaspin for some nice feedback while scrapping?

It will be something neat to add in to the default logging profile. However, I'm going to wait on it for a bit because how the scraping works is will be changing a lot after this PR gets merged.

Kiro47 commented 4 years ago

Out of scope of PR: we could use asyncio/tornado to send multiple requests at once and significantly speed it up.

That's going to be dealt with in #22