Bot is currently not responsive

[bchooxg]

Hi just wondering if the bot is still functional ?

Just went to try this out and the bot is not returning any responses

[xlanor]

71 .

Development has stopped as I've more pressing projects at work.

[Poonjh95]

hi @xlanor, wondering if you could teach me how to get the bot up and running again, it was a really useful tool and i am hoping i could learn something from working on it. 1.5 year more to grad haha

thanks in advance!

[xlanor]

Hi,

To run the bot you need some general experience in running bare metal on linux systems. For your learning process I suggest that you upgrade it instead. It will teach you more about modern practices than just running some legacy code that was written years ago when I was starting out as a developer and still fiddling around

• containerise it by using docker.
• run a cache somewhere in Netherlands while the scraping component remains in sg because telegrams bot APIs only point of access/exit is in Netherlands. This is a major source of latency that is very obvious when you scroll through the timetable. Think about how to handle cache updating, cache misses, etc. how do you know that the data is inaccurate if you are running a scraper and have no control over the actual data source?Can you introduce an additional condition such as a cache TTL instead, and only repopulate the cache after the TTL has expired? How will this negatively impact the user base and is it an acceptable outcome?

There's some options here, you can either use redis, which is very fast, and deal with caching policies and synchronisation, or you can run a replicated database with a read replica sitting in NL

• be very careful about overloading the SIM website. You do not own the website, nor do you pay costs for bandwidth incurred, and you do not want to be hauled in front of a disciplinary board for inadvertently ddosing a website. For this reason some form of initial caching is done by scraping the results to a local database on request, but there could also be some people who would try to overload the website through your bot. How do you handle that? I did it by limiting the number of concurrently executing scrapes per user, but can you improve on that by using an additional caching mechanism such that a scrape won't even trigger? It can be as simple as adding a maximum number of scrapes per day, but we can improve beyond that. How do we ensure that it doesn't even hit our local database, but if you've tried to access the same records in say, the last five minutes, we avoid the most computationally expensive operation (on our system) of going to the db?
• don't use mongoDB, because the only purpose of using mongo was to play around with non relational databases. The data here can be redesigned to better fit a relational database and benefit from it
• upgrade the python version to better support newer things like type hinting
• potentially remove the queue component by dynamically spinning up threads or using a thread pool instead of adding an external queue mechanism which may be confusing if you've never worked with it before (although it's relatively simple and definitely does the job of handling high load due to the SIM websites slow response. Take note that celery can probably do this better than if you were to write it yourself, but the upsides would be that you get to familiarise yourself with MT and that you cut down on the overall number of seperate components that need to be deployed.)
• write Tests.

Tests at the unit level should mainly revolve around the parser code. I come from the school of thought that mocking databases in unit tests is a waste of time due to the frequently changing queries

Tests at the integration level should ideally run a request end to end, but mock the SIM website by spinning up a local server serving HTML code. I'm not sure about the entry point from python telegram bot library, but I think monkeypatching should be available at least to mock some data to suit certain scenarios in the request to save you some trouble. What you can do is to read the upstream API code to see what kind of data is expected to be received at the socket and construct your own requests from there to be used during testing ( to simulate a client calling the telegram bot and telegram relaying the message to you through their bot api server)

At this point you should also spin up a database and constantly populate and repopulate with test datasets to test that your queries work fine and as expected.

I wrote this during my first year at SIM and have not touched it much since then so I do not think that the website has changed much if at all. The scraping code should still be fine, but if not it's simply a case of whipping out chromes developer tools and looking for identifiers in the HTML tree

Beyond that, I cant assist further. I don't have the time for it, and I tried handing the project over to a few people but most of them lost interest pretty quickly.

Based on my time in UOW, almost all of the above, except for containerisation, would have been taught at some point or other in the curriculum. Even though the above suggestions are extremely over engineered for the use case, It would be a good fun project to take on during the holidays and use the opportunity to build on the mainly theoretical knowledge of the above in the curriculum, as well as an opportunity to venture outside of a guided sandbox in the forms of exams and assignments.

If you have any questions about the suggestions for improvements feel free to ask