Added scraping function for fetching user details through coding ninjas platform

Yash-Parsana / CrazyCoderServerlessApi

This is backend api of CrazyCoder App which serves realtime data from various platforms through web scraping hosted on Netlify Serverless function. CrazyCoder App: https://github.com/Yash-Parsana/CrazyCoderApp | CrazyCoder website : https://crazycoder-eight.vercel.app

https://play.google.com/store/apps/details?id=com.parsanatech.crazycoder

15 stars 15 forks source link

Added scraping function for fetching user details through coding ninjas platform #35

Closed GbCderr14 closed 4 months ago

GbCderr14 commented 4 months ago

Now _/codingninjas/:username will give the details of the user's profile from coding ninjas platform.. for eg.

fixes #23

GbCderr14 commented 4 months ago

@Yash-Parsana have a look on it please

Yash-Parsana commented 4 months ago

@GbCderr14 great work, it looks good but one think is, it has very high latency around 10X as compare to other apis so try to improve that.

Some tips :

There are so many await statements you can use Promise.all for independent statements.
I think waitUntil: 'networkidle2' might taking more time so try to improve it.
You may follow same module and logic that used to scrap other platform like code chef or at coder.

GbCderr14 commented 4 months ago

@Yash-Parsana i tried with existing modules but those were not able to fetch the data may be due to high security measures applied by coding ninjas using normal fetch function was just fetching the scripts of the page and not the html so using pupeeteer was the best option for this situation.

GbCderr14 commented 4 months ago

please check it now modified it.. @Yash-Parsana