search

alfred82santa / whalesong

Whalesong is an asyncio python library to manage WebApps remotely. Currently WhatsappWeb is implemented
https://whalesong.readthedocs.io
MIT License
50 stars 19 forks source link

Official port of whalesong #97

Open parthibd opened 5 years ago

parthibd commented 5 years ago

It's mentioned in your comments you will officially drop support for FirefoxDriver at some point of time and move to pypeteer (perhaps?)

Instead of using selenium at all why don't we port whalesong totally to puppeteer ? I know it will be quiet and effort . But think about it , the whole async nature of node.js would benefit whalesong extensively . Even though pypetter looks promising but we never know when that project gets abandoned . Any suggestion ?

Note : I have gone through whalesong-js . It's quiet good , I have used it .

alfred82santa commented 5 years ago

Well, I have an issue about to split code in two repositories. One for python code (It means whalesong core) and one for js code (it means js scriptlet). Once we split code, it would be easy to maintain more than one driver. I'm not really interested on js driver, but I understand others could be. In this case, the most important is to share scriptlet code.

About to drop Firefox driver, there are other solutions. Regular browsers consume too much memory and cpu. So, puppeteer is not a solution, either. We only need a webview where inject our scriptlet and a way to send results. Puppeteer or Pyppeteer (even using firefox, https://aslushnikov.github.io/ispuppeteerfirefoxready/) are not a good solution.

parthibd commented 5 years ago

Either way we need browser or webview automation as you are saying. What do you have in mind? Any libraries?

parthibd commented 5 years ago

I understand abandoning FirefoxDriver but I don't believe that there is any browser automation libraries out there as established as Puppeteer or Selenium, who happen to have a lot of support. If Selenium and Puppeteer is out of the question, what else is there?

parthibd commented 5 years ago

I don't think dropping selenium support is such a very good idea because you'll have to depend on puppeteer which I have looked into doesn't have a lot of active develop. Even though browsers can be too heavy for whalesong, I think we have not other option because one way or another we are automating a browser. And depending upon pypeteer will be like putting all the eggs in the same basket.

alfred82santa commented 5 years ago

We don't need any automation library... We just need a webview with an access point. Actually, with a few lines of code we could build a webview with QtWebKit or WebkitGtk (I don't like those solutions, either)... We don't need all methods designed for automation. We just need a way to inject our scriptlet and a way to send commands and to retrieve results. Whalesong is not an automation tool, we use them because it was easier at the beginning.

However, any backend solution will work with drivers build with python or javascript... or go... or rust...

About pyppeteer, you are right, it looks like it is dying... But the code actually is very simple, we could fork it and maintain what we need, just a few methods.

If you want to investigate more: I like WPEWebkit (low resources and headless mode) and Servo (it's new, maybe it's good).

parthibd commented 5 years ago

So you basically want to implement a custom webview to satisfy the requirements of whalesong ? That sounds like a good idea . And your'e right we need to make a separate repo for the scriplet . That way we can have different drivers . But one thing is for sure we can't be using selenium for much long .As I have tested with multiple instances with all the various optimizations possible firefox/chrome is a RAM and CPU hog .

And we can maintain a separate repo for pypeteer by forking it . But it will spread out the development time that could have gone into making whalesong awesome :) I like qtwebkit as I have done some development using it before .

And one reason not to use pypeteer is because it uses a patch version of chrome . In my experience using a patched version instead of a generally available one is not a very great idea . The problem starts to occur when you move from one platform to another linux to windows .

But for the time being sticking with Selenium is the only way until we decide to come up with replacement . So I hope you'll give it a thought before dropping FirefoxDriver altogether .

alfred82santa commented 5 years ago

Be aware, patched version of chrome used by pyppeteer is the same one used by puppeteer, it is maintained by google, I don't think it will be abandoned.

alfred82santa commented 5 years ago

We will not remove Firefox backend until we get a better solution, don't worry.

parthibd commented 5 years ago

I might be wrong but the chrome bundled with puppeteer is not a patched version . Its a version that works in the most stable manner with the API . I guess pypetter does the same .

parthibd commented 5 years ago

I love working with whalesong . And its already in a great shape and form . I wish to ensure continuous development of it . I'll look into webkit and qtwebview and see if I can get a simple webview for whalesong to serve its purposes.

parthibd commented 5 years ago

I ran some trials with webviews . I tried qtwebview , javafx webview , and even python webview without making an api for whalesong . I came to the conclusion that whenever a webview is launched it takes more or less the same amount of ram and cpu as firefox or chromium . Sure there can be optimizations .

The only ones that seem viable are WPEWebkit , servo (didnt try) . Maybe I am wrong but if what we want to achieve is using a simple webview we have to customize the rendering engine for the most optimal webview.