joelgriffith
commented
7 years ago I share you frustration. There's a lot to learn in driving a browser, and then even more to learn in scraping. I thinks there's opportunity for improvement in that regard!
I'd hate for you to do a ton of work as, on Monday, there's going to be an announcement that will likely change which module you work with. When that happens we should talk again and plan some strategies on how to solve this better
shackweed
I've only recently started looking into web scrapers and I've been trying to get some headway with them. Until I stumbled across Navalia I was trying to learn Nightmarejs, but since Navalia can work with multiple tabs I think it's a better scraper for me to cut my teeth on.
What has frustrated me with Nightmare and Navalia is a shortage of simple code examples (looping through pagination links, scraping thumbnail URLs, downloading an image, etc). What tutorials/forum discussions I've found assume a lot of previous knowledge or they're too simple and they just open a page and close it. The programming structure is also very different from what I (and I'd say most people) are used to (asynchronous). So in addition to pouring through the API you also have to get used to the way the programming flow works. This makes for a steep learning curve.
But this all brings me to something that I'd like to contribute to the project: code examples. I'm willing to tinker and toil for weeks if I have to in order to get something to work, so I'm not asking for hand-holding. Once I know how to do a few concrete things I can start to get a sense of how things work and start making my own improvements on them.
I'd like to create code snippets for doing basic tasks on specific websites. casperjs comes with a lot of code examples like this, which helped me a lot. If anyone here is willing to answer basic questions now & then I'm willing to do the work of creating code examples from the perspective of a noob in order to help others get started with Navalia.