search

elixir-crawly / crawly

Crawly, a high-level web crawling & scraping framework for Elixir.
https://hexdocs.pm/crawly
Apache License 2.0
988 stars 116 forks source link

A new fetcher for Puppeteer based JS rendering #272

Closed oltarasenko closed 7 months ago

oltarasenko commented 1 year ago

Creates a splash replacement. I have tested it with just one target so far, it's hard to say it's perfect, but it might be an alternative to splash that is seems to be a bit dead right now

github-actions[bot] commented 7 months ago

Tests summary

Finished in 10.6 seconds (0.5s async, 10.1s sync) Randomized with seed 216268

:white_check_mark: 0 failures (131 tests) :white_check_mark: 77.38% coverage (75% is the minimum)

Coverage details ``` Percentage | Module -----------|-------------------------- 0.00% | Crawly.Fetchers.Splash 0.00% | Crawly.Loggers.SendToUiBackend 0.00% | Crawly.Response 0.00% | Mix.Tasks.Crawly.Gen.Config 0.00% | String.Chars.Crawly.Request 0.00% | UtilsTestSpider 40.00% | PipelineTestSpider 42.86% | TestSpider 55.56% | Mix.Tasks.Crawly.Gen.Spider 61.49% | Crawly.API.Router 61.54% | Crawly.Models.YMLSpider 66.67% | Crawly.Models.Job 75.00% | Crawly.Pipelines.JSONEncoder 76.47% | Crawly.SimpleStorage 76.92% | Crawly.Fetchers.CrawlyRenderServer 77.78% | Crawly.Pipelines.Experimental.SendToUI 80.00% | Crawly.EngineSup 80.77% | Crawly 82.61% | Crawly.Pipelines.WriteToFile 83.33% | Crawly.Middlewares.UserAgent 85.33% | Crawly.Engine 87.50% | Crawly.Application 87.50% | TestUtils 88.37% | Crawly.RequestsStorage 88.46% | Crawly.RequestsStorage.Worker 89.06% | Crawly.Manager 90.00% | Crawly.Pipelines.Validate 91.84% | Crawly.Worker 92.05% | Crawly.Utils 92.11% | Crawly.DataStorage 92.86% | Crawly.DataStorage.Worker 100.00% | Crawly.Fetchers.Fetcher 100.00% | Crawly.Fetchers.HTTPoisonFetcher 100.00% | Crawly.ManagerSup 100.00% | Crawly.Middlewares.AutoCookiesManager 100.00% | Crawly.Middlewares.DomainFilter 100.00% | Crawly.Middlewares.RequestOptions 100.00% | Crawly.Middlewares.RobotsTxt 100.00% | Crawly.Middlewares.SameDomainFilter 100.00% | Crawly.Middlewares.UniqueRequest 100.00% | Crawly.ParsedItem 100.00% | Crawly.Pipeline 100.00% | Crawly.Pipelines.CSVEncoder 100.00% | Crawly.Pipelines.DuplicatesFilter 100.00% | Crawly.Request 100.00% | Crawly.Settings 100.00% | Crawly.Spider -----------|-------------------------- 77.38% | Total ```