elixir-crawly / crawly

Crawly, a high-level web crawling & scraping framework for Elixir.
https://hexdocs.pm/crawly
Apache License 2.0
988 stars 116 forks source link

Fix Crawly interactive shell start for Elixir 1.17 #299

Closed oltarasenko closed 4 months ago

oltarasenko commented 4 months ago

Crawly's Engine could not start with a new version of Elixir, due to the fact that Crawly.Utils.list_spiders() was taking too long (and probably could not be loaded) I have moved this operation to a handle_continue, so the App starts normally

github-actions[bot] commented 4 months ago

Tests summary

Finished in 11.3 seconds (0.7s async, 10.6s sync) Randomized with seed 51908

:white_check_mark: 0 failures (132 tests) :white_check_mark: 77.06% coverage (75% is the minimum)

Coverage details ``` Percentage | Module -----------|-------------------------- 0.00% | Crawly.Fetchers.Splash 0.00% | Crawly.Loggers.SendToUiBackend 0.00% | Crawly.Response 0.00% | Mix.Tasks.Crawly.Gen.Config 0.00% | String.Chars.Crawly.Request 0.00% | UtilsTestSpider 40.00% | PipelineTestSpider 42.86% | TestSpider 55.56% | Mix.Tasks.Crawly.Gen.Spider 60.67% | Crawly.API.Router 61.54% | Crawly.Models.YMLSpider 66.67% | Crawly.Models.Job 75.00% | Crawly.Pipelines.JSONEncoder 76.47% | Crawly.SimpleStorage 76.92% | Crawly.Fetchers.CrawlyRenderServer 77.78% | Crawly.Pipelines.Experimental.SendToUI 80.00% | Crawly.EngineSup 80.77% | Crawly 82.61% | Crawly.Pipelines.WriteToFile 83.33% | Crawly.Middlewares.UserAgent 85.53% | Crawly.Engine 87.50% | Crawly.Application 87.50% | TestUtils 88.37% | Crawly.RequestsStorage 88.46% | Crawly.RequestsStorage.Worker 89.06% | Crawly.Manager 89.36% | Crawly.Utils 90.00% | Crawly.Pipelines.Validate 91.84% | Crawly.Worker 92.11% | Crawly.DataStorage 92.86% | Crawly.DataStorage.Worker 100.00% | Crawly.Fetchers.Fetcher 100.00% | Crawly.Fetchers.HTTPoisonFetcher 100.00% | Crawly.ManagerSup 100.00% | Crawly.Middlewares.AutoCookiesManager 100.00% | Crawly.Middlewares.DomainFilter 100.00% | Crawly.Middlewares.RequestOptions 100.00% | Crawly.Middlewares.RobotsTxt 100.00% | Crawly.Middlewares.SameDomainFilter 100.00% | Crawly.Middlewares.UniqueRequest 100.00% | Crawly.ParsedItem 100.00% | Crawly.Pipeline 100.00% | Crawly.Pipelines.CSVEncoder 100.00% | Crawly.Pipelines.DuplicatesFilter 100.00% | Crawly.Request 100.00% | Crawly.Settings 100.00% | Crawly.Spider -----------|-------------------------- 77.06% | Total ```