search

tubndgit / scrapyx-luminati

Luminati middleware for Scrapy
2 stars 2 forks source link

Doesn't work when using scrapy_splash #2

Closed mindey closed 2 years ago

mindey commented 2 years ago

When working with dockerized version of scrapy-splash (docker run -p 8050:8050 scrapinghub/splash), I see, error in Luminati Proxy Manager Dashboard, saying: Bad Port. Ports we support: https://brightdata.com/faq#integration-ports

Settings excerpt include:

SPIDER_MIDDLEWARES = {
    'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
    'scrapy_deltafetch.DeltaFetch': 100,
}

DELTAFETCH_ENABLED = False

# Proxies
LUMINATI_ENABLED = True
LUMINATI_URL = 'http://127.0.0.1:24000'

DOWNLOADER_MIDDLEWARES = {
    'scrapyx_luminati.LumninatiProxyMiddleware': 610,
    'scrapy_splash.SplashCookiesMiddleware': 723,
    'scrapy_splash.SplashMiddleware': 725,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}

Used versions

scrapy-splash==0.8.0
scrapyx-luminati==0.1.6
mindey commented 2 years ago

Fix above specifies proxy HTTP header differently for the case when 'splash' is among the HTTP headers, and resolves the issue in my case.