search

scrapy / scrapyd

A service daemon to run Scrapy spiders
https://scrapyd.readthedocs.io/en/stable/
BSD 3-Clause "New" or "Revised" License
2.92k stars 569 forks source link

encoding errors when running a spider via scrapyd #462

Closed dedepene closed 1 year ago

dedepene commented 1 year ago

I'm running into this error (please see below) that is not present if i just do "scrapy crawl spider_name". Seems there's something wrong with the encoding? Also, cyrillic prints out just fine in the spider logs until the spider is started by scrapyd - then I'm getting it all messed up like so 'name': 'Панталон чино по тялото',

scrapyd log:


2022-12-07 14:06:37 [scrapy.utils.log] INFO: Versions: lxml 4.9.1.0, libxml2 2.9.12, cssselect 1.2.0, parsel 1.7.0, w3lib 2.0.1, Twisted 22.10.0, Python 3.10.8 (tags/v3.10.8:aaaf517, Oct 11 2022, 16:50:30) [MSC v.1933 64 bit (AMD64)], pyOpenSSL 22.1.0 (OpenSSL 3.0.7 1 Nov 2022), cryptography 38.0.3, Platform Windows-10-10.0.22000-SP0
2022-12-07 14:06:37 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'scrapy_app',
 'CONCURRENT_REQUESTS_PER_DOMAIN': 4,
 'DOWNLOAD_DELAY': 3,
 'LOG_FILE': 'logs\\default\\FashionDays\\990436b2762711edaefbb42e99ec0e4f.log',
 'NEWSPIDER_MODULE': 'scrapy_app.spiders',
 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7',
 'ROBOTSTXT_OBEY': True,
 'SPIDER_MODULES': ['scrapy_app.spiders']}
2022-12-07 14:06:37 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2022-12-07 14:06:37 [scrapy.extensions.telnet] INFO: Telnet Password: 17764e54054cab8f
2022-12-07 14:06:38 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
2022-12-07 14:06:38 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedBrowserType] Family: Android
2022-12-07 14:06:38 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other, Brand: None, Model: None
2022-12-07 14:06:38 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other, Brand: None, Model: None
2022-12-07 14:06:38 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other, Brand: None, Model: None
2022-12-07 14:06:38 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Sony BDV13, Brand: Sony, Model: BDV13
2022-12-07 14:06:38 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Nintendo DSi, Brand: Nintendo, Model: DSi
2022-12-07 14:06:38 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other, Brand: None, Model: None
2022-12-07 14:06:38 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedBrowserType] Family: Other
2022-12-07 14:06:38 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: LYF F90M, Brand: LYF, Model: F90M
2022-12-07 14:06:38 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Sony BDV14, Brand: Sony, Model: BDV14
2022-12-07 14:06:38 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: LG Web0S SmartTV, Brand: LG, Model: Web0S SmartTV
2022-12-07 14:06:38 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other, Brand: None, Model: None
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other, Brand: None, Model: None
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other, Brand: None, Model: None
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other, Brand: None, Model: None
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other, Brand: None, Model: None
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other, Brand: None, Model: None
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other, Brand: None, Model: None
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Sony BDV11, Brand: Sony, Model: BDV11
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other, Brand: None, Model: None
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other, Brand: None, Model: None
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other, Brand: None, Model: None
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other, Brand: None, Model: None
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Nintendo DSi, Brand: Nintendo, Model: DSi
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: LYF LF-2403N, Brand: LYF, Model: LF-2403N
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: LYF F90M, Brand: LYF, Model: F90M
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other, Brand: None, Model: None
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other, Brand: None, Model: None
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedBrowserType] Family: Applebot
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other, Brand: None, Model: None
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedBrowserType] Family: Applebot
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedBrowserType] Family: Other
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedBrowserType] Family: Other
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedBrowserType] Family: PhantomJS
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedBrowserType] Family: SMTBot
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedBrowserType] Family: PhantomJS
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedBrowserType] Family: PhantomJS
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedBrowserType] Family: WebKit Nightly
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedBrowserType] Family: PhantomJS
2022-12-07 14:06:39 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other, Brand: None, Model: None
2022-12-07 14:06:40 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedBrowserType] Family: PhantomJS
2022-12-07 14:06:40 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedBrowserType] Family: PhantomJS
2022-12-07 14:06:40 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedBrowserType] Family: Robot
2022-12-07 14:06:40 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other, Brand: None, Model: None
2022-12-07 14:06:40 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedBrowserType] Family: Zune
2022-12-07 14:06:40 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy_user_agents.middlewares.RandomUserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2022-12-07 14:06:40 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2022-12-07 14:06:40 [scrapy.middleware] INFO: Enabled item pipelines:
['scrapy_app.pipelines.ScrapyAppPipeline']
2022-12-07 14:06:40 [scrapy.core.engine] INFO: Spider opened
2022-12-07 14:06:40 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-12-07 14:06:40 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2022-12-07 14:06:40 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36
2022-12-07 14:06:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.fashiondays.bg/robots.txt> (referer: None)
2022-12-07 14:06:40 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36
2022-12-07 14:06:40 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36
2022-12-07 14:06:40 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31
2022-12-07 14:06:40 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36
2022-12-07 14:06:40 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.73 Safari/537.36
2022-12-07 14:06:40 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36
2022-12-07 14:06:40 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36
2022-12-07 14:06:40 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36
2022-12-07 14:06:40 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36
2022-12-07 14:06:40 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36
2022-12-07 14:06:40 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36
2022-12-07 14:06:40 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
2022-12-07 14:06:40 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36
2022-12-07 14:06:40 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36
2022-12-07 14:06:40 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36
2022-12-07 14:06:40 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36
2022-12-07 14:06:42 [filelock] DEBUG: Attempting to acquire lock 1876492010160 on i:\dev\projects\price_tracker_django\.venv\lib\site-packages\tldextract\.suffix_cache/publicsuffix.org-tlds\de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2022-12-07 14:06:42 [filelock] DEBUG: Lock 1876492010160 acquired on i:\dev\projects\price_tracker_django\.venv\lib\site-packages\tldextract\.suffix_cache/publicsuffix.org-tlds\de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2022-12-07 14:06:42 [filelock] DEBUG: Attempting to release lock 1876492010160 on i:\dev\projects\price_tracker_django\.venv\lib\site-packages\tldextract\.suffix_cache/publicsuffix.org-tlds\de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2022-12-07 14:06:42 [filelock] DEBUG: Lock 1876492010160 released on i:\dev\projects\price_tracker_django\.venv\lib\site-packages\tldextract\.suffix_cache/publicsuffix.org-tlds\de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2022-12-07 14:06:42 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.fashiondays.bg/p/%D0%9F%D0%B0%D0%BD%D1%82%D0%B0%D0%BB%D0%BE%D0%BD-%D1%87%D0%B8%D0%BD%D0%BE-%D0%BF%D0%BE-%D1%82%D1%8F%D0%BB%D0%BE%D1%82%D0%BE-%D0%9C%D1%8A%D0%B6%D0%B5-gap-p5664120-2/> (referer: None)
2022-12-07 14:06:42 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36
2022-12-07 14:06:43 [FashionDays] INFO: Parse function called on https://www.fashiondays.bg/p/%D0%9F%D0%B0%D0%BD%D1%82%D0%B0%D0%BB%D0%BE%D0%BD-%D1%87%D0%B8%D0%BD%D0%BE-%D0%BF%D0%BE-%D1%82%D1%8F%D0%BB%D0%BE%D1%82%D0%BE-%D0%9C%D1%8A%D0%B6%D0%B5-gap-p5664120-2/
2022-12-07 14:06:43 [scrapy.core.scraper] ERROR: Error processing {'availability': 'InStock',
 'id': 'sku:000440943-288_fdays',
 'name': 'Панталон чино по тялото',
 'price': Decimal('63.99')}
Traceback (most recent call last):
  File "i:\dev\projects\price_tracker_django\.venv\lib\site-packages\twisted\internet\defer.py", line 892, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
  File "i:\dev\projects\price_tracker_django\.venv\lib\site-packages\scrapy\utils\defer.py", line 285, in f
    return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
  File "I:\Dev\projects\price_tracker_django\scrapy_app\scrapy_app\pipelines.py", line 25, in process_item
    print ("pipeline item:", item['name'], " matched to database item: ", i.title, "with DB price:", i.current_price, "and online price:", item['price'], "should be decimal:", type(item['price']), 'AVAILABILITY: ', item['availability'])
  File "C:\Users\dedepene\AppData\Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-7: character maps to <undefined>
2022-12-07 14:06:45 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.fashiondays.bg/p/%D0%91%D0%B5%D0%B7%D1%88%D0%B5%D0%B2%D0%BD%D0%B0-%D0%B1%D1%80%D0%B0%D0%B7%D0%B8%D0%BB%D0%B8%D0%B0%D0%BD%D0%B0-3-%D1%87%D0%B8%D1%84%D1%82%D0%B0-%D0%96%D0%B5%D0%BD%D0%B8-tommy-hilfiger-p5662809-1/> (referer: None)
2022-12-07 14:06:45 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.65 Safari/537.36
2022-12-07 14:06:46 [FashionDays] INFO: Parse function called on https://www.fashiondays.bg/p/%D0%91%D0%B5%D0%B7%D1%88%D0%B5%D0%B2%D0%BD%D0%B0-%D0%B1%D1%80%D0%B0%D0%B7%D0%B8%D0%BB%D0%B8%D0%B0%D0%BD%D0%B0-3-%D1%87%D0%B8%D1%84%D1%82%D0%B0-%D0%96%D0%B5%D0%BD%D0%B8-tommy-hilfiger-p5662809-1/
2022-12-07 14:06:46 [scrapy.core.scraper] ERROR: Error processing {'availability': 'InStock',
 'id': 'sku:UW0UW03871-0SL_fdays',
 'name': 'Безшевна бразилиана, 3 чифта',
 'price': Decimal('55.99')}
Traceback (most recent call last):
  File "i:\dev\projects\price_tracker_django\.venv\lib\site-packages\twisted\internet\defer.py", line 892, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
  File "i:\dev\projects\price_tracker_django\.venv\lib\site-packages\scrapy\utils\defer.py", line 285, in f
    return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
  File "I:\Dev\projects\price_tracker_django\scrapy_app\scrapy_app\pipelines.py", line 25, in process_item
    print ("pipeline item:", item['name'], " matched to database item: ", i.title, "with DB price:", i.current_price, "and online price:", item['price'], "should be decimal:", type(item['price']), 'AVAILABILITY: ', item['availability'])
  File "C:\Users\dedepene\AppData\Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-7: character maps to <undefined>
2022-12-07 14:06:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.fashiondays.bg/p/%D0%91%D0%B5%D0%B7%D1%88%D0%B5%D0%B2%D0%BD%D0%B0-%D0%B1%D1%80%D0%B0%D0%B7%D0%B8%D0%BB%D0%B8%D0%B0%D0%BD%D0%B0-3-%D1%87%D0%B8%D1%84%D1%82%D0%B0-%D0%96%D0%B5%D0%BD%D0%B8-tommy-hilfiger-p5662815-1/> (referer: None)
2022-12-07 14:06:50 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36
2022-12-07 14:06:50 [FashionDays] INFO: Parse function called on https://www.fashiondays.bg/p/%D0%91%D0%B5%D0%B7%D1%88%D0%B5%D0%B2%D0%BD%D0%B0-%D0%B1%D1%80%D0%B0%D0%B7%D0%B8%D0%BB%D0%B8%D0%B0%D0%BD%D0%B0-3-%D1%87%D0%B8%D1%84%D1%82%D0%B0-%D0%96%D0%B5%D0%BD%D0%B8-tommy-hilfiger-p5662815-1/
2022-12-07 14:06:50 [scrapy.core.scraper] ERROR: Error processing {'availability': 'InStock',
 'id': 'sku:UW0UW03871-0W5_fdays',
 'name': 'Безшевна бразилиана, 3 чифта',
 'price': Decimal('55.99')}
Traceback (most recent call last):
  File "i:\dev\projects\price_tracker_django\.venv\lib\site-packages\twisted\internet\defer.py", line 892, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
  File "i:\dev\projects\price_tracker_django\.venv\lib\site-packages\scrapy\utils\defer.py", line 285, in f
    return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
  File "I:\Dev\projects\price_tracker_django\scrapy_app\scrapy_app\pipelines.py", line 25, in process_item
    print ("pipeline item:", item['name'], " matched to database item: ", i.title, "with DB price:", i.current_price, "and online price:", item['price'], "should be decimal:", type(item['price']), 'AVAILABILITY: ', item['availability'])
  File "C:\Users\dedepene\AppData\Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-7: character maps to <undefined>```
jpmckinney commented 1 year ago

Looks like you're running on Windows. I suspect that when you run scrapy crawl spider_name, your shell is properly configured to set an appropriate encoding. However, in the environment in which scrapyd is running, the encoding is misconfigured, resulting in an error. I'm not familiar with deployment on Windows, so don't know how to set encodings.

jpmckinney commented 1 year ago

Closing as no response.