search

manolo-rocks / django-manolo

Manolo buscador de visitantes a las entidades del Estado peruano
http://manolo.rocks
Other
2 stars 0 forks source link

arreglar spider produce #135

Closed aniversarioperu closed 3 years ago

josevaldiviaromero commented 3 years ago

Igual aqui acabo de corer el de produce:

(manolo_env) ➜  manolo_scraper git:(master) ✗ scrapy crawl produce
2020-11-28 14:57:43 [scrapy.extensions.telnet] INFO: Telnet Password: 7f379c42dec47de9
2020-11-28 14:57:44 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-11-28 14:57:44 [py.warnings] WARNING: /Users/jose.valdivia/manolo/manolo_env/lib/python3.7/site-packages/scrapy/spidermiddlewares/offsite.py:65: URLWarning: allowed_domains accepts only domains, not URLs. Ignoring URL entry http://www2.produce.gob.pe in allowed_domains.
  warnings.warn(message, URLWarning)

2020-11-28 14:57:44 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
SCRAPING: 2020-11-14
2020-11-28 14:57:45 [scrapy.downloadermiddlewares.cookies] DEBUG: Received cookies from: <200 http://www2.produce.gob.pe/produce/transparencia/visitas/>
Set-Cookie: cookiesession1=627BAB80IOTNMMOREAQT98GJZSG29179;Path=/;HttpOnly

SCRAPING: 2020-11-15
2020-11-28 14:57:45 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <POST http://www2.produce.gob.pe/produce/transparencia/visitas/>
Cookie: cookiesession1=627BAB80IOTNMMOREAQT98GJZSG29179

SCRAPING: 2020-11-16
2020-11-28 14:57:50 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <POST http://www2.produce.gob.pe/produce/transparencia/visitas/>
Cookie: cookiesession1=627BAB80IOTNMMOREAQT98GJZSG29179

SCRAPING: 2020-11-17
2020-11-28 14:57:55 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <POST http://www2.produce.gob.pe/produce/transparencia/visitas/>
Cookie: cookiesession1=627BAB80IOTNMMOREAQT98GJZSG29179

saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
SCRAPING: 2020-11-18
2020-11-28 14:58:00 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <POST http://www2.produce.gob.pe/produce/transparencia/visitas/>
Cookie: cookiesession1=627BAB80IOTNMMOREAQT98GJZSG29179

saving to db item
saving to db item
saving to db item
saving to db item
SCRAPING: 2020-11-19
2020-11-28 14:58:07 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <POST http://www2.produce.gob.pe/produce/transparencia/visitas/>
Cookie: cookiesession1=627BAB80IOTNMMOREAQT98GJZSG29179

saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
SCRAPING: 2020-11-20
2020-11-28 14:58:14 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <POST http://www2.produce.gob.pe/produce/transparencia/visitas/>
Cookie: cookiesession1=627BAB80IOTNMMOREAQT98GJZSG29179

saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
SCRAPING: 2020-11-21
2020-11-28 14:58:20 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <POST http://www2.produce.gob.pe/produce/transparencia/visitas/>
Cookie: cookiesession1=627BAB80IOTNMMOREAQT98GJZSG29179

saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
SCRAPING: 2020-11-22
2020-11-28 14:58:26 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <POST http://www2.produce.gob.pe/produce/transparencia/visitas/>
Cookie: cookiesession1=627BAB80IOTNMMOREAQT98GJZSG29179

SCRAPING: 2020-11-23
2020-11-28 14:58:33 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <POST http://www2.produce.gob.pe/produce/transparencia/visitas/>
Cookie: cookiesession1=627BAB80IOTNMMOREAQT98GJZSG29179

SCRAPING: 2020-11-24
2020-11-28 14:58:40 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <POST http://www2.produce.gob.pe/produce/transparencia/visitas/>
Cookie: cookiesession1=627BAB80IOTNMMOREAQT98GJZSG29179

saving to db item
saving to db item
saving to db item
2020-11-28 14:58:44 [scrapy.extensions.logstats] INFO: Crawled 10 pages (at 10 pages/min), scraped 57 items (at 57 items/min)
SCRAPING: 2020-11-25
2020-11-28 14:58:47 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <POST http://www2.produce.gob.pe/produce/transparencia/visitas/>
Cookie: cookiesession1=627BAB80IOTNMMOREAQT98GJZSG29179

saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
SCRAPING: 2020-11-26
2020-11-28 14:58:52 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <POST http://www2.produce.gob.pe/produce/transparencia/visitas/>
Cookie: cookiesession1=627BAB80IOTNMMOREAQT98GJZSG29179

saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
SCRAPING: 2020-11-27
2020-11-28 14:58:58 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <POST http://www2.produce.gob.pe/produce/transparencia/visitas/>
Cookie: cookiesession1=627BAB80IOTNMMOREAQT98GJZSG29179

saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
SCRAPING: 2020-11-28
2020-11-28 14:59:05 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <POST http://www2.produce.gob.pe/produce/transparencia/visitas/>
Cookie: cookiesession1=627BAB80IOTNMMOREAQT98GJZSG29179

saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
saving to db item
Found 0 errors: []
2020-11-28 14:59:10 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 7878,
 'downloader/request_count': 15,
 'downloader/request_method_count/POST': 15,
 'downloader/response_bytes': 166469,
 'downloader/response_count': 15,
 'downloader/response_status_count/200': 15,
 'elapsed_time_seconds': 86.653364,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2020, 11, 28, 14, 59, 10, 971096),
 'item_scraped_count': 118,
 'log_count/DEBUG': 15,
 'log_count/INFO': 4,
 'log_count/WARNING': 1,
 'memusage/max': 83021824,
 'memusage/startup': 79851520,
 'response_received_count': 15,
 'scheduler/dequeued': 15,
 'scheduler/dequeued/memory': 15,
 'scheduler/enqueued': 15,
 'scheduler/enqueued/memory': 15,
 'start_time': datetime.datetime(2020, 11, 28, 14, 57, 44, 317732)}

Recibo esto de output, y en manolo

image

aniversarioperu commented 3 years ago

no se esta escrapeando entidad ni officina/sede

Screen Shot 2020-11-28 at 12 41 35 Screen Shot 2020-11-28 at 12 41 54
aniversarioperu commented 3 years ago

y no se porque se escrapea Despacho ministerial como oficina/cargo, deberia ser Ministro de la produccion