Comprehensive examples

[danielnaab]

Pomp looks like a nice and simple design - I'm going to give it a try while migrating an existing Scrapy project to Python 3.

However, I would really like to see some more comprehensive examples in the documentation or the repository.

For instance, a larger project would:

• Use requests middleware for auth, proxies, etc
• Use asyncio with some kind of queue
• Export to a database
• Run as a daemon or a scheduled job

These things can be rolled together by any competent Python dev, but I think demonstrating one or more ways to build a full-scale production deployment might help gain a few users.

[estin]

I agree.

In plans build demo project in separate repo under docker containers via docker-compose

Were would be:

• postgres as database (preferred by me)
• django admin panel to view data
• export data to the database via django models with detection new data (bulk insert or import csv)
• centralized queue on redis
• pomp worker builded on top of requests or aiohttp with lxml, where item extracting would be paralleled (ConcurrentCrawler)

This example show how use Pomp to build distributed robust app.

Do you know what web resource would be the best target for that? And allow legal scraping.

Sorry for my poor English

[danielnaab]

That sounds great. I'm using Django models also, but haven't worked out how to handle queuing yet... Another idea for an example is throttling speed by domain.

Here are some data ideas:

• Crawl Ratebeer or BeerAdvocate to create a database of beers and their styles.
• Crawl Github for top projects & contributors
• Scrape every city in Craigslist for a specific search term, to get an idea of what market prices for used items are.

I had an additional question after reading through the code: is using pomp.core.item.Item and pomp.core.item.Field required, or can the crawlers just return dictionaries instead? If not required, maybe they should go in contrib? (I was looking into using voluptuous or marshmallow for schema.)

[estin]

maybe they should go in contrib? (I was looking into using voluptuous or marshmallow for schema.)

Yes! It must be placed in contrib and do not restrict users. Thanks!

pomp.core.item moved to pomp.contrib.item without backward compatibility

Item interface necessary for developing plugged pipelines like pomp.contrib.pipelines.CsvPipeline where order of fields required.

[estin]

big example - Craigslist crawler Soon I will publish screencast.

But this example much more about how build cluster of web crawlers and less about pomp features and it internals.

[estin]

screencast without audio of craigslist crawler

and don`t know when I will write more detailed description and publish it on reddit or others resources

[danielnaab]

Awesome, thanks!