search

juliomalegria / python-craigslist

Simple Craigslist wrapper
MIT No Attribution
387 stars 117 forks source link

refactor to separate IO from parsing? #27

Closed AlJohri closed 7 years ago

AlJohri commented 7 years ago

hi @juliomalegria, I was trying to use your library in some code I'm working on where I'm controlling much of the IO myself. I was having a little trouble because the IO and parsing logic is currently intertwined. The parts that I was hoping to import are:

1) parsing the listing page 2) getting the filters that are available on different types of sections 3) getting the base url depending on the site (city) or section

The main reason for handling the IO myself is so I can handle caching, use whichever IO library I want (requests, aiohttp, etc.), perform threading or multiprocessing whenever and wherever I want to, and download and parse the actual post itself in addition to the geotagging data.

Is this something you'd be interested in doing? It's definitely not a simple change.

juliomalegria commented 7 years ago

This all sound like good improvements! I'll try to work on them whenever I get some free time, but I cannot define any ETA. I'd be very willing to accept a pull request though, if you feel like contributing :)

AlJohri commented 7 years ago

hi @juliomalegria, I ended up implementing my own version to separate the IO from the business logic, allow using asyncio, dynamically grab the filters available on different sections, add tests, and create a simple CLI.

You can check it out here: https://github.com/AlJohri/craigslist

Would love any feedback!